Loading...

Type: Bug
Resolution: Done
Priority: Critical
Fix Version/s: quay-v3.9.0
Affects Version/s: quay-v3.9.0
Component/s: quay
Labels:
- ReleaseBlocker
- triaged

Blocked:
False
Blocked Reason:
None
Ready:
False
Epic Link:
Splunk Forwarding
Intelligence Requested:
Market:

RICE Score:
0

SFDC Cases Links:
SFDC Cases Counter:

Description:

This is an issue found in Quay 3.9.0, when deploy Quay with "LOGS_MODEL: splunk", and choose to verify SSL, during creating Quay config bundle Secret, provide the CA Cert of Splunk Server, the results is Quay upgrade POD is crashed, checked POD logs get error message "ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)", here the problem is Quay upgrade POD can't trust the self-signed Cert of Splunk Server. Pls review this issue.

Quay Version: quay-quay-operator-bundle:v3.9.0-74

Steps:

Deploy Quay Operator
Create Quay config bundle secret with "oc create secret generic --from-file config.yaml=./config_390.yaml --from-file extra_ca_cert_splunksercer.crt=./splunksercer.crt config-bundle-secret"
Create QuayRegistry with "oc create -f quayregistry_390.yaml"
Check Quay PODs status

Expected Results:

All Quay PODs come to running status, and QuayRegistry is ready.

Actual Results:

Quay Upgrade POD is crashed, with error message "ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)"

Quay Config.yaml:

LOGS_MODEL: splunk
LOGS_MODEL_CONFIG:
  producer: splunk
  splunk_config:
    host: 3.20.237.130
    port: 8089
    bearer_token: 'eyJraWQiOiJzcGx1bmsuc2VjcmV0IiwiYWxnIjoiSFM1MTIiLCJ2ZXIiOiJ2MiIsInR0eXAiOiJzdGF0aWMifQ.eyJpc3MiOiJhZG1pbiBmcm9tIGNkZTIxY2NmZDQ3YSIsInN1YiI6ImFkbWluIiwiYXVkIjoidGVzdCBxdWF5MzkwIiwiaWRwIjoiU3BsdW5rIiwianRpIjoiNjQ3ZmI1Nzc4NjRjOTlhM2Q5MjBmNGUwNzE5MDUzMjFhN2ExNzYyNTYyMDA4NDcwODY2MWRhYWFmMzBmZWM5NSIsImlhdCI6MTY4Mzc3NjU1MCwiZXhwIjoxNjg2MzY4NTUwLCJuYnIiOjE2ODM3NzY1NTB9.zQLVkVoK3S-JDGuvAy6wiujQXc6N-L8XUq-CRvin7bxUfo2hUc7ZHixL5vZW9U7sFWSF1wdO7Ig20GHXOdd9dQ'
    url_scheme: https
    verify_ssl: True
    index_prefix: quay39074
CREATE_PRIVATE_REPO_ON_PUSH: true
CREATE_NAMESPACE_ON_PUSH: true
FEATURE_EXTENDED_REPOSITORY_NAMES: true
FEATURE_QUOTA_MANAGEMENT: true
FEATURE_PROXY_CACHE: true
FEATURE_USER_INITIALIZE: true
ALLOWED_OCI_ARTIFACT_TYPES: 
    application/vnd.cncf.helm.config.v1+json: 
    - application/tar+gzip
    application/vnd.oci.image.layer.v1.tar+gzip+encrypted:
    - application/vnd.oci.image.layer.v1.tar+gzip+encrypted
DEFAULT_TAG_EXPIRATION: 4w
TAG_EXPIRATION_OPTIONS:
- 2w
- 4w
- 8w
FEATURE_GENERAL_OCI_SUPPORT: true
FEATURE_HELM_OCI_SUPPORT: true
SUPER_USERS:
  - quay
  - admin

Quay POD logs:

oc get pod
NAME                                            READY   STATUS             RESTARTS        AGE
quay-operator.v3.9.0-64d6f6b7dd-7ddzb           1/1     Running            0               27m
quay39074-clair-app-b8cdb58fb-8gtw4             1/1     Running            0               76s
quay39074-clair-app-b8cdb58fb-8sdm8             1/1     Running            0               76s
quay39074-clair-app-b8cdb58fb-dd4mm             1/1     Running            0               2m1s
quay39074-clair-app-b8cdb58fb-ddv66             1/1     Running            0               2m1s
quay39074-clair-app-b8cdb58fb-f8cgd             1/1     Running            0               8m1s
quay39074-clair-app-b8cdb58fb-g5cvr             1/1     Running            0               2m16s
quay39074-clair-app-b8cdb58fb-jlh9t             1/1     Running            0               46s
quay39074-clair-app-b8cdb58fb-qb48v             1/1     Running            0               7m46s
quay39074-clair-postgres-7d4854b65f-72g6k       1/1     Running            1 (7m46s ago)   8m1s
quay39074-quay-app-upgrade-6wfhn                0/1     CrashLoopBackOff   3 (26s ago)     87s
quay39074-quay-config-editor-6dbbc5f48b-jrmfq   1/1     Running            0               7m51s
quay39074-quay-database-f7f98944b-qdkh8         1/1     Running            0               8m1s
quay39074-quay-mirror-5954b8774f-qc7xn          0/1     Init:0/1           3 (34s ago)     7m20s
quay39074-quay-mirror-5954b8774f-rstxh          0/1     Init:0/1           3 (31s ago)     7m20s
quay39074-quay-redis-58969fd647-xkbt4           1/1     Running            0               8m1s


oc logs quay39074-quay-app-upgrade-6wfhn
   __   __
  /  \ /  \     ______   _    _     __   __   __
 / /\ / /\ \   /  __  \ | |  | |   /  \  \ \ / /
/ /  / /  \ \  | |  | | | |  | |  / /\ \  \   /
\ \  \ \  / /  | |__| | | |__| | / ____ \  | |
 \ \/ \ \/ /   \_  ___/  \____/ /_/    \_\ |_|
  \__/ \__/      \ \__
                  \___\ by Red Hat
 Build, Store, and Distribute your Containers


Startup timestamp: 
Thu May 11 10:23:16 UTC 2023


Entering migration mode to version: head
Traceback (most recent call last):
  File "/usr/local/bin/alembic", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/alembic/config.py", line 575, in main
    CommandLine(prog=prog).main(argv=argv)
  File "/usr/local/lib/python3.9/site-packages/alembic/config.py", line 569, in main
    self.run_cmd(cfg, options)
  File "/usr/local/lib/python3.9/site-packages/alembic/config.py", line 546, in run_cmd
    fn(
  File "/usr/local/lib/python3.9/site-packages/alembic/command.py", line 298, in upgrade
    script.run_env()
  File "/usr/local/lib/python3.9/site-packages/alembic/script/base.py", line 489, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 98, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.9/site-packages/alembic/util/compat.py", line 173, in load_module_py
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "data/migrations/env.py", line 13, in <module>
    from app import app
  File "/quay-registry/app.py", line 344, in <module>
    logs_model.configure(app.config)
  File "/quay-registry/data/logs_model/__init__.py", line 67, in configure
    logs_model.initialize(_LOG_MODELS[model_name](**model_config))
  File "/quay-registry/data/logs_model/splunk_logs_model.py", line 26, in __init__
    self._logs_producer.initialize(SplunkLogsProducer(**splunk_config))
  File "/quay-registry/data/logs_model/logs_producer/splunk_logs_producer.py", line 34, in __init__
    self.index = service.indexes[index_prefix]
  File "/usr/local/lib/python3.9/site-packages/splunklib/client.py", line 1376, in __getitem__
    response = self.get(key)
  File "/usr/local/lib/python3.9/site-packages/splunklib/client.py", line 1804, in get
    return super(Collection, self).get(name, owner, app, sharing, **query)
  File "/usr/local/lib/python3.9/site-packages/splunklib/client.py", line 862, in get
    return self.service.get(path,
  File "/usr/local/lib/python3.9/site-packages/splunklib/binding.py", line 292, in wrapper
    return request_fun(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/splunklib/binding.py", line 73, in new_f
    val = f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/splunklib/binding.py", line 699, in get
    response = self.http.get(path, all_headers, **query)
  File "/usr/local/lib/python3.9/site-packages/splunklib/binding.py", line 1232, in get
    return self.request(url, { 'method': "GET", 'headers': headers })
  File "/usr/local/lib/python3.9/site-packages/splunklib/binding.py", line 1294, in request
    response = self.handler(url, message, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/splunklib/binding.py", line 1450, in request
    connection.request(method, path, body, head)
  File "/usr/lib64/python3.9/http/client.py", line 1285, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output
    self.send(msg)
  File "/usr/lib64/python3.9/http/client.py", line 980, in send
    self.connect()
  File "/usr/lib64/python3.9/http/client.py", line 1454, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/usr/lib64/python3.9/ssl.py", line 501, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib64/python3.9/ssl.py", line 1041, in _create
    self.do_handshake()
  File "/usr/lib64/python3.9/ssl.py", line 1310, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)

Checked Quay Config editor, the CA Cert of Splunk Server is mounted correctly.

Quay Config Editor:

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

image-2023-05-11-18-25-07-683.png
434 kB
2023/05/11 10:25 AM
image-2023-05-24-17-05-08-011.png
215 kB
2023/05/24 9:05 AM
splunkserver.crt
1 kB
2023/06/13 12:13 AM

Details

Description

Attachments

Attachments

Activity

People

Dates