-
Bug
-
Resolution: Done
-
Critical
-
quay-v3.9.0
-
False
-
None
-
False
-
-
Description:
This is an issue found in Quay 3.9.0, when deploy Quay with "LOGS_MODEL: splunk", and choose to verify SSL, during creating Quay config bundle Secret, provide the CA Cert of Splunk Server, the results is Quay upgrade POD is crashed, checked POD logs get error message "ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)", here the problem is Quay upgrade POD can't trust the self-signed Cert of Splunk Server. Pls review this issue.
Quay Version: quay-quay-operator-bundle:v3.9.0-74
Steps:
- Deploy Quay Operator
- Create Quay config bundle secret with "oc create secret generic --from-file config.yaml=./config_390.yaml --from-file extra_ca_cert_splunksercer.crt=./splunksercer.crt config-bundle-secret"
- Create QuayRegistry with "oc create -f quayregistry_390.yaml"
- Check Quay PODs status
Expected Results:
All Quay PODs come to running status, and QuayRegistry is ready.
Actual Results:
Quay Upgrade POD is crashed, with error message "ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)"
Quay Config.yaml:
LOGS_MODEL: splunk LOGS_MODEL_CONFIG: producer: splunk splunk_config: host: 3.20.237.130 port: 8089 bearer_token: 'eyJraWQiOiJzcGx1bmsuc2VjcmV0IiwiYWxnIjoiSFM1MTIiLCJ2ZXIiOiJ2MiIsInR0eXAiOiJzdGF0aWMifQ.eyJpc3MiOiJhZG1pbiBmcm9tIGNkZTIxY2NmZDQ3YSIsInN1YiI6ImFkbWluIiwiYXVkIjoidGVzdCBxdWF5MzkwIiwiaWRwIjoiU3BsdW5rIiwianRpIjoiNjQ3ZmI1Nzc4NjRjOTlhM2Q5MjBmNGUwNzE5MDUzMjFhN2ExNzYyNTYyMDA4NDcwODY2MWRhYWFmMzBmZWM5NSIsImlhdCI6MTY4Mzc3NjU1MCwiZXhwIjoxNjg2MzY4NTUwLCJuYnIiOjE2ODM3NzY1NTB9.zQLVkVoK3S-JDGuvAy6wiujQXc6N-L8XUq-CRvin7bxUfo2hUc7ZHixL5vZW9U7sFWSF1wdO7Ig20GHXOdd9dQ' url_scheme: https verify_ssl: True index_prefix: quay39074 CREATE_PRIVATE_REPO_ON_PUSH: true CREATE_NAMESPACE_ON_PUSH: true FEATURE_EXTENDED_REPOSITORY_NAMES: true FEATURE_QUOTA_MANAGEMENT: true FEATURE_PROXY_CACHE: true FEATURE_USER_INITIALIZE: true ALLOWED_OCI_ARTIFACT_TYPES: application/vnd.cncf.helm.config.v1+json: - application/tar+gzip application/vnd.oci.image.layer.v1.tar+gzip+encrypted: - application/vnd.oci.image.layer.v1.tar+gzip+encrypted DEFAULT_TAG_EXPIRATION: 4w TAG_EXPIRATION_OPTIONS: - 2w - 4w - 8w FEATURE_GENERAL_OCI_SUPPORT: true FEATURE_HELM_OCI_SUPPORT: true SUPER_USERS: - quay - admin
Quay POD logs:
oc get pod NAME READY STATUS RESTARTS AGE quay-operator.v3.9.0-64d6f6b7dd-7ddzb 1/1 Running 0 27m quay39074-clair-app-b8cdb58fb-8gtw4 1/1 Running 0 76s quay39074-clair-app-b8cdb58fb-8sdm8 1/1 Running 0 76s quay39074-clair-app-b8cdb58fb-dd4mm 1/1 Running 0 2m1s quay39074-clair-app-b8cdb58fb-ddv66 1/1 Running 0 2m1s quay39074-clair-app-b8cdb58fb-f8cgd 1/1 Running 0 8m1s quay39074-clair-app-b8cdb58fb-g5cvr 1/1 Running 0 2m16s quay39074-clair-app-b8cdb58fb-jlh9t 1/1 Running 0 46s quay39074-clair-app-b8cdb58fb-qb48v 1/1 Running 0 7m46s quay39074-clair-postgres-7d4854b65f-72g6k 1/1 Running 1 (7m46s ago) 8m1s quay39074-quay-app-upgrade-6wfhn 0/1 CrashLoopBackOff 3 (26s ago) 87s quay39074-quay-config-editor-6dbbc5f48b-jrmfq 1/1 Running 0 7m51s quay39074-quay-database-f7f98944b-qdkh8 1/1 Running 0 8m1s quay39074-quay-mirror-5954b8774f-qc7xn 0/1 Init:0/1 3 (34s ago) 7m20s quay39074-quay-mirror-5954b8774f-rstxh 0/1 Init:0/1 3 (31s ago) 7m20s quay39074-quay-redis-58969fd647-xkbt4 1/1 Running 0 8m1s oc logs quay39074-quay-app-upgrade-6wfhn __ __ / \ / \ ______ _ _ __ __ __ / /\ / /\ \ / __ \ | | | | / \ \ \ / / / / / / \ \ | | | | | | | | / /\ \ \ / \ \ \ \ / / | |__| | | |__| | / ____ \ | | \ \/ \ \/ / \_ ___/ \____/ /_/ \_\ |_| \__/ \__/ \ \__ \___\ by Red Hat Build, Store, and Distribute your Containers Startup timestamp: Thu May 11 10:23:16 UTC 2023 Entering migration mode to version: head Traceback (most recent call last): File "/usr/local/bin/alembic", line 8, in <module> sys.exit(main()) File "/usr/local/lib/python3.9/site-packages/alembic/config.py", line 575, in main CommandLine(prog=prog).main(argv=argv) File "/usr/local/lib/python3.9/site-packages/alembic/config.py", line 569, in main self.run_cmd(cfg, options) File "/usr/local/lib/python3.9/site-packages/alembic/config.py", line 546, in run_cmd fn( File "/usr/local/lib/python3.9/site-packages/alembic/command.py", line 298, in upgrade script.run_env() File "/usr/local/lib/python3.9/site-packages/alembic/script/base.py", line 489, in run_env util.load_python_file(self.dir, "env.py") File "/usr/local/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 98, in load_python_file module = load_module_py(module_id, path) File "/usr/local/lib/python3.9/site-packages/alembic/util/compat.py", line 173, in load_module_py spec.loader.exec_module(module) File "<frozen importlib._bootstrap_external>", line 850, in exec_module File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed File "data/migrations/env.py", line 13, in <module> from app import app File "/quay-registry/app.py", line 344, in <module> logs_model.configure(app.config) File "/quay-registry/data/logs_model/__init__.py", line 67, in configure logs_model.initialize(_LOG_MODELS[model_name](**model_config)) File "/quay-registry/data/logs_model/splunk_logs_model.py", line 26, in __init__ self._logs_producer.initialize(SplunkLogsProducer(**splunk_config)) File "/quay-registry/data/logs_model/logs_producer/splunk_logs_producer.py", line 34, in __init__ self.index = service.indexes[index_prefix] File "/usr/local/lib/python3.9/site-packages/splunklib/client.py", line 1376, in __getitem__ response = self.get(key) File "/usr/local/lib/python3.9/site-packages/splunklib/client.py", line 1804, in get return super(Collection, self).get(name, owner, app, sharing, **query) File "/usr/local/lib/python3.9/site-packages/splunklib/client.py", line 862, in get return self.service.get(path, File "/usr/local/lib/python3.9/site-packages/splunklib/binding.py", line 292, in wrapper return request_fun(self, *args, **kwargs) File "/usr/local/lib/python3.9/site-packages/splunklib/binding.py", line 73, in new_f val = f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/splunklib/binding.py", line 699, in get response = self.http.get(path, all_headers, **query) File "/usr/local/lib/python3.9/site-packages/splunklib/binding.py", line 1232, in get return self.request(url, { 'method': "GET", 'headers': headers }) File "/usr/local/lib/python3.9/site-packages/splunklib/binding.py", line 1294, in request response = self.handler(url, message, **kwargs) File "/usr/local/lib/python3.9/site-packages/splunklib/binding.py", line 1450, in request connection.request(method, path, body, head) File "/usr/lib64/python3.9/http/client.py", line 1285, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output self.send(msg) File "/usr/lib64/python3.9/http/client.py", line 980, in send self.connect() File "/usr/lib64/python3.9/http/client.py", line 1454, in connect self.sock = self._context.wrap_socket(self.sock, File "/usr/lib64/python3.9/ssl.py", line 501, in wrap_socket return self.sslsocket_class._create( File "/usr/lib64/python3.9/ssl.py", line 1041, in _create self.do_handshake() File "/usr/lib64/python3.9/ssl.py", line 1310, in do_handshake self._sslobj.do_handshake() ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)
Checked Quay Config editor, the CA Cert of Splunk Server is mounted correctly.
Quay Config Editor:
