-
Bug
-
Resolution: Unresolved
-
Blocker
-
quay-v3.13.4
Description:
Quay 3.13.4-5 quay mirror pod can't startup when TLS is unmanged, this is a regression
quayregistry-quay-mirror-f4bdc5566-ghhfv 0/1 Init:CrashLoopBackOff 6 (16s ago) 5m57s quayregistry-quay-mirror-f4bdc5566-ntrqk 0/1 Init:Error 6 (2m59s ago) 5m57s
Index Image: quay-operator-bundle-container-v3.13.4-5
Steps:
1, Deploy Quay 3.13.4-5 Operator & registry with tls unmanaged
spec:
components:
- kind: tls
managed: false
2, Create secret bundle with prepared ssl key/cert
oc create secret generic --from-file config.yaml=config.yaml --from-file ssl.key=ssl.key --from-file extra_ca_cert_build_cluster.crt=build_cluster.crt --from-file ssl.cert=ssl.cert config-bundle-secret
3, check pod status
Actual Result:
$ oc get pod NAME READY STATUS RESTARTS AGE NAME READY STATUS RESTARTS AGE NAME READY STATUS RESTARTS AGE quay-operator.v3.13.4-66b984d48c-c2lgn 1/1 Running 0 10m quayregistry-clair-app-d754f6795-4tk6d 1/1 Running 0 8m47s quayregistry-clair-app-d754f6795-cgsvf 1/1 Running 0 3m32s quayregistry-clair-app-d754f6795-grqw6 1/1 Running 0 3m32s quayregistry-clair-app-d754f6795-kf9mg 1/1 Running 2 (9m46s ago) 9m47s quayregistry-clair-app-d754f6795-s2zzg 1/1 Running 0 8m47s quayregistry-clair-app-d754f6795-s5tl7 1/1 Running 2 (9m46s ago) 9m48s quayregistry-clair-app-d754f6795-zgb4g 1/1 Running 0 4m2s quayregistry-clair-app-d754f6795-zq7b9 1/1 Running 0 4m2s quayregistry-clair-postgres-94fbf65dc-pdsdp 1/1 Running 0 9m47s quayregistry-quay-app-7749744f89-d2h9g 1/1 Running 0 9m7s quayregistry-quay-app-7749744f89-zrk6l 1/1 Running 0 9m2s quayregistry-quay-app-upgrade-xgkpg 0/1 Completed 2 9m49s quayregistry-quay-database-59fdcb986b-4tncq 1/1 Running 0 9m46s quayregistry-quay-mirror-f4bdc5566-ghhfv 0/1 Init:CrashLoopBackOff 6 (4m5s ago) 9m46s quayregistry-quay-mirror-f4bdc5566-ntrqk 0/1 Init:CrashLoopBackOff 6 (3m59s ago) 9m46s quayregistry-quay-redis-67d78dc97-5gs62 1/1 Running 0
$ oc logs quayregistry-quay-mirror-f4bdc5566-ghhfv -c quay-mirror-init Traceback (most recent call last): File "/app/lib/python3.9/site-packages/urllib3/connectionpool.py", line 715, in urlopen httplib_response = self._make_request( File "/app/lib/python3.9/site-packages/urllib3/connectionpool.py", line 404, in _make_request self._validate_conn(conn) File "/app/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1060, in _validate_conn conn.connect() File "/app/lib/python3.9/site-packages/urllib3/connection.py", line 419, in connect self.sock = ssl_wrap_socket( File "/app/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket ssl_sock = _ssl_wrap_socket_impl( File "/app/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl return ssl_context.wrap_socket(sock, server_hostname=server_hostname) File "/usr/lib64/python3.9/ssl.py", line 501, in wrap_socket return self.sslsocket_class._create( File "/usr/lib64/python3.9/ssl.py", line 1074, in _create self.do_handshake() File "/usr/lib64/python3.9/ssl.py", line 1343, in do_handshake self._sslobj.do_handshake() ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1147) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/app/lib/python3.9/site-packages/requests/adapters.py", line 589, in send resp = conn.urlopen( File "/app/lib/python3.9/site-packages/urllib3/connectionpool.py", line 801, in urlopen retries = retries.increment( File "/app/lib/python3.9/site-packages/urllib3/util/retry.py", line 594, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='quayregistry-quay-app', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1147)')))
config.yaml
SERVER_HOSTNAME: quayregistry-quay-builder-quay-enterprise-15141.apps.quaytest-15141.qe.devcluster.openshift.com BROWSER_API_CALLS_XHR_ONLY: false PERMANENTLY_DELETE_TAGS: true RESET_CHILD_MANIFEST_EXPIRATION: true FEATURE_EXTENDED_REPOSITORY_NAMES: true CREATE_NAMESPACE_ON_PUSH: true FEATURE_QUOTA_MANAGEMENT: true FEATURE_PROXY_CACHE: true FEATURE_USER_INITIALIZE: true FEATURE_PROXY_STORAGE: true IGNORE_UNKNOWN_MEDIATYPES: true FEATURE_UI_V2: true FEATURE_SUPERUSERS_FULL_ACCESS: true FEATURE_AUTO_PRUNE: true FEATURE_IMAGE_EXPIRY_TRIGGER: true NOTIFICATION_TASK_RUN_MINIMUM_INTERVAL_MINUTES: 5 SUPER_USERS: - quay - admin GLOBAL_READONLY_SUPER_USERS: - superglobalro
% oc get quayregistry quayregistry -ojson
{
"apiVersion": "quay.redhat.com/v1",
"kind": "QuayRegistry",
"metadata": {
"annotations": {
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"quay.redhat.com/v1\",\"kind\":\"QuayRegistry\",\"metadata\":{\"annotations\":{},\"name\":\"quayregistry\",\"namespace\":\"quay-enterprise-15141\"},\"spec\":{\"components\":[{\"kind\":\"quay\",\"managed\":true,\"overrides\":{\"env\":[{\"name\":\"DEBUGLOG\",\"value\":\"true\"}]}},{\"kind\":\"postgres\",\"managed\":true},{\"kind\":\"clair\",\"managed\":true},{\"kind\":\"redis\",\"managed\":true},{\"kind\":\"horizontalpodautoscaler\",\"managed\":true},{\"kind\":\"objectstorage\",\"managed\":true},{\"kind\":\"route\",\"managed\":true},{\"kind\":\"mirror\",\"managed\":true},{\"kind\":\"monitoring\",\"managed\":false},{\"kind\":\"tls\",\"managed\":false},{\"kind\":\"clairpostgres\",\"managed\":true}],\"configBundleSecret\":\"config-bundle-secret\"}}\n"
},
"creationTimestamp": "2025-02-14T02:15:57Z",
"finalizers": [
"quay-operator/finalizer"
],
"generation": 1,
"name": "quayregistry",
"namespace": "quay-enterprise-15141",
"resourceVersion": "88146",
"uid": "fe7ed49d-9252-4bac-872e-f4655ff35e1c"
},
"spec": {
"components": [
{
"kind": "quay",
"managed": true,
"overrides": {
"env": [
{
"name": "DEBUGLOG",
"value": "true"
}
]
}
},
{
"kind": "postgres",
"managed": true
},
{
"kind": "clair",
"managed": true
},
{
"kind": "redis",
"managed": true
},
{
"kind": "horizontalpodautoscaler",
"managed": true
},
{
"kind": "objectstorage",
"managed": true
},
{
"kind": "route",
"managed": true
},
{
"kind": "mirror",
"managed": true
},
{
"kind": "monitoring",
"managed": false
},
{
"kind": "tls",
"managed": false
},
{
"kind": "clairpostgres",
"managed": true
}
],
"configBundleSecret": "config-bundle-secret"
},
"status": {
"conditions": [
{
"lastTransitionTime": "2025-02-14T02:16:01Z",
"lastUpdateTime": "2025-02-14T02:30:14Z",
"message": "Horizontal pod autoscaler found",
"reason": "ComponentReady",
"status": "True",
"type": "ComponentHPAReady"
},
{
"lastTransitionTime": "2025-02-14T02:29:12Z",
"lastUpdateTime": "2025-02-14T02:30:14Z",
"message": "Route not fully admitted",
"reason": "ComponentNotReady",
"status": "False",
"type": "ComponentRouteReady"
},
{
"lastTransitionTime": "2025-02-14T02:15:57Z",
"lastUpdateTime": "2025-02-14T02:30:14Z",
"message": "Monitoring not managed by the operator",
"reason": "ComponentNotManaged",
"status": "True",
"type": "ComponentMonitoringReady"
},
{
"lastTransitionTime": "2025-02-14T02:16:45Z",
"lastUpdateTime": "2025-02-14T02:30:14Z",
"message": "Deployment quayregistry-quay-database healthy",
"reason": "ComponentReady",
"status": "True",
"type": "ComponentPostgresReady"
},
{
"lastTransitionTime": "2025-02-14T02:16:01Z",
"lastUpdateTime": "2025-02-14T02:30:14Z",
"message": "Object bucket claim bound",
"reason": "ComponentReady",
"status": "True",
"type": "ComponentObjectStorageReady"
},
{
"lastTransitionTime": "2025-02-14T02:23:04Z",
"lastUpdateTime": "2025-02-14T02:30:14Z",
"message": "Clair component healthy",
"reason": "ComponentReady",
"status": "True",
"type": "ComponentClairReady"
},
{
"lastTransitionTime": "2025-02-14T02:16:45Z",
"lastUpdateTime": "2025-02-14T02:30:14Z",
"message": "ClairPostgres component healthy",
"reason": "ComponentReady",
"status": "True",
"type": "ComponentClairPostgresReady"
},
{
"lastTransitionTime": "2025-02-14T02:15:57Z",
"lastUpdateTime": "2025-02-14T02:30:14Z",
"message": "Config bundle contains certs",
"reason": "ComponentReady",
"status": "True",
"type": "ComponentTLSReady"
},
{
"lastTransitionTime": "2025-02-14T02:16:15Z",
"lastUpdateTime": "2025-02-14T02:30:14Z",
"message": "Deployment quayregistry-quay-redis healthy",
"reason": "ComponentReady",
"status": "True",
"type": "ComponentRedisReady"
},
{
"lastTransitionTime": "2025-02-14T02:23:04Z",
"lastUpdateTime": "2025-02-14T02:30:14Z",
"message": "Quay component healthy",
"reason": "ComponentReady",
"status": "True",
"type": "ComponentQuayReady"
},
{
"lastTransitionTime": "2025-02-14T02:15:57Z",
"lastUpdateTime": "2025-02-14T02:30:14Z",
"message": "Deployment quayregistry-quay-mirror has zero replicas available",
"reason": "ComponentNotReady",
"status": "False",
"type": "ComponentMirrorReady"
},
{
"lastTransitionTime": "2025-02-14T02:15:57Z",
"lastUpdateTime": "2025-02-14T02:30:14Z",
"message": "Some components are not ready",
"reason": "ComponentNotReady",
"status": "False",
"type": "Available"
},
{
"lastTransitionTime": "2025-02-14T02:16:45Z",
"lastUpdateTime": "2025-02-14T02:16:45Z",
"message": "All registry components created",
"reason": "ComponentsCreationSuccess",
"status": "True",
"type": "ComponentsCreated"
},
{
"lastTransitionTime": "2025-02-14T02:30:14Z",
"lastUpdateTime": "2025-02-14T02:30:14Z",
"message": "All objects created/updated successfully",
"reason": "ComponentsCreationSuccess",
"status": "False",
"type": "RolloutBlocked"
}
],
"currentVersion": "v3.13.4",
"lastUpdated": "2025-02-14 02:30:14.061271784 +0000 UTC",
"registryEndpoint": "https://quayregistry-quay-builder-quay-enterprise-15141.apps.quaytest-15141.qe.devcluster.openshift.com"
}
}
Note: this makes 4 automation cases failed
OCP-42374-Quay-High-Deploy Operator with override quay hostname with unmanaged tls OCP-42387-Quay-Medium-Deploy Operator with provided TLS Certificate and Key OCP-42393-Quay-High-Deploy with managed route and unmanaged tls with certs OCP-42396-Quay-High-Deploy Operator with unmanaged route and unmanaged tls
- links to
-
RHBA-2025:1079
Red Hat Quay v3.13.4 bug fix release
- mentioned on
