-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
quay-v3.8.0
-
False
-
None
-
False
-
-
Description of problem:
In IPv6 environment, mirror pod will go Init:CrashLoopBackOff by following below steps.
1. Configure "Red Hat Quay handles TLS" and upload self-signed certificates in config-tool web page
configure TLS
2. Reconfigure the quay
3. Check all pods reconciled successfully
4. Update any one configuration in config-tool web page, such as adding a new super user.
Add a new super user
5. reconfigure the quay again.
6. mirror pod goes Init:CrashLoopBackOff
$ oc get pod
NAME READY STATUS RESTARTS AGE
quay-operator.v3.8.0-548ddd9676-kpwln 1/1 Running 0 9h
quayregistry-clair-app-69db7b4b67-krpjq 1/1 Running 0 58m
quayregistry-clair-app-69db7b4b67-kvpb2 1/1 Running 0 57m
quayregistry-clair-postgres-776db9c859-ksjcl 1/1 Running 1 (89m ago) 90m
quayregistry-quay-app-f9cc7dd4d-r7dgb 1/1 Running 0 55m
quayregistry-quay-app-f9cc7dd4d-wf78x 1/1 Running 0 55m
quayregistry-quay-app-upgrade-xwh7z 0/1 Completed 1 90m
quayregistry-quay-config-editor-fc655db57-gmzv6 1/1 Running 0 55m
quayregistry-quay-database-546c54576b-pf7c5 1/1 Running 0 90m
quayregistry-quay-mirror-5476b49959-2qn74 0/1 Init:CrashLoopBackOff 15 (3m32s ago) 55m
quayregistry-quay-mirror-5476b49959-57gmt 0/1 Init:CrashLoopBackOff 15 (3m23s ago) 55m
quayregistry-quay-redis-6d68fd764-qj764 1/1 Running 0 90m
Version-Release number of selected component (if applicable):
quay 3.8.0 quay-operator-bundle-container-v3.8.0-12 registry.redhat.io/quay/quay-operator-rhel8@sha256:182b420e2b3e606547a771b0ca0b07098a3ada45ac0c85461152d2b44f251459 registry.redhat.io/quay/quay-rhel8@sha256:a97945f7a39973f6e217ea4ecbe2fc77c81632df8104e88dc190be81d2aad3a6
Expected results:
mirror pod should be reconciled successfully
Additional info:
Following the same steps in IPv4 environment, everything is fine.
The self-signed certificate
$ openssl x509 -in ssl.cert -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
70:bf:ba:f9:0d:bd:f6:ef:bd:39:de:50:34:18:17:17:9c:cc:39:9b
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = CN, ST = Beijing, L = BJ, O = Quay team, OU = Quay QE Team, CN = .apps.whuipv618.qe.devcluster.openshift.com
Validity
Not Before: Dec 5 12:03:46 2022 GMT
Not After : Nov 26 12:03:46 2023 GMT
Subject: C = CN, ST = Beijing, L = BJ, O = Quay team, OU = Quay QE Team, CN = *.apps.whuipv618.qe.devcluster.openshift.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:c7:5d:7d:62:9c:2a:bc:1a:fd:0c:0f:eb:66:88:
.......
5f:91
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
X509v3 Key Usage:
Digital Signature, Non Repudiation, Key Encipherment
X509v3 Subject Alternative Name:
DNS:quayregistry-quay-builder-quay-enterprise.apps.whuipv618.qe.devcluster.openshift.com, DNS:quayregistry-quay-quay-enterprise.apps.whuipv618.qe.devcluster.openshift.com, DNS:*.apps.whuipv618.qe.devcluster.openshift.com
Signature Algorithm: sha256WithRSAEncryption
63:d5:39:23:b5:2b:06:68:af:3e:d1:36:19:3a:2e:66:f1:e4:
......
69:25:dc:e7
The config.yaml
$ oc -n quay-enterprise rsh $(oc get pod -l app=quay -o NAME -n quay-enterprise|head -n 1) cat /conf/stack/config.yaml ALLOW_PULLS_WITHOUT_STRICT_LOGGING: false AUTHENTICATION_TYPE: Database AVATAR_KIND: local BROWSER_API_CALLS_XHR_ONLY: false BUILDLOGS_REDIS: host: quayregistry-quay-redis port: 6379 CREATE_NAMESPACE_ON_PUSH: true CREATE_PRIVATE_REPO_ON_PUSH: true DATABASE_SECRET_KEY: qNy0tt-3rDuRzA20jgfoCu2yblwRmER5xWdNITOh1MZWJBvDylKR6XWrbO76OPCDFSU8HKbYNGc0Psdr DB_CONNECTION_ARGS: autorollback: true threadlocals: true DB_URI: postgresql://quayregistry-quay-database:sIolseE5lus8WT1BZCcViVKuhJtgzs3LeW4eKrlzY0DoT59jlVPXKyL97-raqCmeMFp7w3ADZW2kA0CW@quayregistry-quay-database:5432/quayregistry-quay-database DEFAULT_TAG_EXPIRATION: 2w DISTRIBUTED_STORAGE_CONFIG: default: - S3Storage - host: s3.dualstack.us-east-2.amazonaws.com s3_access_key: AK....UP s3_bucket: whuaws s3_secret_key: c...4 storage_path: /quaydata DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: - default DISTRIBUTED_STORAGE_PREFERENCE: - default ENTERPRISE_LOGO_URL: /static/img/RH_Logo_Quay_Black_UX-horizontal.svg EXTERNAL_TLS_TERMINATION: false FEATURE_ACTION_LOG_ROTATION: false FEATURE_ANONYMOUS_ACCESS: true FEATURE_APP_REGISTRY: true FEATURE_APP_SPECIFIC_TOKENS: true FEATURE_BITBUCKET_BUILD: false FEATURE_BLACKLISTED_EMAILS: false FEATURE_BUILD_SUPPORT: false FEATURE_CHANGE_TAG_EXPIRATION: true FEATURE_DIRECT_LOGIN: true FEATURE_EXTENDED_REPOSITORY_NAMES: true FEATURE_FIPS: false FEATURE_GITHUB_BUILD: false FEATURE_GITHUB_LOGIN: false FEATURE_GITLAB_BUILD: false FEATURE_GOOGLE_LOGIN: false FEATURE_INVITE_ONLY_USER_CREATION: false FEATURE_LISTEN_IP_VERSION: IPv6 FEATURE_MAILING: true FEATURE_NONSUPERUSER_TEAM_SYNCING_SETUP: false FEATURE_PARTIAL_USER_AUTOCOMPLETE: true FEATURE_PROXY_CACHE: true FEATURE_PROXY_STORAGE: false FEATURE_QUOTA_MANAGEMENT: true FEATURE_REPO_MIRROR: true FEATURE_RESTRICTED_USERS: true FEATURE_SECURITY_NOTIFICATIONS: true FEATURE_SECURITY_SCANNER: true FEATURE_STORAGE_REPLICATION: false FEATURE_SUPERUSERS_FULL_ACCESS: true FEATURE_TEAM_SYNCING: false FEATURE_UI_V2: true FEATURE_USER_CREATION: true FEATURE_USER_INITIALIZE: true FEATURE_USER_LAST_ACCESSED: true FEATURE_USER_LOG_ACCESS: false FEATURE_USER_METADATA: false FEATURE_USER_RENAME: false FEATURE_USERNAME_CONFIRMATION: false FRESH_LOGIN_TIMEOUT: 10m GITHUB_LOGIN_CONFIG: {} GITHUB_TRIGGER_CONFIG: {} GITLAB_TRIGGER_KIND: {} LDAP_ALLOW_INSECURE_FALLBACK: false LDAP_EMAIL_ATTR: mail LDAP_UID_ATTR: uid LDAP_URI: ldap://localhost LOGS_MODEL: database LOGS_MODEL_CONFIG: {} PREFERRED_URL_SCHEME: https REGISTRY_TITLE: whu Red Hat Quay REGISTRY_TITLE_SHORT: Red Hat Quay REPO_MIRROR_INTERVAL: 30 REPO_MIRROR_TLS_VERIFY: true RESTRICTED_USERS_WHITELIST: - user1 - whuquay - whutest SEARCH_MAX_RESULT_PAGE_COUNT: 10 SEARCH_RESULTS_PER_PAGE: 10 SECRET_KEY: wQri1TZCOV4ZmMDhSXrDSDRF1TnCGKrj5natD-dLT-oNjgRgeGrXhxFcWQWtwp08PVGoDJNQUohjlmC0 SECURITY_SCANNER_INDEXING_INTERVAL: 30 SECURITY_SCANNER_V4_ENDPOINT: http://quayregistry-clair-app.quay-enterprise.svc.cluster.local SECURITY_SCANNER_V4_NAMESPACE_WHITELIST: - admin SECURITY_SCANNER_V4_PSK: RGZSaTQwSTJ2ZFhrd05Zd3hsOTYwU0dERm5lWG5PdVk= SERVER_HOSTNAME: quayregistry-quay-quay-enterprise.apps.whuipv618.qe.devcluster.openshift.com SETUP_COMPLETE: true SUPER_USERS: - whuquay - whutest - whu1234 - whu2345 TAG_EXPIRATION_OPTIONS: - 2w TEAM_RESYNC_STALE_TIME: 60m TESTING: false USER_EVENTS_REDIS: host: quayregistry-quay-redis port: 6379 USER_RECOVERY_TOKEN_LIFETIME: 30m
The information about crashed mirror pod.
$ oc describe pod quayregistry-quay-mirror-5476b49959-2qn74 Name: quayregistry-quay-mirror-5476b49959-2qn74 Namespace: quay-enterprise Priority: 0 Node: worker-00.whuipv618.qe.devcluster.openshift.com/2604:1380:4642:7e00::35 Start Time: Mon, 05 Dec 2022 20:07:24 +0800 Labels: pod-template-hash=5476b49959 quay-component=quay-mirror quay-operator/quayregistry=quayregistry ..... Status: Pending IP: fd01:0:0:4::77 IPs: IP: fd01:0:0:4::77 Controlled By: ReplicaSet/quayregistry-quay-mirror-5476b49959 Init Containers: quay-mirror-init: Container ID: cri-o://48de0ec6bd61b432e813b3598c27bcf6c42335929d90f607c3abe062fce51593 Image: registry.redhat.io/quay/quay-rhel8@sha256:a97945f7a39973f6e217ea4ecbe2fc77c81632df8104e88dc190be81d2aad3a6 Image ID: registry.redhat.io/quay/quay-rhel8@sha256:7f7ac0670e0074f88a7771a7d48007973f188a9c44210c234fc0ea0a50dec657 Port: <none> Host Port: <none> Command: /bin/sh -c curl $QUAY_APP_SERVICE_HOST --connect-timeout 360 State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 7 Started: Mon, 05 Dec 2022 20:23:27 +0800 Finished: Mon, 05 Dec 2022 20:23:27 +0800 Ready: False Restart Count: 8 Environment: QUAY_APP_SERVICE_HOST: quayregistry-quay-app Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nmbqx (ro) Containers: quay-mirror: Container ID: Image: registry.redhat.io/quay/quay-rhel8@sha256:a97945f7a39973f6e217ea4ecbe2fc77c81632df8104e88dc190be81d2aad3a6 Image ID: Port: <none> Host Port: <none> Command: /quay-registry/quay-entrypoint.sh Args: repomirror-nomigrate State: Waiting Reason: PodInitializing Ready: False Restart Count: 0 Limits: cpu: 1 memory: 2Gi Requests: cpu: 500m memory: 512Mi Environment: QE_K8S_CONFIG_SECRET: quayregistry-quay-config-secret-dc24kfg5d2 QE_K8S_NAMESPACE: quay-enterprise (v1:metadata.namespace) DEBUGLOG: false ENSURE_NO_MIGRATION: true HTTP_PROXY: <set to the key 'HTTP_PROXY' in secret 'quayregistry-quay-proxy-config-2bt6m898b9'> Optional: false HTTPS_PROXY: <set to the key 'HTTPS_PROXY' in secret 'quayregistry-quay-proxy-config-2bt6m898b9'> Optional: false NO_PROXY: <set to the key 'NO_PROXY' in secret 'quayregistry-quay-proxy-config-2bt6m898b9'> Optional: false Mounts: /conf/stack from config (rw) /conf/stack/extra_ca_certs from extra-ca-certs (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nmbqx (ro) Conditions: Type Status Initialized False Ready False ContainersReady False PodScheduled True Volumes: config: Type: Projected (a volume that contains injected data from multiple sources) SecretName: quayregistry-quay-config-secret-dc24kfg5d2 SecretOptionalName: <nil> SecretName: quayregistry-quay-config-tls-4868dk6h67 SecretOptionalName: <nil> extra-ca-certs: Type: Projected (a volume that contains injected data from multiple sources) ConfigMapName: quayregistry-cluster-service-ca ConfigMapOptional: <nil> ConfigMapName: quayregistry-cluster-trusted-ca ConfigMapOptional: <nil> SecretName: quayregistry-extra-ca-certs-d9b8hk6665 SecretOptionalName: <nil> SecretName: quayregistry-quay-config-tls-4868dk6h67 SecretOptionalName: <nil> kube-api-access-nmbqx: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: <nil> QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 17m default-scheduler Successfully assigned quay-enterprise/quayregistry-quay-mirror-5476b49959-2qn74 to worker-00.whuipv618.qe.devcluster.openshift.com by master-01.whuipv618.qe.devcluster.openshift.com Normal AddedInterface 16m multus Add eth0 [fd01:0:0:4::77/64] from ovn-kubernetes Normal Pulled 15m (x5 over 16m) kubelet Container image "registry.redhat.io/quay/quay-rhel8@sha256:a97945f7a39973f6e217ea4ecbe2fc77c81632df8104e88dc190be81d2aad3a6" already present on machine Normal Created 15m (x5 over 16m) kubelet Created container quay-mirror-init Normal Started 15m (x5 over 16m) kubelet Started container quay-mirror-init Warning BackOff 107s (x70 over 16m) kubelet Back-off restarting failed container
It seems mirror pod initialization return error 7.
Run below test in quay pod.
$ oc rsh quayregistry-quay-app-f9cc7dd4d-r7dgb sh-4.4$ curl quayregistry-quay-app --connect-timeout 360 curl: (7) Failed to connect to quayregistry-quay-app port 80: Connection refused sh-4.4$ echo $? 7
The information about "quayregistry-quay-app"
$ oc describe service quayregistry-quay-app Name: quayregistry-quay-app Namespace: quay-enterprise Labels: app=quay quay-component=quay quay-operator/quayregistry=quayregistry Annotations: quay-buildmanager-hostname: quay-component: quay quay-operator-service-endpoint: http://quay-operator.quay-enterprise.svc.cluster.local:7071 quay-registry-hostname: quayregistry-quay-quay-enterprise.apps.whuipv618.qe.devcluster.openshift.com Selector: app=quay,quay-component=quay-app,quay-operator/quayregistry=quayregistry Type: ClusterIP IP Family Policy: SingleStack IP Families: IPv6 IP: fd02::cb27 IPs: fd02::cb27 Port: https 443/TCP TargetPort: 8443/TCP Endpoints: [fd01:0:0:4::78]:8443,[fd01:0:0:5::43]:8443 Port: http 80/TCP TargetPort: 8080/TCP Endpoints: [fd01:0:0:4::78]:8080,[fd01:0:0:5::43]:8080 Port: jwtproxy 8081/TCP TargetPort: 8081/TCP Endpoints: [fd01:0:0:4::78]:8081,[fd01:0:0:5::43]:8081 Port: grpc 55443/TCP TargetPort: 55443/TCP Endpoints: [fd01:0:0:4::78]:55443,[fd01:0:0:5::43]:55443 Session Affinity: None Events: <none>
