-
Bug
-
Resolution: Unresolved
-
Critical
-
None
-
quay-v3.10.0, quay-v3.9.6
-
False
-
None
-
False
-
0
Description:
After upgrade Quay 3.7 Operator to 3.9.6, user can't login Quay by UI or CLI, postgres-old pods (clair-postgres-old and quay-database-old) are still there.
Index Image:
rh-osbs/quay-quay-operator-bundle:v3.9.6-1
Index image v4.14: registry-proxy.engineering.redhat.com/rh-osbs/iib:626971
Steps:
1, Deploy Quay latest 3.7 Operator on OCP, create valid quay registry and push a image,
2, Follow the guide, create ISCP, CatalogSouce etc 3.9.6 related resources,
3, Upgrade to v3.9.6 iib:626971, check Pod status
Result:
DB migration doesn't finished, clair-postgres-old and quay-database-old pod still there
NAME READY STATUS RESTARTS AGE
quay-operator.v3.9.6-58c8d7c768-l4h4w 1/1 Running 0 126m
quayregistry-clair-app-5f6ffb74cf-49pcm 1/1 Running 2 (125m ago) 125m
quayregistry-clair-app-5f6ffb74cf-4pxzq 1/1 Running 2 (125m ago) 125m
quayregistry-clair-postgres-6886ff4898-82xqs 1/1 Running 0 123m
quayregistry-clair-postgres-old-59cc74f6fc-wwqbh 1/1 Running 0 126m
quayregistry-clair-postgres-upgrade-c2w5j 0/1 Completed 2 126m
quayregistry-quay-app-7c9d794d66-2x758 1/1 Running 0 125m
quayregistry-quay-app-7c9d794d66-lrfzk 1/1 Running 0 125m
quayregistry-quay-app-upgrade-r9pdp 0/1 Completed 0 125m
quayregistry-quay-config-editor-9d6db6669-vgkm5 1/1 Running 0 126m
quayregistry-quay-database-85994bd985-5ng5h 1/1 Running 0 125m
quayregistry-quay-database-old-6d68965c68-n477p 1/1 Running 0 126m
quayregistry-quay-mirror-7f45f9c978-78csf 1/1 Running 0 125m
quayregistry-quay-mirror-7f45f9c978-z7vk6 1/1 Running 0 125m
quayregistry-quay-postgres-upgrade-78mvr 0/1 Completed 0 126m
quayregistry-quay-redis-85897d8685-pbf9b 1/1 Running 0 126m
also in deployment and job,
[cloud-user@quay-sean-standalone ~]$ oc get deployment NAME READY UP-TO-DATE AVAILABLE AGE quay-operator.v3.9.6 1/1 1 1 126m quayregistry-clair-app 2/2 2 2 134m quayregistry-clair-postgres 1/1 1 1 134m quayregistry-clair-postgres-old 1/1 1 1 126m quayregistry-quay-app 2/2 2 2 134m quayregistry-quay-config-editor 1/1 1 1 134m quayregistry-quay-database 1/1 1 1 134m quayregistry-quay-database-old 1/1 1 1 126m quayregistry-quay-mirror 2/2 2 2 134m quayregistry-quay-redis 1/1 1 1 134m [cloud-user@quay-sean-standalone ~]$ oc get job NAME COMPLETIONS DURATION AGE quayregistry-clair-postgres-upgrade 1/1 2m22s 126m quayregistry-quay-app-upgrade 1/1 23s 126m quayregistry-quay-postgres-upgrade 1/1 45s 126m [cloud-user@quay-sean-standalone ~]$ oc get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE quay-operator ClusterIP 172.30.69.209 <none> 7071/TCP 136m quayregistry-clair-app ClusterIP 172.30.203.253 <none> 80/TCP,8089/TCP 134m quayregistry-clair-postgres ClusterIP 172.30.45.79 <none> 5432/TCP 134m quayregistry-quay-app ClusterIP 172.30.162.211 <none> 443/TCP,80/TCP,8081/TCP,55443/TCP 134m quayregistry-quay-config-editor ClusterIP 172.30.134.4 <none> 80/TCP 134m quayregistry-quay-database ClusterIP 172.30.69.34 <none> 5432/TCP 134m quayregistry-quay-redis ClusterIP 172.30.125.154 <none> 6379/TCP 134m
quay-database pod description:
[cloud-user@quay-sean-standalone ~]$ oc describe pod quayregistry-quay-database-85994bd985-5ng5h
.................
Containers:
postgres:
Container ID: cri-o://97337b7f885cf7a216aebc7d7df406ba1114d99532b7d2638d5e6d29415ffa8a
Image: registry.redhat.io/rhel8/postgresql-13@sha256:f0083c3398501e3b7c82e7f865cd3377ff14cbfb14b1f8f91d7889232afa4796
Image ID: registry.redhat.io/rhel8/postgresql-13@sha256:3a260b1a031cae02a86f0201592342e0368060b8fcc4b986443597bc04482935
login failure, always stay at the login page:
in Quay app pod log,
gunicorn-registry stdout | 2023-11-29 07:22:24,906 [208] [ERROR] [util.security.registry_jwt] Could not find requested service key 80XieSzPMOGQzzygHPzJ3Y56NM3vGBd7gE591-N-ca0 with encoded JWT: eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6IjgwWGllU3pQTU9HUXp6eWdIUHpKM1k1Nk5NM3ZHQmQ3Z0U1OTEtTi1jYTAifQ.eyJpc3MiOiJxdWF5IiwiYXVkIjoicXVheXJlZ2lzdHJ5LXF1YXktcXVheS1lbnRlcnByaXNlLTE0NDk2LmFwcHMucXVheXRlc3QtMTQ0OTYucWUuZGV2Y2x1c3Rlci5vcGVuc2hpZnQuY29tIiwibmJmIjoxNzAxMjQyNTQ0LCJpYXQiOjE3MDEyNDI1NDQsImV4cCI6MTcwMTI0NjE0NCwic3ViIjoicXVheSIsImFjY2VzcyI6W10sImNvbnRleHQiOnsidmVyc2lvbiI6MiwiZW50aXR5X2tpbmQiOiJ1c2VyIiwiZW50aXR5X3JlZmVyZW5jZSI6IjIwOTdkNTY1LWI5YmQtNDY2YS04YjY4LTk5MGVlYzczMGM3YSIsImtpbmQiOiJ1c2VyIiwidXNlciI6InF1YXkiLCJjb20uYXBvc3RpbGxlLnJvb3RzIjp7fSwiY29tLmFwb3N0aWxsZS5yb290IjoiJGRpc2FibGVkIn19.Kh2n9PZVN_-qkxH9aReVglHtlyH7E76p2XYZ1T3vA0NtmifFGa3dcVmESLwTOaqtAM7Co7O8MpPvZPSWvfN92b4JVxxuztRyRcaW_oA0LU6zuIOsVYw9hOErkuEWGlwSHTxZYyp4kJnG2vNPA777_nagDkO11LJ0oDRpuheBFTcN26yZHMTs0zAigWMl5uJrvNe4xg2OngMTR9pCGn0FPAqnZEUfhQHXhCaT3lMWz6tjd0Rev7xulGtOZvupHozjyMhJnbLppZ5JycPTlOBOH1bj4kMFzdNew4LzGhoLkWyEiIPfFEmoKjc2fw7q1vAWeRDQLJheftxDMxZPiY4EuA gunicorn-registry stdout | 2023-11-29 07:22:24,906 [208] [ERROR] [auth.registry_jwt_auth] Invalid bearer token: Unknown service key gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/auth/registry_jwt_auth.py", line 104, in identity_from_bearer_token gunicorn-registry stdout | payload = decode_bearer_header(bearer_header, instance_keys, app.config) gunicorn-registry stdout | File "/quay-registry/util/security/registry_jwt.py", line 54, in decode_bearer_header gunicorn-registry stdout | return decode_bearer_token(encoded_jwt, instance_keys, config) gunicorn-registry stdout | File "/quay-registry/util/security/registry_jwt.py", line 70, in wrapper gunicorn-registry stdout | raise e gunicorn-registry stdout | File "/quay-registry/util/security/registry_jwt.py", line 67, in wrapper gunicorn-registry stdout | rv = func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/util/security/registry_jwt.py", line 105, in decode_bearer_token gunicorn-registry stdout | raise InvalidBearerTokenException("Unknown service key") gunicorn-registry stdout | util.security.registry_jwt.InvalidBearerTokenException: Unknown service key gunicorn-registry stdout | 2023-11-29 07:22:24,907 [208] [ERROR] [util.http] Error 401: Unknown service key; Arguments: {'url': 'https://quayregistry-quay-quay-enterprise-14496.apps.quaytest-14496.qe.devcluster.openshift.com/v2/', 'status_code': 401, 'message': 'Unknown service key'} nginx stdout | 10.128.2.2 (-) - - [29/Nov/2023:07:22:24 +0000] "GET /v2/ HTTP/1.1" 401 32 "-" "containers/5.28.0 (github.com/containers/image)" (0.010 1544 0.010) gunicorn-registry stdout | 2023-11-29 07:22:24,908 [208] [INFO] [gunicorn.access] 10.128.2.2 - - [29/Nov/2023:07:22:24 +0000] "GET /v2/ HTTP/1.1" 401 32 "-" "containers/5.28.0 (github.com/containers/image)"
config.yaml
BROWSER_API_CALLS_XHR_ONLY: false CREATE_PRIVATE_REPO_ON_PUSH: true CREATE_NAMESPACE_ON_PUSH: true FEATURE_QUOTA_MANAGEMENT: true FEATURE_PROXY_CACHE: true CREATE_REPOSITORY_ON_PUSH_PUBLIC: true FEATURE_EXTENDED_REPOSITORY_NAMES: true FEATURE_USER_INITIALIZE: true FEATURE_GENERAL_OCI_SUPPORT: true FEATURE_HELM_OCI_SUPPORT: true PERMANENTLY_DELETE_TAGS: true SUPER_USERS: - quay - admin FEATURE_UI_V2: true FEATURE_SUPERUSERS_FULL_ACCESS: true #FEATURE_AUTO_PRUNE: true #FEATURE_UI_V2_REPO_SETTINGS: true DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: - default DISTRIBUTED_STORAGE_PREFERENCE: - default DISTRIBUTED_STORAGE_CONFIG: default: - S3Storage - s3_bucket: quayprowci25592 storage_path: /datafile s3_access_key: xxx s3_secret_key: xxx host: s3.us-east-2.amazonaws.com s3_region: us-east-2
quay registry
spec: components: - kind: quay managed: true - kind: postgres managed: true - kind: clair managed: true - kind: redis managed: true - kind: horizontalpodautoscaler managed: false - kind: objectstorage managed: false - kind: route managed: true - kind: mirror managed: true - kind: monitoring managed: false - kind: tls managed: true - kind: clairpostgres managed: true
- is related to
-
-
- Testing
-
- links to
