Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-6468

postgres-old pods don't removed after upgrade impact login

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • None
    • quay-v3.10.0, quay-v3.9.6
    • quay-operator
    • False
    • None
    • False

      Description:

      After upgrade Quay 3.7 Operator to 3.9.6, user can't login Quay by UI or CLI, postgres-old pods (clair-postgres-old and quay-database-old) are still there.

       

      Index Image:

      rh-osbs/quay-quay-operator-bundle:v3.9.6-1

      Index image v4.14: registry-proxy.engineering.redhat.com/rh-osbs/iib:626971

       

      Steps:

      1, Deploy Quay latest 3.7 Operator on OCP, create valid quay registry and push a image,

      2, Follow the guide, create ISCP, CatalogSouce etc 3.9.6 related resources,

      3, Upgrade to v3.9.6 iib:626971, check Pod status

       

      Result:

      DB migration doesn't finished, clair-postgres-old and quay-database-old pod still there

      NAME                                               READY   STATUS      RESTARTS       AGE
quay-operator.v3.9.6-58c8d7c768-l4h4w              1/1     Running     0              126m
quayregistry-clair-app-5f6ffb74cf-49pcm            1/1     Running     2 (125m ago)   125m
quayregistry-clair-app-5f6ffb74cf-4pxzq            1/1     Running     2 (125m ago)   125m
quayregistry-clair-postgres-6886ff4898-82xqs       1/1     Running     0              123m
quayregistry-clair-postgres-old-59cc74f6fc-wwqbh   1/1     Running     0              126m
quayregistry-clair-postgres-upgrade-c2w5j          0/1     Completed   2              126m
quayregistry-quay-app-7c9d794d66-2x758             1/1     Running     0              125m
quayregistry-quay-app-7c9d794d66-lrfzk             1/1     Running     0              125m
quayregistry-quay-app-upgrade-r9pdp                0/1     Completed   0              125m
quayregistry-quay-config-editor-9d6db6669-vgkm5    1/1     Running     0              126m
quayregistry-quay-database-85994bd985-5ng5h        1/1     Running     0              125m
quayregistry-quay-database-old-6d68965c68-n477p    1/1     Running     0              126m
quayregistry-quay-mirror-7f45f9c978-78csf          1/1     Running     0              125m
quayregistry-quay-mirror-7f45f9c978-z7vk6          1/1     Running     0              125m
quayregistry-quay-postgres-upgrade-78mvr           0/1     Completed   0              126m
quayregistry-quay-redis-85897d8685-pbf9b           1/1     Running     0              126m

      also in deployment and job,

      [cloud-user@quay-sean-standalone ~]$ oc get deployment
NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
quay-operator.v3.9.6              1/1     1            1           126m
quayregistry-clair-app            2/2     2            2           134m
quayregistry-clair-postgres       1/1     1            1           134m
quayregistry-clair-postgres-old   1/1     1            1           126m
quayregistry-quay-app             2/2     2            2           134m
quayregistry-quay-config-editor   1/1     1            1           134m
quayregistry-quay-database        1/1     1            1           134m
quayregistry-quay-database-old    1/1     1            1           126m
quayregistry-quay-mirror          2/2     2            2           134m
quayregistry-quay-redis           1/1     1            1           134m 

[cloud-user@quay-sean-standalone ~]$ oc get job 
NAME                                  COMPLETIONS   DURATION   AGE 
quayregistry-clair-postgres-upgrade   1/1           2m22s      126m 
quayregistry-quay-app-upgrade         1/1           23s        126m 
quayregistry-quay-postgres-upgrade    1/1           45s        126m

[cloud-user@quay-sean-standalone ~]$ oc get svc
NAME                              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                             AGE
quay-operator                     ClusterIP   172.30.69.209    <none>        7071/TCP                            136m
quayregistry-clair-app            ClusterIP   172.30.203.253   <none>        80/TCP,8089/TCP                     134m
quayregistry-clair-postgres       ClusterIP   172.30.45.79     <none>        5432/TCP                            134m
quayregistry-quay-app             ClusterIP   172.30.162.211   <none>        443/TCP,80/TCP,8081/TCP,55443/TCP   134m
quayregistry-quay-config-editor   ClusterIP   172.30.134.4     <none>        80/TCP                              134m
quayregistry-quay-database        ClusterIP   172.30.69.34     <none>        5432/TCP                            134m
quayregistry-quay-redis           ClusterIP   172.30.125.154   <none>        6379/TCP                            134m

       

      quay-database pod description:

      [cloud-user@quay-sean-standalone ~]$ oc describe pod quayregistry-quay-database-85994bd985-5ng5h
.................
Containers:
  postgres:
    Container ID:   cri-o://97337b7f885cf7a216aebc7d7df406ba1114d99532b7d2638d5e6d29415ffa8a
    Image:          registry.redhat.io/rhel8/postgresql-13@sha256:f0083c3398501e3b7c82e7f865cd3377ff14cbfb14b1f8f91d7889232afa4796
    Image ID:       registry.redhat.io/rhel8/postgresql-13@sha256:3a260b1a031cae02a86f0201592342e0368060b8fcc4b986443597bc04482935

       

      login failure, always stay at the login page:

      in Quay app pod log,

      gunicorn-registry stdout | 2023-11-29 07:22:24,906 [208] [ERROR] [util.security.registry_jwt] Could not find requested service key 80XieSzPMOGQzzygHPzJ3Y56NM3vGBd7gE591-N-ca0 with encoded JWT: eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6IjgwWGllU3pQTU9HUXp6eWdIUHpKM1k1Nk5NM3ZHQmQ3Z0U1OTEtTi1jYTAifQ.eyJpc3MiOiJxdWF5IiwiYXVkIjoicXVheXJlZ2lzdHJ5LXF1YXktcXVheS1lbnRlcnByaXNlLTE0NDk2LmFwcHMucXVheXRlc3QtMTQ0OTYucWUuZGV2Y2x1c3Rlci5vcGVuc2hpZnQuY29tIiwibmJmIjoxNzAxMjQyNTQ0LCJpYXQiOjE3MDEyNDI1NDQsImV4cCI6MTcwMTI0NjE0NCwic3ViIjoicXVheSIsImFjY2VzcyI6W10sImNvbnRleHQiOnsidmVyc2lvbiI6MiwiZW50aXR5X2tpbmQiOiJ1c2VyIiwiZW50aXR5X3JlZmVyZW5jZSI6IjIwOTdkNTY1LWI5YmQtNDY2YS04YjY4LTk5MGVlYzczMGM3YSIsImtpbmQiOiJ1c2VyIiwidXNlciI6InF1YXkiLCJjb20uYXBvc3RpbGxlLnJvb3RzIjp7fSwiY29tLmFwb3N0aWxsZS5yb290IjoiJGRpc2FibGVkIn19.Kh2n9PZVN_-qkxH9aReVglHtlyH7E76p2XYZ1T3vA0NtmifFGa3dcVmESLwTOaqtAM7Co7O8MpPvZPSWvfN92b4JVxxuztRyRcaW_oA0LU6zuIOsVYw9hOErkuEWGlwSHTxZYyp4kJnG2vNPA777_nagDkO11LJ0oDRpuheBFTcN26yZHMTs0zAigWMl5uJrvNe4xg2OngMTR9pCGn0FPAqnZEUfhQHXhCaT3lMWz6tjd0Rev7xulGtOZvupHozjyMhJnbLppZ5JycPTlOBOH1bj4kMFzdNew4LzGhoLkWyEiIPfFEmoKjc2fw7q1vAWeRDQLJheftxDMxZPiY4EuA
gunicorn-registry stdout | 2023-11-29 07:22:24,906 [208] [ERROR] [auth.registry_jwt_auth] Invalid bearer token: Unknown service key
gunicorn-registry stdout | Traceback (most recent call last):
gunicorn-registry stdout |   File "/quay-registry/auth/registry_jwt_auth.py", line 104, in identity_from_bearer_token
gunicorn-registry stdout |     payload = decode_bearer_header(bearer_header, instance_keys, app.config)
gunicorn-registry stdout |   File "/quay-registry/util/security/registry_jwt.py", line 54, in decode_bearer_header
gunicorn-registry stdout |     return decode_bearer_token(encoded_jwt, instance_keys, config)
gunicorn-registry stdout |   File "/quay-registry/util/security/registry_jwt.py", line 70, in wrapper
gunicorn-registry stdout |     raise e
gunicorn-registry stdout |   File "/quay-registry/util/security/registry_jwt.py", line 67, in wrapper
gunicorn-registry stdout |     rv = func(*args, **kwargs)
gunicorn-registry stdout |   File "/quay-registry/util/security/registry_jwt.py", line 105, in decode_bearer_token
gunicorn-registry stdout |     raise InvalidBearerTokenException("Unknown service key")
gunicorn-registry stdout | util.security.registry_jwt.InvalidBearerTokenException: Unknown service key
gunicorn-registry stdout | 2023-11-29 07:22:24,907 [208] [ERROR] [util.http] Error 401: Unknown service key; Arguments: {'url': 'https://quayregistry-quay-quay-enterprise-14496.apps.quaytest-14496.qe.devcluster.openshift.com/v2/', 'status_code': 401, 'message': 'Unknown service key'}
nginx stdout | 10.128.2.2 (-) - - [29/Nov/2023:07:22:24 +0000] "GET /v2/ HTTP/1.1" 401 32 "-" "containers/5.28.0 (github.com/containers/image)" (0.010 1544 0.010) 
gunicorn-registry stdout | 2023-11-29 07:22:24,908 [208] [INFO] [gunicorn.access] 10.128.2.2 - - [29/Nov/2023:07:22:24 +0000] "GET /v2/ HTTP/1.1" 401 32 "-" "containers/5.28.0 (github.com/containers/image)"

       

      config.yaml

      BROWSER_API_CALLS_XHR_ONLY: false
CREATE_PRIVATE_REPO_ON_PUSH: true
CREATE_NAMESPACE_ON_PUSH: true
FEATURE_QUOTA_MANAGEMENT: true
FEATURE_PROXY_CACHE: true
CREATE_REPOSITORY_ON_PUSH_PUBLIC: true
FEATURE_EXTENDED_REPOSITORY_NAMES: true
FEATURE_USER_INITIALIZE: true
FEATURE_GENERAL_OCI_SUPPORT: true
FEATURE_HELM_OCI_SUPPORT: true
PERMANENTLY_DELETE_TAGS: true
SUPER_USERS:
  - quay
  - admin 
FEATURE_UI_V2: true
FEATURE_SUPERUSERS_FULL_ACCESS: true
#FEATURE_AUTO_PRUNE: true
#FEATURE_UI_V2_REPO_SETTINGS: true

DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS:
  - default
DISTRIBUTED_STORAGE_PREFERENCE:
  - default
DISTRIBUTED_STORAGE_CONFIG:
  default:
    - S3Storage
    - s3_bucket: quayprowci25592
      storage_path: /datafile
      s3_access_key: xxx
      s3_secret_key: xxx
      host: s3.us-east-2.amazonaws.com
      s3_region: us-east-2

       

      quay registry

      spec:
  components:
  - kind: quay
    managed: true
  - kind: postgres
    managed: true
  - kind: clair
    managed: true
  - kind: redis
    managed: true
  - kind: horizontalpodautoscaler
    managed: false
  - kind: objectstorage
    managed: false
  - kind: route
    managed: true
  - kind: mirror
    managed: true
  - kind: monitoring
    managed: false
  - kind: tls
    managed: true
  - kind: clairpostgres
    managed: true

       

        1. image-2023-11-29-16-41-46-390.png
          image-2023-11-29-16-41-46-390.png
          97 kB
        2. job_quayregistry-clair-postgres-upgrade.describe
          3 kB
        3. job_quayregistry-quay-app-upgrade.describe
          3 kB
        4. job_quayregistry-quay-postgres-upgrade.describe
          3 kB
        5. quayregistry-clair-postgres.log
          0.6 kB
        6. quayregistry-clair-postgres-old.log
          1 kB
        7. quayregistry-clair-postgres-upgrade.log
          5 kB
        8. quayregistry-quay-app-upgrade.log
          2 kB
        9. quayregistry-quay-database.log
          0.8 kB
        10. quayregistry-quay-database-old.log
          1 kB

              Unassigned Unassigned
              szhao@redhat.com Sean Zhao
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: