Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-3775

quayregistry-quay-app-upgrade pod can't be reconciled after change one azure storage to another azure storage by config-tool UI

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • None
    • quay-v3.7.0
    • quay-operator
    • None
    • False
    • None
    • False
    • 0

    Description

      Description of problem:

      quayregistry-quay-app-upgrade pod can't be reconciled after change one azure storage to another azure storage by config-tool UI.

      Version-Release number of selected component (if applicable):

      quay-operator-bundle-container-v3.7.0-113
      Index image v4.8: registry-proxy.engineering.redhat.com/rh-osbs/iib:227057
      

      How reproducible:

      always

      Steps to Reproduce:
      1. deploy quay with first azure storage

      DB_URI: postgresql://whu..ql1:Am...2@whupsql1.postgres.database.azure.com:5432/quayregistry_quay_database
      DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: 
        - default
      DISTRIBUTED_STORAGE_PREFERENCE:
        - default
      DISTRIBUTED_STORAGE_CONFIG:
        default:
          - AzureStorage
          - azure_account_name: whusc1 
            azure_account_key: P14cQ...sg==
            azure_container: whusc1container 
            sas_token: ?sv=201...%3D 
            storage_path: /quaydata
      FEATURE_ANONYMOUS_ACCESS: true 
      FEATURE_QUOTA_MANAGEMENT: true
      FEATURE_PROXY_CACHE: true
      BROWSER_API_CALLS_XHR_ONLY: false
      CREATE_PRIVATE_REPO_ON_PUSH: true
      CREATE_NAMESPACE_ON_PUSH: true
      FEATURE_USERNAME_CONFIRMATION: false
      FEATURE_USER_INITIALIZE: true 
      AUTHENTICATION_TYPE: Database
      

      2. When the quay run successfully, open config-tool UI and change the storage configuration to the second azure storage

      p1

      3. After validation pass, reconfigure quay

      Actual results:

      The quayregistry-quay-app-upgrade pod can't be reconciled and quay pod goes to error.

      $ oc get pod
      NAME                                              READY   STATUS             RESTARTS   AGE
      quay-operator.v3.7.0-7db9b77548-j96cx             1/1     Running            0          9m33s
      quayregistry-clair-app-dc7f5d984-4hsm2            1/1     Running            0          76s
      quayregistry-clair-app-dc7f5d984-52hlg            1/1     Running            0          96s
      quayregistry-clair-app-dc7f5d984-8x6dg            1/1     Running            0          82s
      quayregistry-clair-app-dc7f5d984-t2wdf            1/1     Running            0          96s
      quayregistry-clair-postgres-c44cd579d-zbw4w       1/1     Running            1          9m10s
      quayregistry-quay-app-568fcd4945-g4tnw            1/1     Running            0          8m40s
      quayregistry-quay-app-568fcd4945-vjxqh            1/1     Running            0          8m43s
      quayregistry-quay-app-9dd9fbf44-hkkgc             0/1     CrashLoopBackOff   3          96s
      quayregistry-quay-app-upgrade-jc7mf               0/1     Completed          0          9m10s
      quayregistry-quay-config-editor-d4fdbc5fc-v2bsk   1/1     Running            0          96s
      quayregistry-quay-mirror-6f6fcbc74-8tczh          0/1     Error              3          55s
      quayregistry-quay-mirror-6f6fcbc74-pr8j5          1/1     Running            3          55s
      quayregistry-quay-redis-f7b9c95f8-qcvfm           1/1     Running            0          9m10s
      
      $ oc logs quayregistry-quay-app-9dd9fbf44-hkkgc 
      .......
        self._legacy_model = NoopV2SecurityScanner()
      Traceback (most recent call last):
        File "/usr/local/lib/python3.8/site-packages/peewee.py", line 3057, in execute_sql
          cursor.execute(sql, params or ())
      psycopg2.errors.UndefinedTable: relation "imagestoragelocation" does not exist
      LINE 1: SELECT "t1"."id", "t1"."name" FROM "imagestoragelocation" AS...
                                                 ^
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "./boot.py", line 147, in <module>
          main()
        File "./boot.py", line 138, in main
          sync_database_with_config(app.config)
        File "/quay-registry/util/config/database.py", line 13, in sync_database_with_config
          model.image.ensure_image_locations(*location_names)
        File "/quay-registry/data/model/image.py", line 485, in ensure_image_locations
          for location in locations:
        File "/usr/local/lib/python3.8/site-packages/peewee.py", line 6731, in __iter__
          self.execute()
        File "/usr/local/lib/python3.8/site-packages/peewee.py", line 1845, in inner
          return method(self, database, *args, **kwargs)
        File "/usr/local/lib/python3.8/site-packages/peewee.py", line 1916, in execute
          return self._execute(database)
        File "/usr/local/lib/python3.8/site-packages/peewee.py", line 2087, in _execute
          cursor = database.execute(self)
        File "/usr/local/lib/python3.8/site-packages/peewee.py", line 3070, in execute
          return self.execute_sql(sql, params, commit=commit)
        File "/quay-registry/data/database.py", line 226, in execute_sql
          cursor = super(RetryOperationalError, self).execute_sql(sql, params, commit)
        File "/usr/local/lib/python3.8/site-packages/peewee.py", line 3064, in execute_sql
          self.commit()
        File "/usr/local/lib/python3.8/site-packages/peewee.py", line 2831, in __exit__
          reraise(new_type, new_type(exc_value, *exc_args), traceback)
        File "/usr/local/lib/python3.8/site-packages/peewee.py", line 183, in reraise
          raise value.with_traceback(tb)
        File "/usr/local/lib/python3.8/site-packages/peewee.py", line 3057, in execute_sql
          cursor.execute(sql, params or ())
      peewee.ProgrammingError: relation "imagestoragelocation" does not exist
      LINE 1: SELECT "t1"."id", "t1"."name" FROM "imagestoragelocation" AS...
      

      Expected results:

      The quay should connect the second Azure storage successfully and work well.

      Additional info:

      The attachment includes the quay-operator pod logs

      Attachments

        Activity

          People

            Unassigned Unassigned
            rhwhu Weihua Hu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: