Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-5627

Quay 3.9.0 was failed to remove site from quay geo replication

XMLWordPrintable

    • False
    • None
    • False

      Description:

      This is an issue found when test "remove site from geo replication", after deploy two quay on different sites with geo replication, using "python -m util.removelocation" to remove site hit error "AttributeError: module 'features' has no attribute 'STORAGE_REPLICATION'", but actually the feature "FEATURE_STORAGE_REPLICATION: true" is enabled on all quay deployments already, pls review this issue.

      Quay:  quay-operator-bundle-container-v3.9.0-114

      The following steps are trying to remove the 2nd site "euwest2":

      oc get pod
      NAME                                                READY   STATUS    RESTARTS       AGE
      quay-operator.v3.9.0-7df7c66b95-7pj6w               1/1     Running   0              42m
      quay390useast2-clair-app-7869567d54-hdml2           1/1     Running   0              106m
      quay390useast2-clair-app-7869567d54-vl8nh           1/1     Running   0              100m
      quay390useast2-clair-postgres-6d6cb47f56-zpvtd      1/1     Running   1 (106m ago)   106m
      quay390useast2-quay-app-69c4b4fb74-pdwdr            1/1     Running   0              54m
      quay390useast2-quay-app-69c4b4fb74-qgf24            1/1     Running   0              42m
      quay390useast2-quay-config-editor-bffbb9474-pj6ms   1/1     Running   0              42m
      quay390useast2-quay-mirror-848f65585c-ckqrx         1/1     Running   0              42m
      quay390useast2-quay-mirror-848f65585c-dpcp5         1/1     Running   0              54m
      
      oc rsh quay390useast2-quay-app-69c4b4fb74-pdwdr
      sh-4.4$ cat conf/stack/config.yaml | grep FEATURE_STORAGE_REPLICATION
      FEATURE_STORAGE_REPLICATION: true
      
      
      sh-4.4$ python -m util.removelocation euwest2
      Traceback (most recent call last):
        File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
          return _run_code(code, main_globals, None,
        File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
          exec(code, run_globals)
        File "/quay-registry/util/removelocation.py", line 61, in <module>
          if not features.STORAGE_REPLICATION:
      AttributeError: module 'features' has no attribute 'STORAGE_REPLICATION' 

      Quay config.yaml:

      ALLOW_PULLS_WITHOUT_STRICT_LOGGING: false
      AUTHENTICATION_TYPE: Database
      BUILDLOGS_REDIS:
        host: 18.191.246.139
        port: 6379
      DATABASE_SECRET_KEY: FHoXX5DaOuLWIgW69j1D5DJzYWo7Un83LVPxrzU2GAAiR2M2QebEzGqeajPQaai9PgFgStKHB29OYitB
      DB_CONNECTION_ARGS:
        autorollback: true
        threadlocals: true
      DB_URI: postgresql://***:***@terraform-20230609034638859400000001.cmqwuswughvh.us-east-2.rds.amazonaws.com:5432/quay
      DEFAULT_TAG_EXPIRATION: 2w
      DISTRIBUTED_STORAGE_CONFIG:
        euwest2:
        - S3Storage
        - host: s3.eu-west-2.amazonaws.com
          s3_access_key: ***
          s3_bucket: quay3900euwest2
          s3_secret_key: ***
          storage_path: /quay390
        useast2:
        - S3Storage
        - host: s3.us-east-2.amazonaws.com
          s3_access_key: ***
          s3_bucket: quay390useast2
          s3_secret_key: ***
          storage_path: /quay390
      DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS:
      - useast2
      - euwest2
      DISTRIBUTED_STORAGE_PREFERENCE:
      - useast2
      - euwest2
      ENTERPRISE_LOGO_URL: /static/img/RH_Logo_Quay_Black_UX-horizontal.svg
      EXTERNAL_TLS_TERMINATION: false
      FEATURE_BUILD_SUPPORT: false
      FEATURE_DIRECT_LOGIN: true
      FEATURE_MAILING: false
      FEATURE_PROXY_CACHE: true
      FEATURE_QUOTA_MANAGEMENT: true
      FEATURE_REPO_MIRROR: true
      FEATURE_SECURITY_NOTIFICATIONS: true
      FEATURE_SECURITY_SCANNER: true
      FEATURE_STORAGE_REPLICATION: true
      PREFERRED_URL_SCHEME: https
      REGISTRY_TITLE: Red Hat Quay
      REGISTRY_TITLE_SHORT: Red Hat Quay
      REPO_MIRROR_INTERVAL: 30
      REPO_MIRROR_TLS_VERIFY: true
      SECRET_KEY: c5M78xdrGuyF9LApCmBqf2OnVmPSXK0E0pnIgK1VQ1RnLY9AZg3eSztlmT7Wntkj001x6fXTJtv7H--K
      SECURITY_SCANNER_INDEXING_INTERVAL: 30
      SECURITY_SCANNER_V4_ENDPOINT: http://quay390useast2-clair-app.quay-enterprise-940.svc.cluster.local
      SECURITY_SCANNER_V4_NAMESPACE_WHITELIST:
      - admin
      SECURITY_SCANNER_V4_PSK: SVNsMmQwMG1DalNGdTdtTlNneVJTR1N2ZGlMTFBtS1g=
      SERVER_HOSTNAME: quaygeo.qe.devcluster.openshift.com
      SETUP_COMPLETE: true
      TAG_EXPIRATION_OPTIONS:
      - 2w
      TEAM_RESYNC_STALE_TIME: 60m
      TESTING: false
      USER_EVENTS_REDIS:
        host: 18.191.246.139
        port: 6379 

      QuayRegistry CR of the 1st site:

      apiVersion: quay.redhat.com/v1
      kind: QuayRegistry
      metadata:
        name: quay390useast2
      spec:
        configBundleSecret: georep-config-bundle
        components:
        - kind: quay
          managed: true
          overrides:
            env:
            - name: QUAY_DISTRIBUTED_STORAGE_PREFERENCE
              value: useast2
            - name: DEBUGLOG
              value: "true"
        - kind: postgres
          managed: false
        - kind: redis
          managed: false
        - kind: objectstorage
          managed: false
        - kind: tls
          managed: false
        - kind: mirror
          managed: true
          overrides:
            env:
            - name: QUAY_DISTRIBUTED_STORAGE_PREFERENCE
              value: useast2 

      QuayRegistry CR of the 2nd site:

      apiVersion: quay.redhat.com/v1
      kind: QuayRegistry
      metadata:
        name: quay390
      spec:
        configBundleSecret: georep-config-bundle
        components:
        - kind: quay
          managed: true
          overrides:
            env:
            - name: QUAY_DISTRIBUTED_STORAGE_PREFERENCE
              value: euwest2
            - name: DEBUGLOG
              value: "true"
        - kind: postgres
          managed: false
        - kind: redis
          managed: false
        - kind: objectstorage
          managed: false
        - kind: tls
          managed: false
        - kind: mirror
          managed: true
          overrides:
            env:
            - name: QUAY_DISTRIBUTED_STORAGE_PREFERENCE
              value: euwest2 

              Unassigned Unassigned
              lzha1981 luffy zhang
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: