Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Done
Priority: Blocker
Fix Version/s: None
Affects Version/s: quay-v3.8.0
Component/s: quay, quay-operator
Labels:
- TestBlocker
- triaged

Blocked:
False
Blocked Reason:
None
Ready:
False

RICE Score:
0

SFDC Cases Links:
SFDC Cases Counter:

Description

Description of problem:

When set up a geo-replication environment by quay3.8.0, quayregistry-quay-app-upgrade pod crashes for error "Exception: Missing storage preference, did you perhaps forget to define QUAY_DISTRIBUTED_STORAGE_PREFERENCE variable?"

Version-Release number of selected component (if applicable):

quay 3.8.0
IIB: quay-operator-bundle-container-v3.8.0-99

How reproducible:

always

Steps to Reproduce:
1. follow Chapter 14. Geo-replication to set up a geo-replication cluster with quay3.8.0

Actual results:

quayregistry-quay-app-upgrade pod crashes for error "Exception: Missing storage preference, did you perhaps forget to define QUAY_DISTRIBUTED_STORAGE_PREFERENCE variable?"

$ oc get pod
NAME                                               READY   STATUS                  RESTARTS       AGE
quay-operator.v3.8.0-7cd9ddddfd-jjfdq              1/1     Running                 0              82m
......
quayregistry-quay-app-upgrade-7hm7d                0/1     Error                   3 (29s ago)    53s
quayregistry-quay-config-editor-85d8986f7c-t5xtw   1/1     Running                 0              51m
quayregistry-quay-mirror-5c55987865-q8tst          0/1     Init:CrashLoopBackOff   10 (42s ago)   50m
quayregistry-quay-mirror-5c55987865-v4qbx          0/1     Init:CrashLoopBackOff   10 (58s ago)   51m

$oc logs quayregistry-quay-app-upgrade-7hm7d
   __   __
  /  \ /  \     ______   _    _     __   __   __
 / /\ / /\ \   /  __  \ | |  | |   /  \  \ \ / /
/ /  / /  \ \  | |  | | | |  | |  / /\ \  \   /
\ \  \ \  / /  | |__| | | |__| | / ____ \  | |
 \ \/ \ \/ /   \_  ___/  \____/ /_/    \_\ |_|
  \__/ \__/      \ \__
                  \___\ by Red Hat
 Build, Store, and Distribute your Containers
Startup timestamp: 
Tue Oct 25 06:33:40 UTC 2022
Entering migration mode to version: head
Traceback (most recent call last):
  File "/usr/local/bin/alembic", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/alembic/config.py", line 575, in main
    CommandLine(prog=prog).main(argv=argv)
  File "/usr/local/lib/python3.9/site-packages/alembic/config.py", line 569, in main
    self.run_cmd(cfg, options)
  File "/usr/local/lib/python3.9/site-packages/alembic/config.py", line 546, in run_cmd
    fn(
  File "/usr/local/lib/python3.9/site-packages/alembic/command.py", line 298, in upgrade
    script.run_env()
  File "/usr/local/lib/python3.9/site-packages/alembic/script/base.py", line 489, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 98, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.9/site-packages/alembic/util/compat.py", line 173, in load_module_py
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "data/migrations/env.py", line 13, in <module>
    from app import app
  File "/quay-registry/app.py", line 327, in <module>
    raise Exception(
Exception: Missing storage preference, did you perhaps forget to define QUAY_DISTRIBUTED_STORAGE_PREFERENCE variable?

Expected results:

geo-replication feature should work in quay 3.8.0

Additional info:

config.yaml

FEATURE_STORAGE_REPLICATION: true
SERVER_HOSTNAME: quay380.trafficmanager.net 
DB_CONNECTION_ARGS:
  autorollback: true
  threadlocals: true
DB_URI: postgresql://whuquay:Am...2@whupsql1.postgres.database.azure.com:5432/quayregistry_quay_database
BUILDLOGS_REDIS:
  host: whu-ipv4.eastus.cloudapp.azure.com 
  password: r..w 
  port: 6379
USER_EVENTS_REDIS:
  host: whu-ipv4.eastus.cloudapp.azure.com 
  password: r...w
  port: 6379
DISTRIBUTED_STORAGE_CONFIG:
  reg1storage:
    - AzureStorage
    - azure_account_name: whusc21 
      azure_account_key: xO0bAGSGhduuQCuObY8ynZVSXYtFh6TaTk3iNerwQG5Ux7dV+a9tpaYVI4DXmjhGnZs8JuH8Szgs+AStMHghnA==
      azure_container: whusc21container 
      sas_token: ?sv=2021-06-08&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2022-10-25T12:49:59Z&st=2022-10-25T04:49:59Z&spr=https&sig=d1%2FiX0HnvVUQuUUYU66kBPuejXAD2OcPgW%2FO1PfUR1I%3D
      storage_path: /quaydata
  reg2storage:
    - AzureStorage
    - azure_account_name: whusc21 
      azure_account_key: xO0bAGSGhduuQCuObY8ynZVSXYtFh6TaTk3iNerwQG5Ux7dV+a9tpaYVI4DXmjhGnZs8JuH8Szgs+AStMHghnA==
      azure_container: whusc21container 
      sas_token: ?sv=2021-06-08&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2022-10-25T12:49:59Z&st=2022-10-25T04:49:59Z&spr=https&sig=d1%2FiX0HnvVUQuUUYU66kBPuejXAD2OcPgW%2FO1PfUR1I%3D
      storage_path: /quaydata
DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS:
  - reg1storage
  - reg2storage
DISTRIBUTED_STORAGE_PREFERENCE:
  - reg1storage
  - reg2storage
SUPER_USERS:
  - whuquay
  - whutest
FEATURE_UI_V2: True
FEATURE_LISTEN_IP_VERSION: IPv4
FEATURE_SUPERUSERS_FULL_ACCESS: True

The QuayRegistry definition

apiVersion: quay.redhat.com/v1
kind: QuayRegistry
metadata:
  name: quayregistry
  namespace: quay-enterprise 
spec:
  configBundleSecret: config-bundle-secret 
  components:
    - kind: clair
      managed: true 
    - kind: postgres
      managed: false 
    - kind: objectstorage
      managed: false 
    - kind: redis
      managed: false 
    - kind: horizontalpodautoscaler
      managed: true 
    - kind: route
      managed: true 
    - kind: mirror
      managed: true 
    - kind: monitoring
      managed: false 
    - kind: quay
      managed: true
      overrides:
        env:    
        - name: QUAY_DISTRIBUTED_STORAGE_PREFERENCE
          value: reg1storage

$ oc get job quayregistry-quay-app-upgrade -o yaml
apiVersion: batch/v1
kind: Job
metadata:
  annotations:
    quay-buildmanager-hostname: ""
    quay-component: base
    quay-operator-service-endpoint: http://quay-operator.quay-enterprise.svc.cluster.local:7071
    quay-registry-hostname: quay380.trafficmanager.net
  creationTimestamp: "2022-10-25T06:38:58Z"
  generation: 1
  labels:
    quay-component: quay-app-upgrade
    quay-operator/quayregistry: quayregistry
  managedFields:
     ......
  name: quayregistry-quay-app-upgrade
  namespace: quay-enterprise
  ownerReferences:
  - apiVersion: quay.redhat.com/v1
    kind: QuayRegistry
    name: quayregistry
    uid: fd65b6f8-33b5-4a43-9bcf-f02997e46913
  resourceVersion: "81071"
  uid: 1e6ceecb-4bbb-4994-918b-c05920ebe564
spec:
  backoffLimit: 6
  completionMode: NonIndexed
  completions: 1
  parallelism: 1
  selector:
    matchLabels:
      controller-uid: 1e6ceecb-4bbb-4994-918b-c05920ebe564
  suspend: false
  template:
    metadata:
      annotations:
        quay-buildmanager-hostname: ""
        quay-operator-service-endpoint: http://quay-operator.quay-enterprise.svc.cluster.local:7071
        quay-registry-hostname: quay380.trafficmanager.net
      creationTimestamp: null
      labels:
        controller-uid: 1e6ceecb-4bbb-4994-918b-c05920ebe564
        job-name: quayregistry-quay-app-upgrade
        quay-component: quay-app-upgrade
        quay-operator/quayregistry: quayregistry
      name: quay-app-upgrade
    spec:
      containers:
      - args:
        - migrate
        - head
        env:
        - name: QE_K8S_CONFIG_SECRET
          value: quayregistry-quay-config-secret-267km67c62
        - name: QE_K8S_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: DEBUGLOG
          value: "false"
        - name: WORKER_COUNT_WEB
          value: "4"
        - name: WORKER_COUNT_SECSCAN
          value: "2"
        - name: WORKER_COUNT_REGISTRY
          value: "8"
        image: registry.redhat.io/quay/quay-rhel8@sha256:a88d9da2e64785522b1040791f28505eca1416da804865b13dc83b97a5d0faea
        imagePullPolicy: IfNotPresent
        name: quay-app-upgrade
        ports:
        - containerPort: 8443
          protocol: TCP
        - containerPort: 8080
          protocol: TCP
        - containerPort: 8081
          protocol: TCP
        resources:
          limits:
            cpu: "2"
            memory: 8Gi
          requests:
            cpu: "1"
            memory: 3Gi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /conf/stack
          name: config
        - mountPath: /conf/stack/extra_ca_certs
          name: extra-ca-certs
          readOnly: true
      dnsPolicy: ClusterFirst
      restartPolicy: OnFailure
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: quayregistry-quay-app
      serviceAccountName: quayregistry-quay-app
      terminationGracePeriodSeconds: 30
      volumes:
      - name: config
        secret:
          defaultMode: 420
          secretName: quayregistry-quay-config-secret-267km67c62
      - name: extra-ca-certs
        projected:
          defaultMode: 420
          sources:
          - configMap:
              name: quayregistry-cluster-service-ca
          - configMap:
              name: quayregistry-cluster-trusted-ca
          - secret:
              name: quayregistry-extra-ca-certs-46f8b28mk5
status:
  active: 1
  startTime: "2022-10-25T06:38:58Z"

Found some new codes were committed into quay 3.8.0 app.py file

 # Check if georeplication is turned on and whether env. variables exist:
if os.environ.get("QUAY_DISTRIBUTED_STORAGE_PREFERENCE") is None and app.config.get(
    "FEATURE_STORAGE_REPLICATION", False
):
    raise Exception(
        "Missing storage preference, did you perhaps forget to define QUAY_DISTRIBUTED_STORAGE_PREFERENCE variable?"
    )

Attachments

Issue Links

links to

quay/quay-operator#739: upgrade: Add Quay env overrides to upgrade pod (PROJQUAY-4660)

quay/quay-operator#740: [redhat-3.8] upgrade: Add Quay env overrides to upgrade pod (PROJQUAY-4660)

mentioned on

Merge request - Updated US source to: b4d59a8 upgrade: Add Quay env overrides to upgrade pod (PROJQUAY-4660)

Activity

People

Assignee:: Jonathan King

Reporter:: Weihua Hu

Votes:: 0 Vote for this issue

Watchers:: 6 Start watching this issue

Dates

Created:: 2022/10/25 6:49 AM

Updated:: 2022/11/07 4:07 PM

Resolved:: 2022/11/07 4:07 PM