Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-3364

Quay 3.7.0 Operator can't scale up quay, mirror and clair

XMLWordPrintable

    • False
    • None
    • False

      Description:

      This is an issue found of Quay 3.7.0 Operator, when add env variable to scale down the quay, clair, mirror POD to 0, later scale up the quay, clair. mirror deployment, hit error message "invalid overrides: cannot override replicas with managed HPA""

      Quay image:  quay-operator-bundle-container-v3.7.0-24

      Quay OperatorPOD logs:

      apiVersion: quay.redhat.com/v1
kind: QuayRegistry
metadata:
  creationTimestamp: '2022-03-08T02:51:01Z'
  finalizers:
    - quay-operator/finalizer
  generation: 7
  managedFields:
    - apiVersion: quay.redhat.com/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:spec':
          .: {}
          'f:components': {}
          'f:configBundleSecret': {}
      manager: Mozilla
      operation: Update
      time: '2022-03-08T02:51:01Z'
    - apiVersion: quay.redhat.com/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:finalizers':
            .: {}
            'v:"quay-operator/finalizer"': {}
        'f:spec':
          'f:configBundleSecret': {}
        'f:status':
          .: {}
          'f:conditions': {}
          'f:configEditorCredentialsSecret': {}
          'f:configEditorEndpoint': {}
          'f:currentVersion': {}
          'f:lastUpdated': {}
          'f:registryEndpoint': {}
      manager: manager
      operation: Update
      time: '2022-03-08T02:51:57Z'
  name: demo370
  namespace: quay370-3055
  resourceVersion: '178790385'
  uid: bbdd92ff-bb79-4046-ac6d-a7f0a8468fe1
spec:
  components:
    - kind: base
      managed: true
      overrides:
        env:
          - name: DEBUGLOG
            value: 'true'
          - name: WORKER_COUNT_WEB
            value: '20'
        replicas: 2
    - kind: clair
      managed: true
      overrides:
        replicas: 2
    - kind: postgres
      managed: true
    - kind: objectstorage
      managed: true
    - kind: redis
      managed: true
    - kind: horizontalpodautoscaler
      managed: true
    - kind: route
      managed: true
    - kind: mirror
      managed: true
      overrides:
        replicas: 2
    - kind: monitoring
      managed: false
    - kind: tls
      managed: true
    - kind: clairpostgres
      managed: true
  configBundleSecret: demo370-config-bundle-ftqtg
status:
  conditions:
    - lastTransitionTime: '2022-03-08T02:51:08Z'
      lastUpdateTime: '2022-03-08T10:17:50Z'
      message: Horizontal pod autoscaler found
      reason: ComponentReady
      status: 'True'
      type: ComponentHPAReady
    - lastTransitionTime: '2022-03-08T02:51:08Z'
      lastUpdateTime: '2022-03-08T10:17:50Z'
      message: Route admitted
      reason: ComponentReady
      status: 'True'
      type: ComponentRouteReady
    - lastTransitionTime: '2022-03-08T02:51:01Z'
      lastUpdateTime: '2022-03-08T10:17:50Z'
      message: Monitoring not managed by the operator
      reason: ComponentNotManaged
      status: 'True'
      type: ComponentMonitoringReady
    - lastTransitionTime: '2022-03-08T02:51:45Z'
      lastUpdateTime: '2022-03-08T10:17:50Z'
      message: Deployment demo370-quay-database healthy
      reason: ComponentReady
      status: 'True'
      type: ComponentPostgresReady
    - lastTransitionTime: '2022-03-08T02:51:25Z'
      lastUpdateTime: '2022-03-08T10:17:50Z'
      message: Object bucket claim bound
      reason: ComponentReady
      status: 'True'
      type: ComponentObjectStorageReady
    - lastTransitionTime: '2022-03-08T02:51:45Z'
      lastUpdateTime: '2022-03-08T10:17:50Z'
      message: Clair component healthy
      reason: ComponentReady
      status: 'True'
      type: ComponentClairReady
    - lastTransitionTime: '2022-03-08T02:51:25Z'
      lastUpdateTime: '2022-03-08T10:17:50Z'
      message: ClairPostgres component healthy
      reason: ComponentReady
      status: 'True'
      type: ComponentClairPostgresReady
    - lastTransitionTime: '2022-03-08T02:51:01Z'
      lastUpdateTime: '2022-03-08T10:17:50Z'
      message: Using cluster wildcard certs
      reason: ComponentReady
      status: 'True'
      type: ComponentTLSReady
    - lastTransitionTime: '2022-03-08T02:51:25Z'
      lastUpdateTime: '2022-03-08T10:17:50Z'
      message: Deployment demo370-quay-redis healthy
      reason: ComponentReady
      status: 'True'
      type: ComponentRedisReady
    - lastTransitionTime: '2022-03-08T02:54:00Z'
      lastUpdateTime: '2022-03-08T10:17:50Z'
      message: Base component healthy
      reason: ComponentReady
      status: 'True'
      type: ComponentBaseReady
    - lastTransitionTime: '2022-03-08T02:54:00Z'
      lastUpdateTime: '2022-03-08T10:17:50Z'
      message: Deployment demo370-quay-mirror healthy
      reason: ComponentReady
      status: 'True'
      type: ComponentMirrorReady
    - lastTransitionTime: '2022-03-08T02:54:00Z'
      lastUpdateTime: '2022-03-08T10:17:50Z'
      message: All components reporting as healthy
      reason: HealthChecksPassing
      status: 'True'
      type: Available
    - lastTransitionTime: '2022-03-08T02:51:45Z'
      lastUpdateTime: '2022-03-08T02:51:45Z'
      message: All registry components created
      reason: ComponentsCreationSuccess
      status: 'True'
      type: ComponentsCreated
    - lastTransitionTime: '2022-03-08T10:17:50Z'
      lastUpdateTime: '2022-03-08T10:17:50Z'
      message: 'invalid overrides: cannot override replicas with managed HPA'
      reason: ComponentOverrideInvalid
      status: 'True'
      type: RolloutBlocked
  configEditorCredentialsSecret: demo370-quay-config-editor-credentials-hkm7hgb94g
  configEditorEndpoint: >-
    https://demo370-quay-config-editor-quay370-3055.apps.quay-perf-796.perfscale.devcluster.openshift.com
  currentVersion: 3.7.0
  lastUpdated: '2022-03-08 10:17:50.120704783 +0000 UTC'
  registryEndpoint: >-
    https://demo370-quay-quay370-3055.apps.quay-perf-796.perfscale.devcluster.openshift.com

      Steps:

      1. Deploy Quay Operator under single OCP namespace
      2. Deploy Quay with Quay Operator, choose to all managed component except monitoring component
      3. After quayregistry in healthy status, add override env variables to quay, mirror, clair, set replicas to 0
      4. wait until the POD number of quay,mirror,clair is 0
      5. wait for more than 30 mins
      6. Update Quayregistry CR to set override env variables, now set the replicas of quay,mirorr,clair to 2
      7. Wait Quay Operator to reconcile the change

      Expected Results:

      Quay Operator should set the replicas of quay,clair,mirror deployments to 2, and new PODs are provisioning and come to ready status.

      Actual Results:

      Quay Operator can't reconcile the change, report error message "invalid overrides: cannot override replicas with managed HPA""

        1. quay370_operator_pod.logs
          6.91 MB

              lzha1981 luffy zhang
              lzha1981 luffy zhang
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: