Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-7977

Quay 3.12 Operator should not reconcile to update minReplicas for Clair-APP and Mirror when HPA is set to unmanaged

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • quay-v3.12.6
    • quay-v3.12.3
    • quay
    • False
    • None
    • False

      Description:

      This is an issue found in Quay 3.12.3, when deploy Quay with managed HPA, then update the QuayRegistry CR to use unmanaged HPA, Quay Operator will continue to terminate and reset the minReplicas to default value 2 for Mirror and Clair-App HPA, this issue is not existed with Quay APP HPA, pls review this issue.

      Quay: 3.12.3(quay-operator-bundle-container-v3.12.2-3)

      Steps:

      1. Deploy Quay with Quay Operator with managed HPA
      2. Edit the Quay CRD and set the HPA as "managed: false"
      3. Edit the "minReplicas" for Quay, Mirror and Clair HPA to set as customized value(e.g 5)
      4. Check Quay, Mirror and Clair deployments.
      5. The results is Quay Operator will not terminate Quay APP PODS
      6. The results is Quay Operator will terminate Mirror and Clair PODS, then the HPA will provisioning new PODS to meet the minReplicas, but Quay Operator will terminate PODS to set the minReplicas to 2 again and again.
      Quay 3.12.3 Operator continue to terminate the PODS of Mirror and Clair-App PODS:

      oc get pod
NAME                      READY  STATUS    RESTARTS  AGE
quay-operator.v3.12.3-7769ddb5d-9zq6b      1/1   Running    0     112m
quay3123hcpcs-clair-app-7b75b8d464-tg6j2    1/1   Running    0     14m
quay3123hcpcs-clair-app-7b75b8d464-whd7j    1/1   Running    0     14m
quay3123hcpcs-clair-postgres-6544687f48-njmkq  1/1   Running    0     62m
quay3123hcpcs-quay-app-85bfd96dbf-5qn8l     1/1   Running    0     14m
quay3123hcpcs-quay-app-85bfd96dbf-622v8     1/1   Running    0     3m11s
quay3123hcpcs-quay-app-85bfd96dbf-9frc4     1/1   Running    0     3m11s
quay3123hcpcs-quay-app-85bfd96dbf-m2tjd     1/1   Running    0     13m
quay3123hcpcs-quay-app-85bfd96dbf-zkt2p     1/1   Running    0     3m11s
quay3123hcpcs-quay-app-upgrade-g9b6l      0/1   Completed   2     66m
quay3123hcpcs-quay-database-cd9cfbd6d-rmvzx   1/1   Running    0     65m
quay3123hcpcs-quay-mirror-5f68595776-flfn4   1/1   Running    0     14m
quay3123hcpcs-quay-mirror-5f68595776-s9nnr   1/1   Running    0     14m
quay3123hcpcs-quay-mirror-5f68595776-w7dvs   1/1   Terminating  0     56s
quay3123hcpcs-quay-mirror-5f68595776-z5p2v   1/1   Terminating  0     56s
quay3123hcpcs-quay-redis-769697698f-kqg4g    1/1   Running    0     65m

oc get hpa
NAME            REFERENCE               TARGETS                   MINPODS  MAXPODS  REPLICAS  AGE
quay3123hcpcs-clair-app   Deployment/quay3123hcpcs-clair-app   cpu: <unknown>/90%, memory: <unknown>/90%  3     10    2     66m
quay3123hcpcs-quay-app   Deployment/quay3123hcpcs-quay-app   cpu: 5%/90%, memory: 39%/90%        5     20    5     66m
quay3123hcpcs-quay-mirror  Deployment/quay3123hcpcs-quay-mirror  cpu: <unknown>/90%, memory: <unknown>/90%  4     20    2     66m

oc get hpa
NAME                        REFERENCE                              TARGETS                                     MINPODS   MAXPODS   REPLICAS   AGE
quay3123hcpcs-clair-app     Deployment/quay3123hcpcs-clair-app     cpu: <unknown>/90%, memory: <unknown>/90%   3         10        2          69m
quay3123hcpcs-quay-app      Deployment/quay3123hcpcs-quay-app      cpu: 7%/90%, memory: 39%/90%                5         20        5          69m
quay3123hcpcs-quay-mirror   Deployment/quay3123hcpcs-quay-mirror   cpu: 0%/90%, memory: 27%/90%                4         20        4

        1. image-2024-09-23-14-17-08-983.png
          3.34 MB
          luffy zhang
        2. image-2024-12-05-14-35-30-942.png
          3.21 MB
          luffy zhang
        3. image-2024-12-05-14-37-28-622.png
          1.69 MB
          luffy zhang
        4. image-2024-12-05-15-33-21-801.png
          1.61 MB
          luffy zhang
        5. image-2024-12-06-15-35-22-013.png
          2.31 MB
          luffy zhang

              rh-ee-shudeshp Shubhra Jayant Deshpande
              lzha1981 luffy zhang
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: