Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-25848

CSR not approved in large scaled AWS cluster with 252 worker nodes

XMLWordPrintable

    • Important
    • No
    • Rejected
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      Not approved csrs are detected in a AWS cluster with 252 worker nodes.
      
      The installation with 252 worker nodes were successful, then we run some performance test, after the test, we run a heath check step to check 'not approved' csrs. There are  some ‘not approved’ csrs. 
      
      2023-12-04 23:29:14,076 [WARNING] There are CSR's that are currently not approved
      2023-12-04 23:29:14,076 [WARNING] Csr's that are not approved: ['csr-5g78p', 'csr-5l9gn', 'csr-5tzq8', 'csr-6cm2l', 'csr-6n2w7', 'csr-8cshp', 'csr-8qlzt', 'csr-9dvrq', 'csr-c9mb5', 'csr-cnr8t', 'csr-fq8lm', 'csr-ftf24', 'csr-gkr72', 'csr-ht8h4', 'csr-jbdqk', 'csr-jgvh4', 'csr-kjzsz', 'csr-l7kwt', 'csr-lj65v', 'csr-lmpwv', 'csr-m8d8z', 'csr-m98pb', 'csr-mhbf7', 'csr-mkdtz', 'csr-mxbbz', 'csr-n4pbq', 'csr-n6pph', 'csr-nnmls', 'csr-nwxcv', 'csr-nzlmm', 'csr-qd2sc', 'csr-qdmh2', 'csr-rhvq5', 'csr-svfvw', 'csr-sxrz6', 'csr-t6szq', 'csr-t9zhm', 'csr-w2dzw', 'csr-wm22r', 'csr-xj592']

      Version-Release number of selected component (if applicable):

      4.15.0-0.nightly-2023-12-02-123536

      How reproducible:

      Only see this once till now

      Steps to Reproduce:

      1. Install AWS cluster with 252 worker nodes
      2. Run a heath check step to check 'not approved' csrs.

      Actual results:

      Not approved csrs are detected

      Expected results:

      Not approved csrs are not detected  

      Additional info:

          

      Code that detects 'not approved' csrs: https://github.com/redhat-chaos/cerberus/blob/main/start_cerberus.py#L394-L406 

      Failed Prow test job: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-qe-ocp-qe-perfscale-ci-main-aws-4.15-nightly-x86-ovnic-node-density-cni-252nodes-220ppn/1731750367294656512 

      Step node-density-cni-252nodes-220ppn-redhat-chaos-cerberus-one-run's log that detected 'not approved' csrs: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-qe-ocp-qe-perfscale-ci-main-aws-4.15-nightly-x86-ovnic-node-density-cni-252nodes-220ppn/1731750367294656512/artifacts/node-density-cni-252nodes-220ppn/redhat-chaos-cerberus-one-run/build-log.txt 

            joelspeed Joel Speed
            rhn-support-qili Qiujie Li
            Zhaohua Sun Zhaohua Sun
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: