Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-16363

Api-checks-pod crash as got a null pod value on a aws-ipi-ovn-hypershift-mce-mgmt cluster

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Normal Normal
    • None
    • 4.13.z
    • Compliance Operator
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • 5
    • Important
    • No
    • None
    • None
    • OSDOCS Sprint 242
    • 1
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

       

      Api-checks-pod crash as getting a null pod value on a aws-ipi-ovn-hypershift-mce-mgmt cluster:
      $ oc logs pod/hypershift-cisk57aw88gry-api-checks-pod --all-containers 
      …
      Fetching URI: '/api/v1/namespaces/clusters-79136a1bdb84b3c13217/pods?labelSelector=app%3Dkube-controller-manager'
      FATAL:Error fetching resources: couldn't filter '{"kind":"PodList","apiVersion":"v1","metadata":{"resourceVersion":"188114"},"items":[]}
      ': cannot iterate over: null
      Error from server (BadRequest): container "log-collector" in pod "hypershift-cisk57aw88gry-api-checks-pod" is waiting to start: PodInitializing
      

       

      Version-Release number of selected component (if applicable):

      registry.ci.openshift.org/ocp/release:4.14.0-0.nightly-2023-07-11-092038

      Version-Release number of selected component (if applicable):

      4.13.5

      How reproducible:

       Always

      Steps to Reproduce:

       

      1. Install compliance operator 
      2. Create a tailoredprofile:
      $ oc get hostedcluster -A
      NAMESPACE       NAME                   VERSION   KUBECONFIG                              PROGRESS    AVAILABLE   PROGRESSING   MESSAGE
      local-cluster   79136a1bdb84b3c13217   4.13.5    79136a1bdb84b3c13217-admin-kubeconfig   Completed   True        False         The hosted control plane is available
       
      $ cat tp.json
      {
          "kind": "List",
          "apiVersion": "v1",
          "metadata": {},
          "items": [
              {
                  "apiVersion": "compliance.openshift.io/v1alpha1",
                  "kind": "TailoredProfile",
                  "metadata": {
                      "name": "hypershift-cisk57aw88gry",
                      "namespace": "openshift-compliance"
                  },
                  "spec": {
                      "description": "This profile test required rules",
                      "extends": "ocp4-cis",
                      "setValues": [
                          {
                              "name": "ocp4-hypershift-cluster",
                              "rationale": "This value is used for HyperShift version detection",
                              "value": "79136a1bdb84b3c13217"
                          }
                      ],
                      "title": "My little profile"
                  }
              }
          ]
      }
      $ oc apply -f /tmp/e2e-test-compliance-b5fpn5pl8x-mbrtr-ut2ihk3kktisc-config.json
      tailoredprofile.compliance.openshift.io/hypershift-cisk57aw88gry created
      $ oc get tp
      NAME                       STATE
      hypershift-cisk57aw88gry   READY
      
      3. Create a ssb:
      $ oc compliance bind -N test -S default tailoredprofile/hypershift-cisk57aw88gry
      Creating ScanSettingBinding test
      Actual result:
      The pod hypershift-cisk57aw88gry-api-checks-pod crash.The compliancesuite will stuck at Running status.
      $ oc get pod
      NAME                                              READY   STATUS                  RESTARTS       AGE
      compliance-operator-68577cd558-wj6wg              1/1     Running                 1 (104m ago)   104m
      hypershift-cisk57aw88gry-api-checks-pod           0/2     Init:CrashLoopBackOff   10 (42s ago)   28m
      hypershift-cisk57aw88gry-rs-7866c6b8c9-5qfbk      1/1     Running                 0              28m
      ocp4-openshift-compliance-pp-5f5f567b85-hz5fx     1/1     Running                 0              104m
      rhcos4-openshift-compliance-pp-5c445fd5f5-xrhxx   1/1     Running                 0              104m
      

      Actual results:

       

      The pod hypershift-cisk57aw88gry-api-checks-pod crash.The compliancesuite will stuck at Running status.
      $ oc get pod
      NAME                                              READY   STATUS                  RESTARTS       AGE
      compliance-operator-68577cd558-wj6wg              1/1     Running                 1 (104m ago)   104m
      hypershift-cisk57aw88gry-api-checks-pod           0/2     Init:CrashLoopBackOff   10 (42s ago)   28m
      hypershift-cisk57aw88gry-rs-7866c6b8c9-5qfbk      1/1     Running                 0              28m
      ocp4-openshift-compliance-pp-5f5f567b85-hz5fx     1/1     Running                 0              104m
      rhcos4-openshift-compliance-pp-5c445fd5f5-xrhxx   1/1     Running                 0              104m
      

       

      Expected results:

      The pod hypershift-cisk57aw88gry-api-checks-pod should not crash.The compliancesuite should return COMPLIANT/NON-COMPLIANT soon.

      Additional info:

       

      $ oc get ns -l hypershift.openshift.io/hosted-control-plane=true
      NAME                                 STATUS   AGE
      local-cluster-79136a1bdb84b3c13217   Active   4h30m
      $ oc get ns local-cluster-79136a1bdb84b3c13217 --show-labels
      NAME                                 STATUS   AGE     LABELS
      local-cluster-79136a1bdb84b3c13217   Active   4h27m   hypershift.openshift.io/hosted-control-plane=true,hypershift.openshift.io/monitoring=true,kubernetes.io/metadata.name=local-cluster-79136a1bdb84b3c13217,pod-security.kubernetes.io/audit-version=v1.24,pod-security.kubernetes.io/audit=privileged,pod-security.kubernetes.io/warn-version=v1.24,pod-security.kubernetes.io/warn=privileged
      $ oc get pod -n local-cluster-79136a1bdb84b3c13217 -l app=kube-controller-manager
      NAME                                       READY   STATUS    RESTARTS   AGE
      kube-controller-manager-668b6d8549-rzln9   2/2     Running   0          4h19m
      

              antaylor@redhat.com Andrew Taylor
              xiyuan@redhat.com Xiaojie Yuan
              None
              None
              Xiaojie Yuan Xiaojie Yuan
              None
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: