Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-19901

The kube-apiserver CrashLoopBackOff after enabled auditWebhook

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Undefined Undefined
    • None
    • 4.14
    • HyperShift
    • No
    • Proposed
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      the kube-apiserver CrashLoopBackOff after I enabled  auditWebhook in the hypershift hosted cluster. 
      
      oc logs kube-apiserver-cd8d754c6-cv5gm audit-logs
      tail: cannot open '/var/log/kube-apiserver/audit.log' for reading: No such file or directory
      tail: '/var/log/kube-apiserver/audit.log' has appeared;  following new file
      
      

      Version-Release number of selected component (if applicable):

      4.14.0-0.nightly-2023-09-26-124507
      
      

      How reproducible:

      Always
      
      

      Steps to Reproduce:

      1) On Management Cluster, create a Logging collection httpserver
      1.1) deploy cluster-logging operator
      1.2) create ClusterLogging resource
      cat <<EOF|oc create -f -
      apiVersion: "logging.openshift.io/v1"
      kind: "ClusterLogging"
      metadata:
        name: "instance"
        namespace: openshift-logging
      spec:
        managementState: "Managed"
        logStore:
          type: "elasticsearch"
          elasticsearch:
            nodeCount: 1
            resources:
              limits:
                memory: 2Gi
              requests:
                cpu: 200m
                memory: 2Gi
            storage: {}
            redundancyPolicy: "ZeroRedundancy"
        visualization:
          type: "kibana"
          kibana:
            replicas: 1
        collection:
          type: "vector"
      EOF
      
      1.3) create CLF resource
      
      cat <<EOF |  oc apply -f -
      apiVersion: logging.openshift.io/v1
      kind: ClusterLogForwarder
      metadata:
        name: instance
        namespace: openshift-logging
      spec:
        inputs:
          - name: input-http
            receiver:
              http: 
                format: kubeAPIAudit
                receiverPort: 
                  name: httpserver
                  port: 443
                  targetPort: 8443
        pipelines:
          - name: to-default
            inputRefs:
            - input-http
            outputRefs:
            - default
      EOF
      1.4) make sure the collection pod are running and service httpserver is created
      
      $oc get pods -n openshift-logging
      NAME                                        READY   STATUS    RESTARTS   AGE
      collector-6nk2j                             1/1     Running   0          5h27m
      collector-zx6tt                             1/1     Running   0          5h27m
      $ oc get svc -n openshift-logging
      NAME                               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)     AGE
      httpserver                         ClusterIP   172.30.96.79     <none>        443/TCP     5h27m
      
      2) Enable audit webhook on kube-apiserver,openshift-apiserver,openshift-oauth-apiserver and openshift-ovn-kubernetes pods
      2.1) create auditWebhook secret
      cat <<EOF| tee httpserver.kubeconfig
      apiVersion: v1
      clusters:
      - cluster:
          certificate-authority-data: $(oc extract cm/collector-trusted-ca-bundle --confirm| cat ca-bundle.crt |base64 -w0)
          server: https://httpserver.openshift-logging.svc:443
        name: httpserver-openshift-logging
      contexts:
      - context:
          cluster: httpserver-openshift-logging
        name: httpserver
      current-context: httpserver
      kind: Config
      preferences: {}
      EOF
      oc create secret generic httpserver-secret --from-file=kubeconfig=httpserver.kubeconfig -n clusters-hypershift-ci-23507
      2.2) enable auditWebhook in hostedcontrolplanes.
      oc -n clusters patch hostedclusters.hypershift.openshift.io hypershift-ci-23507 -p='{"spec": {"auditWebhook": {"name":"httpserver-secret"}}}'  --type=merge
      2.3) Make sure the pods kube-apiserver,openshift-apiserver,openshift-oauth-apiserver and openshift-ovn-kubernetes are restarted and running
      3) Make sure auditWebhook can sent logs to Logging collection httpserver
      
      

      Actual results:

      the kube-apiserver CrashLoopBackOff after I enabled  auditwebhook in hypershift hosted cluster. 
      

      Expected results:

      Hypershift-operator can enable auditWebhook for hosted cluster.
      
      

      Additional info:

      That may be  a blocker to https://issues.redhat.com/browse/SDE-3223
      

            Unassigned Unassigned
            rhn-support-anli Anping Li
            Jie Zhao Jie Zhao
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: