Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-19901

The kube-apiserver CrashLoopBackOff after enabled auditWebhook

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Undefined Undefined
    • None
    • 4.14
    • HyperShift
    • No
    • Proposed
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      the kube-apiserver CrashLoopBackOff after I enabled  auditWebhook in the hypershift hosted cluster. 
      
      oc logs kube-apiserver-cd8d754c6-cv5gm audit-logs
      tail: cannot open '/var/log/kube-apiserver/audit.log' for reading: No such file or directory
      tail: '/var/log/kube-apiserver/audit.log' has appeared;  following new file
      
      

      Version-Release number of selected component (if applicable):

      4.14.0-0.nightly-2023-09-26-124507
      
      

      How reproducible:

      Always
      
      

      Steps to Reproduce:

      1) On Management Cluster, create a Logging collection httpserver
      1.1) deploy cluster-logging operator
      1.2) create ClusterLogging resource
      cat <<EOF|oc create -f -
      apiVersion: "logging.openshift.io/v1"
      kind: "ClusterLogging"
      metadata:
        name: "instance"
        namespace: openshift-logging
      spec:
        managementState: "Managed"
        logStore:
          type: "elasticsearch"
          elasticsearch:
            nodeCount: 1
            resources:
              limits:
                memory: 2Gi
              requests:
                cpu: 200m
                memory: 2Gi
            storage: {}
            redundancyPolicy: "ZeroRedundancy"
        visualization:
          type: "kibana"
          kibana:
            replicas: 1
        collection:
          type: "vector"
      EOF
      
      1.3) create CLF resource
      
      cat <<EOF |  oc apply -f -
      apiVersion: logging.openshift.io/v1
      kind: ClusterLogForwarder
      metadata:
        name: instance
        namespace: openshift-logging
      spec:
        inputs:
          - name: input-http
            receiver:
              http: 
                format: kubeAPIAudit
                receiverPort: 
                  name: httpserver
                  port: 443
                  targetPort: 8443
        pipelines:
          - name: to-default
            inputRefs:
            - input-http
            outputRefs:
            - default
      EOF
      1.4) make sure the collection pod are running and service httpserver is created
      
      $oc get pods -n openshift-logging
      NAME                                        READY   STATUS    RESTARTS   AGE
      collector-6nk2j                             1/1     Running   0          5h27m
      collector-zx6tt                             1/1     Running   0          5h27m
      $ oc get svc -n openshift-logging
      NAME                               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)     AGE
      httpserver                         ClusterIP   172.30.96.79     <none>        443/TCP     5h27m
      
      2) Enable audit webhook on kube-apiserver,openshift-apiserver,openshift-oauth-apiserver and openshift-ovn-kubernetes pods
      2.1) create auditWebhook secret
      cat <<EOF| tee httpserver.kubeconfig
      apiVersion: v1
      clusters:
      - cluster:
          certificate-authority-data: $(oc extract cm/collector-trusted-ca-bundle --confirm| cat ca-bundle.crt |base64 -w0)
          server: https://httpserver.openshift-logging.svc:443
        name: httpserver-openshift-logging
      contexts:
      - context:
          cluster: httpserver-openshift-logging
        name: httpserver
      current-context: httpserver
      kind: Config
      preferences: {}
      EOF
      oc create secret generic httpserver-secret --from-file=kubeconfig=httpserver.kubeconfig -n clusters-hypershift-ci-23507
      2.2) enable auditWebhook in hostedcontrolplanes.
      oc -n clusters patch hostedclusters.hypershift.openshift.io hypershift-ci-23507 -p='{"spec": {"auditWebhook": {"name":"httpserver-secret"}}}'  --type=merge
      2.3) Make sure the pods kube-apiserver,openshift-apiserver,openshift-oauth-apiserver and openshift-ovn-kubernetes are restarted and running
      3) Make sure auditWebhook can sent logs to Logging collection httpserver
      
      

      Actual results:

      the kube-apiserver CrashLoopBackOff after I enabled  auditwebhook in hypershift hosted cluster. 
      

      Expected results:

      Hypershift-operator can enable auditWebhook for hosted cluster.
      
      

      Additional info:

      That may be  a blocker to https://issues.redhat.com/browse/SDE-3223
      

            [OCPBUGS-19901] The kube-apiserver CrashLoopBackOff after enabled auditWebhook

            Anping Li added a comment -

            Finally, I can enable auditWebook to send logs to Logging httpserver service.

            Anping Li added a comment - Finally, I can enable auditWebook to send logs to Logging httpserver service.

            Anping Li added a comment -

            Anping Li added a comment - I found the case https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-67713

            Anping Li added a comment - - edited

            Do you mean hostedclusters.hypershift.openshift.io? Can you provide a guide to update auditWebhook?

            Anping Li added a comment - - edited Do you mean hostedclusters.hypershift.openshift.io? Can you provide a guide to update auditWebhook?

            the key of the secret seems wrong, should be webhook-kubeconfig 
            kubectl --kubeconfig ~/Downloads/kubeconfig get -nclusters-hypershift-ci-6211 secret httpserver-secret -oyaml
            Also why are you modifying the HCP instead of the hcluster.Spec.AuditWebhook?

            Please reopen if the issue persist when creating the right secret.

             

            Alberto Garcia Lamela added a comment - the key of the secret seems wrong, should be webhook-kubeconfig  kubectl --kubeconfig ~/Downloads/kubeconfig get -nclusters-hypershift-ci-6211 secret httpserver-secret -oyaml Also why are you modifying the HCP instead of the hcluster.Spec.AuditWebhook? Please reopen if the issue persist when creating the right secret.  

            please provide full hypershift dump

            Alberto Garcia Lamela added a comment - please provide full hypershift dump

              Unassigned Unassigned
              rhn-support-anli Anping Li
              Jie Zhao Jie Zhao
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: