Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-77059

SR-IOV webhook pod doesn't show debug logs

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • 4.18
    • Networking / SR-IOV
    • None
    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      SR-IOV webhook pod doesn't show debug logs even if we set SriovOperatorConfig.spec.logLevel=2.

       

      [root@util ~]# oc get   SriovOperatorConfig default -o yaml |grep logLevel
        logLevel: 2
      
      [root@util ~]# oc logs operator-webhook-lsrjt -n openshift-sriov-network-operator
      2026-02-19T08:24:01.885560158Z    INFO    sriov-network-operator-webhook    cobra/command.go:987    Run sriov-network-operator-webhook
      2026-02-19T08:24:01.891874181Z    INFO    sriov-network-operator-webhook    runtime/asm_amd64.s:1695    start server
      
        ===> There is no debug logs

       

      Version-Release number of selected component (if applicable):

      I found this issue on OCP 4.18.
      This is didn't occur on OCP 4.14.

      How reproducible:

      Steps to Reproduce:{}

      1. Deploy OCP 4.18
      2. Install SRIOV operator
      3. Create SriovOperatorConfig with enableOperatorWebhook=true and logLevel=2
        apiVersion: sriovnetwork.openshift.io/v1
        kind: SriovOperatorConfig
        metadata:
          name: default
          namespace: openshift-sriov-network-operator
        spec:
          disableDrain: false
          enableInjector: true
          enableOperatorWebhook: true
          logLevel: 2 
      1. Try to create a wrong SriovNetworkNodePolicy, and it failed. This fine.
        [root@util ~]# cat sriov-policy.yaml 
        apiVersion: sriovnetwork.openshift.io/v1
        kind: SriovNetworkNodePolicy
        metadata:
          name: test
          namespace: openshift-sriov-network-operator
        spec:
          resourceName: test
          nodeSelector:
            node-role.kubernetes.io/worker: ""
          numVfs: 4
          nicSelector:
            vendor: "15b3"
            deviceID: "101d"
        
        [root@util ~]# oc apply -f   sriov-policy.yaml 
        Error from server (no supported NIC is selected by the nicSelector in CR test): error when creating "sriov-policy.yaml": admission webhook "operator-webhook.sriovnetwork.openshift.io" denied the request: no supported NIC is selected by the nicSelector in CR test
         
      1. Even though the creation of SriovNetworkNodePolicy failed due to webhook validation, webhook pods doesn't show any debug logs. This makes troubleshooting quite difficult:
        [root@util ~]# oc logs operator-webhook-lsrjt -n openshift-sriov-network-operator 2026-02-19T08:24:01.885560158Z    INFO    sriov-network-operator-webhook    cobra/command.go:987    Run sriov-network-operator-webhook 2026-02-19T08:24:01.891874181Z    INFO    sriov-network-operator-webhook    runtime/asm_amd64.s:1695    start server  

      Actual results:

      Webhook pods doesn't show any debug logs.
      It's quite difficult to troubleshoot SriovNetworkNodePolicy validation error without webhook debug logs.

      Expected results:

      Webhook pods doesn't shows debug logs.

      Workaround:

      Debug logs is emitted by adding --zap-log-level=2 option to the webhook pod

       

      1. Stop reconciliation 
      # oc -n openshift-sriov-network-operator scale deploy/sriov-network-operator --replicas=0
       
      2. add --zap-log-level=2 option to the webhook pods
      # oc -n openshift-sriov-network-operator patch ds operator-webhook --type='json' -p='[ {"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--zap-log-level=2"} ]'
       
      3. webhook pods shows debug logs
      # oc logs operator-webhook-zrjjx|cut -c 1-150 
      2026-02-19T08:41:02.230541052Z    INFO    sriov-network-operator-webhook    cobra/command.go:987    Run sriov-network-operator-webhook 
      2026-02-19T08:41:02.236942821Z    INFO    sriov-network-operator-webhook    runtime/asm_amd64.s:1695    start server 
      2026-02-19T08:41:04.739337348Z    LEVEL(-2)    serve    webhook/start.go:127    handling request    {"request-body": "{\"kind\":\"AdmissionReview\",\"apiVersion\":\" 
      2026-02-19T08:41:04.739571222Z    LEVEL(-2)    webhook/start.go:99    validating custom resource 
      2026-02-19T08:41:04.739686491Z    LEVEL(-2)    webhook/webhook.go:66    validateSriovNetworkNodePolicy    {"object": {"kind":"SriovNetworkNodePolicy","apiVersion" 
      2026-02-19T08:41:04.748614613Z    LEVEL(-2)    webhook/validate.go:284    validatePolicyForNodeState(): validate policy for node    {"policy-name": "test", "node- 
      2026-02-19T08:41:04.748651163Z    LEVEL(-2)    webhook/validate.go:284    validatePolicyForNodeState(): validate policy for node    {"policy-name": "test", "node- 
      2026-02-19T08:41:04.748656022Z    LEVEL(-2)    webhook/validate.go:284    validatePolicyForNodeState(): validate policy for node    {"policy-name": "test", "node- 
      2026-02-19T08:41:04.748745491Z    LEVEL(-2)    serve    webhook/start.go:127    sending response    {"response": "{\"kind\":\"AdmissionReview\",\"apiVersion\":\"admi
      

       

      Additional info:

      The issue didn't occur on OCP 4.14.
      I suppose that the following patch changed logging library, and now --zap-log-level=2 option is required: 

      https://github.com/openshift/sriov-network-operator/commit/2542dde8c8d98ee938d4be39b48cab4868a6468e

              bnemeth@redhat.com Balazs Nemeth
              rhn-support-yatanaka Yamato Tanaka
              Zhiqiang Fang Zhiqiang Fang
              None
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: