SR-IOV FEC: Failed to apply sriovfecnodeconfig

NAME CONFIGURED
worker-0 Failed

Spec:
  Physical Functions:
    Bb Dev Config:
      acc100:
        downlink4G:
          aqDepthLog2:         4
          Num Aqs Per Groups:  16
          Num Queue Groups:    2
        downlink5G:
          aqDepthLog2:         4
          Num Aqs Per Groups:  16
          Num Queue Groups:    2
        Max Queue Size:        1024
        Num Vf Bundles:        2
        uplink4G:
          aqDepthLog2:         4
          Num Aqs Per Groups:  16
          Num Queue Groups:    2
        uplink5G:
          aqDepthLog2:         4
          Num Aqs Per Groups:  16
          Num Queue Groups:    2
    Pci Address:               0000:af:00.0
    Pf Driver:                 vfio-pci
    Vf Amount:                 2
    Vf Driver:                 vfio-pci
Status:
  Conditions:
    Last Transition Time:  2024-03-26T14:08:18Z
    Message:               exit status 255
    Observed Generation:   7
    Reason:                Failed
    Status:                False
    Type:                  Configured
  Inventory:
    Sriov Accelerators:
      Device ID:              0d5c
      Driver:                 vfio-pci
      Max Virtual Functions:  16
      Pci Address:            0000:af:00.0
      Vendor ID:              8086
      Virtual Functions: 

{"file":"/workspace-go/pkg/daemon/node_management.go:64","func":"github.com/intel-collab/applications.orchestration.operators.sriov-fec-operator/pkg/daemon.(*NodeConfigurator).isDeviceBoundToDriver","level":"info","msg":"device is bound to driver","path":"/sys/bus/pci/devices/0000:af:00.0/driver","time":"2024-03-26T14:17:01Z"}
{"driver":"/sys/bus/pci/drivers/vfio-pci","file":"/workspace-go/pkg/daemon/node_management.go:81","func":"github.com/intel-collab/applications.orchestration.operators.sriov-fec-operator/pkg/daemon.(*NodeConfigurator).unbindDeviceFromDriver","level":"info","msg":"driver to unbound device from","pciAddress":"0000:af:00.0","time":"2024-03-26T14:17:01Z"}
{"file":"/workspace-go/pkg/daemon/node_management.go:116","func":"github.com/intel-collab/applications.orchestration.operators.sriov-fec-operator/pkg/daemon.(*NodeConfigurator).bindDeviceToDriver","level":"info","msg":"device's driver_override path","path":"/sys/bus/pci/devices/0000:af:00.0/driver_override","time":"2024-03-26T14:17:01Z"}
{"file":"/workspace-go/pkg/daemon/node_management.go:123","func":"github.com/intel-collab/applications.orchestration.operators.sriov-fec-operator/pkg/daemon.(*NodeConfigurator).bindDeviceToDriver","level":"info","msg":"driver bind path","path":"/sys/bus/pci/drivers/vfio-pci/bind","time":"2024-03-26T14:17:01Z"}
Failed to obtain reader, failed to marshal fields to JSON, json: unsupported type: func() error
{"file":"/workspace-go/pkg/daemon/common.go:42","func":"github.com/intel-collab/applications.orchestration.operators.sriov-fec-operator/pkg/daemon.execAndSuppress","level":"info","msg":"commands output","output":"0000:af:00.0 @04 0006\n","time":"2024-03-26T14:17:01Z"}
{"file":"/workspace-go/pkg/daemon/bbdevconfig_ini_generator.go:109","func":"github.com/intel-collab/applications.orchestration.operators.sriov-fec-operator/pkg/daemon.logIniFile","generated BBDevConfig":"[MODE]\npf_mode_en = 0\n\n[VFBUNDLES]\nnum_vf_bundles = 2\n\n[MAXQSIZE]\nmax_queue_size = 1024\n\n[QUL4G]\nnum_qgroups        = 2\nnum_aqs_per_groups = 16\naq_depth_log2      = 4\n\n[QDL4G]\nnum_qgroups        = 2\nnum_aqs_per_groups = 16\naq_depth_log2      = 4\n\n[QUL5G]\nnum_qgroups        = 2\nnum_aqs_per_groups = 16\naq_depth_log2      = 4\n\n[QDL5G]\nnum_qgroups        = 2\nnum_aqs_per_groups = 16\naq_depth_log2      = 4\n","level":"info","msg":"logIniFile","time":"2024-03-26T14:17:01Z"}
{"file":"/workspace-go/pkg/daemon/bbdevconfig.go:104","func":"github.com/intel-collab/applications.orchestration.operators.sriov-fec-operator/pkg/daemon.(*pfBBConfigController).initializePfBBConfig","level":"info","msg":"pf-bb-config file path is : /sriov_workdir/pf_bb_config","time":"2024-03-26T14:17:01Z"}
Failed to obtain reader, failed to marshal fields to JSON, json: unsupported type: func() error
{"action":"Evicted","file":"/workspace-go/pkg/common/drainhelper/drainhelper.go:107","func":"github.com/intel-collab/applications.orchestration.operators.sriov-fec-operator/pkg/common/drainhelper.NewDrainHelper.func1","level":"info","msg":"pod evicted or deleted","pod":"router-default-79fd6dfc67-hlphc/openshift-ingress","time":"2024-03-26T14:17:01Z"}
{"file":"/workspace-go/pkg/common/drainhelper/drainhelper.go:272","func":"github.com/intel-collab/applications.orchestration.operators.sriov-fec-operator/pkg/common/drainhelper.(*DrainHelper).cordonAndDrain","level":"info","msg":"node drained","time":"2024-03-26T14:17:01Z"}
{"file":"/workspace-go/pkg/common/drainhelper/drainhelper.go:201","func":"github.com/intel-collab/applications.orchestration.operators.sriov-fec-operator/pkg/common/drainhelper.(*DrainHelper).Run.func2","level":"info","msg":"worker function - start","time":"2024-03-26T14:17:01Z"}
{"cmd":["/sriov_workdir/pf_bb_config","ACC100","-c","/tmp/0000:af:00.0.ini","-v","02bddbbf-bbb0-4d79-886b-91bad3fbb510","-p","0000:af:00.0"],"error":"exit status 255","file":"/workspace-go/pkg/daemon/common.go:36","func":"github.com/intel-collab/applications.orchestration.operators.sriov-fec-operator/pkg/daemon.execAndSuppress","level":"error","msg":"failed to execute command","output":"== pf_bb_config Version v23.11-0-gf9206ad ==\n","time":"2024-03-26T14:17:02Z"}
{"error":"exit status 255","file":"/workspace-go/pkg/daemon/bbdevconfig.go:111","func":"github.com/intel-collab/applications.orchestration.operators.sriov-fec-operator/pkg/daemon.(*pfBBConfigController).initializePfBBConfig","level":"error","msg":"failed to configure device's queues","pci":"0000:af:00.0","time":"2024-03-26T14:17:02Z"}
{"error":"exit status 255","file":"/workspace-go/pkg/daemon/daemon.go:485","func":"github.com/intel-collab/applications.orchestration.operators.sriov-fec-operator/pkg/daemon.(*NodeConfigReconciler).configureNode.func1","level":"error","msg":"failed applying new PF/VF configuration","time":"2024-03-26T14:17:02Z"}
{"file":"/workspace-go/pkg/common/drainhelper/drainhelper.go:203","func":"github.com/intel-collab/applications.orchestration.operators.sriov-fec-operator/pkg/common/drainhelper.(*DrainHelper).Run.func2","level":"info","msg":"worker function - end","performUncordon":true,"time":"2024-03-26T14:17:02Z"}
{"file":"/workspace-go/pkg/common/drainhelper/drainhelper.go:184","func":"github.com/intel-collab/applications.orchestration.operators.sriov-fec-operator/pkg/common/drainhelper.(*DrainHelper).Run.func2.2","level":"info","msg":"uncordoning node","time":"2024-03-26T14:17:02Z"}
{"file":"/workspace-go/pkg/common/drainhelper/drainhelper.go:295","func":"github.com/intel-collab/applications.orchestration.operators.sriov-fec-operator/pkg/common/drainhelper.(*DrainHelper).uncordon","level":"info","msg":"starting uncordon attempts","time":"2024-03-26T14:17:02Z"}
{"file":"/workspace-go/pkg/common/drainhelper/drainhelper.go:304","func":"github.com/intel-collab/applications.orchestration.operators.sriov-fec-operator/pkg/common/drainhelper.(*DrainHelper).uncordon","level":"info","msg":"node uncordoned","time":"2024-03-26T14:17:02Z"}
{"file":"/workspace-go/pkg/common/drainhelper/drainhelper.go:175","func":"github.com/intel-collab/applications.orchestration.operators.sriov-fec-operator/pkg/common/drainhelper.(*DrainHelper).Run.func2.1","level":"info","msg":"cancelling the context to finish the leadership","time":"2024-03-26T14:17:02Z"}
{"file":"/workspace-go/pkg/common/drainhelper/drainhelper.go:209","func":"github.com/intel-collab/applications.orchestration.operators.sriov-fec-operator/pkg/common/drainhelper.(*DrainHelper).Run.func3","level":"info","msg":"stopped leading","time":"2024-03-26T14:17:02Z"}
{"file":"/workspace-go/pkg/common/drainhelper/drainhelper.go:153","func":"github.com/intel-collab/applications.orchestration.operators.sriov-fec-operator/pkg/common/drainhelper.(*DrainHelper).Run.func1","level":"info","msg":"releasing the lock (bug mitigation)","time":"2024-03-26T14:17:02Z"}
{"error":"exit status 255","file":"/workspace-go/pkg/daemon/daemon.go:157","func":"github.com/intel-collab/applications.orchestration.operators.sriov-fec-operator/pkg/daemon.(*NodeConfigReconciler).Reconcile","level":"error","msg":"error occurred during configuring node","time":"2024-03-26T14:17:02Z"}