Uploaded image for project: 'OpenShift Request For Enhancement'
  1. OpenShift Request For Enhancement
  2. RFE-6939

RFE - Add configuration option to control VMware vSphere CSI driver Logging lower levels

XMLWordPrintable

    • Icon: Feature Request Feature Request
    • Resolution: Can't Do
    • Icon: Normal Normal
    • None
    • None
    • Storage
    • None
    • None
    • Product / Portfolio Work
    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Proposed title of this feature request: VMware vSphere CSI driver Logging option logging levels.

      Following an upgrade from OpenShift 4.12.56 to 4.14.33, a customer reports a high volume of log entries originating from the vmware-vsphere-csi-driver-controller pods on production clusters. The logs are primarily coming from the csi-provisioner container and include frequent error messages indicating volumes attached to nodes. This excessive logging is causing congestion in the customer's log infrastructure and affecting their log management system's performance.

      ~~~
      {{E1016 09:51:38.421016 1 controller.go:1007] error syncing volume "pvc-fcc8d785-1b8f-48fd-94b7-3285167c1a4a": persistentvolume pvc-fcc8d785-1b8f-48fd-94b7-3285167c1a4a is still attached to node app-119.namicgmwd32p.ecs.dyn.nsroot.net
      E1016 09:51:34.679365 1 controller.go:1512] delete "pvc-759c3aab-25fc-430d-89c9-2cfc44041e06": volume deletion failed: persistentvolume pvc-759c3aab-25fc-430d-89c9-2cfc44041e06 is still attached to node app-127.namicgmwd32p.ecs.dyn.nsroot.net}}
      ~~~

      The log volume comprises around 22,547 informational entries and 1,499 errors, largely tied to operations that do not impact workload availability but are taxing the logging pipeline.

      Customer requests a configuration option to control the log verbosity level of the VMware vSphere CSI driver, especially in cases where high-volume activities like build and destroy operations generate extensive logging, but for lower levels. What is already possible is the following, this is just a generic example:

      To update the log level, run the following command:
      $ oc patch clustercsidriver/csi.vsphere.vmware.com --type=json -p '[

      {"op": "replace", "path": "/spec/logLevel", "value": "Debug" }

      ]'
      This will cause all csi-driver related pods to restart in the 'openshift-cluster-csi-drivers' namespace:
      $ oc project openshift-cluster-csi-drivers $ oc get pods
      Once all pods have restarted, wait a few minutes, and then grab an inspect of the csi components:
      $ oc adm inspect ns/openshift-cluster-csi-drivers
      Double check to ensure the vSphere password is not included in the log output file and attach to the case. To revert the log level, run:
      $ oc patch clustercsidriver/csi.vsphere.vmware.com --type=json -p '[

      {"op": "replace", "path": "/spec/logLevel", "value": "Normal" }

      ]'
       
      They want to reduce the verbosity of the log level. "Normal" loglevel is printing all the info messages.
      Suggested Solution:
      Introduce a LOWER configurable log verbosity level for the VMware vSphere CSI driver, enabling users to adjust the granularity of logs generated by the csi-provisioner and associated containers. This would support environments with high transaction volumes and improve system observability by focusing on critical logging events.

              rh-gs-gcharot Gregory Charot
              rhn-support-dahernan David Hernandez Fernandez
              None
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved:
                None
                None