-
Feature Request
-
Resolution: Can't Do
-
Normal
-
None
-
None
-
None
-
None
-
Product / Portfolio Work
-
None
-
False
-
-
None
-
None
-
None
-
-
None
-
None
-
None
-
None
-
None
Proposed title of this feature request: VMware vSphere CSI driver Logging option logging levels.
Following an upgrade from OpenShift 4.12.56 to 4.14.33, a customer reports a high volume of log entries originating from the vmware-vsphere-csi-driver-controller pods on production clusters. The logs are primarily coming from the csi-provisioner container and include frequent error messages indicating volumes attached to nodes. This excessive logging is causing congestion in the customer's log infrastructure and affecting their log management system's performance.
~~~
{{E1016 09:51:38.421016 1 controller.go:1007] error syncing volume "pvc-fcc8d785-1b8f-48fd-94b7-3285167c1a4a": persistentvolume pvc-fcc8d785-1b8f-48fd-94b7-3285167c1a4a is still attached to node app-119.namicgmwd32p.ecs.dyn.nsroot.net
E1016 09:51:34.679365 1 controller.go:1512] delete "pvc-759c3aab-25fc-430d-89c9-2cfc44041e06": volume deletion failed: persistentvolume pvc-759c3aab-25fc-430d-89c9-2cfc44041e06 is still attached to node app-127.namicgmwd32p.ecs.dyn.nsroot.net}}
~~~
The log volume comprises around 22,547 informational entries and 1,499 errors, largely tied to operations that do not impact workload availability but are taxing the logging pipeline.
Customer requests a configuration option to control the log verbosity level of the VMware vSphere CSI driver, especially in cases where high-volume activities like build and destroy operations generate extensive logging, but for lower levels. What is already possible is the following, this is just a generic example:
To update the log level, run the following command:
$ oc patch clustercsidriver/csi.vsphere.vmware.com --type=json -p '[
]'
This will cause all csi-driver related pods to restart in the 'openshift-cluster-csi-drivers' namespace:
$ oc project openshift-cluster-csi-drivers $ oc get pods
Once all pods have restarted, wait a few minutes, and then grab an inspect of the csi components:
$ oc adm inspect ns/openshift-cluster-csi-drivers
Double check to ensure the vSphere password is not included in the log output file and attach to the case. To revert the log level, run:
$ oc patch clustercsidriver/csi.vsphere.vmware.com --type=json -p '[
]'
They want to reduce the verbosity of the log level. "Normal" loglevel is printing all the info messages.
Suggested Solution:
Introduce a LOWER configurable log verbosity level for the VMware vSphere CSI driver, enabling users to adjust the granularity of logs generated by the csi-provisioner and associated containers. This would support environments with high transaction volumes and improve system observability by focusing on critical logging events.
- relates to
-
OCPBUGS-54496 VMware vSphere CSI driver csi-provisioner too noisy
-
- ASSIGNED
-