-
Bug
-
Resolution: Unresolved
-
Normal
-
None
-
4.22.0, 4.21.z
-
None
Description of problem:
[CI testing] flakiness in MutableCSINodeAllocatableCount test
Version-Release number of selected component (if applicable):
4.22 nightly
How reproducible:
Randomly
Steps to Reproduce:
Check the sippy test records samples: https://sippy.dptools.openshift.org/sippy-ng/tests/4.22/details?filters=%7B%22items%22%3A%5B%7B%22columnField%22%3A%22name%22%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22FeatureGate%3AMutableCSINodeAllocatableCount%5D%22%7D%5D%7D&pageSize=100
Actual results:
[sig-storage][FeatureGate:MutableCSINodeAllocatableCount][Jira:“Storage”][Serial][Driver: ebs.csi.aws.com] should automatically update CSINode allocatable count when instance attached ENI count changes [Suite:openshift/conformance/serial] has 1 failed sample in 31 test runs, pass ratio is 96%
Expected results:
[sig-storage][FeatureGate:MutableCSINodeAllocatableCount][Jira:“Storage”][Serial][Driver: ebs.csi.aws.com] should automatically update CSINode allocatable count when instance attached ENI count changes [Suite:openshift/conformance/serial] should always passed.
Additional info:
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.22-e2e-aws-ovn-techpreview-serial-3of3/2016648388887449600/artifacts/e2e-aws-ovn-techpreview-serial/openshift-e2e-test/build-log.txt
I found in the test logs during 2026-01-29T00:54:37Z - 2026-01-29T00:54:40Z the ebs csi driver operator pod created again and the clustercsidriver seems reconcile back again ->
time="2026-01-29T00:54:37Z" level=info msg="event interval matches VsphereConfigurationTestsRollOutTooOften" locator="{Kind map[hmsg:d08fdfb20a namespace:openshift-cluster-csi-drivers replicaset:aws-ebs-csi-driver-operator-76548956d7]}" message="{SuccessfulDelete Deleted pod: aws-ebs-csi-driver-operator-76548956d7-v5w6n map[firstTimestamp:2026-01-29T00:54:37Z lastTimestamp:2026-01-29T00:54:37Z reason:SuccessfulDelete]}"
time="2026-01-29T00:54:37Z" level=info msg="event interval matches VsphereConfigurationTestsRollOutTooOften" locator="{Kind map[hmsg:b568ac0a66 namespace:openshift-cluster-csi-drivers replicaset:aws-ebs-csi-driver-operator-76548956d7]}" message="{SuccessfulCreate Created pod: aws-ebs-csi-driver-operator-76548956d7-mhglr map[firstTimestamp:2026-01-29T00:54:37Z lastTimestamp:2026-01-29T00:54:37Z reason:SuccessfulCreate]}"
time="2026-01-29T00:54:40Z" level=info msg="event interval matches VsphereConfigurationTestsRollOutTooOften" locator="{Kind map[hmsg:ff8cd3d029 namespace:openshift-cluster-csi-drivers node:ip-10-0-0-150.us-west-2.compute.internal pod:aws-ebs-csi-driver-operator-76548956d7-mhglr]}" message="{Created Created container: aws-ebs-csi-driver-operator map[firstTimestamp:2026-01-29T00:54:40Z lastTimestamp:2026-01-29T00:54:40Z reason:Created]}"
time="2026-01-29T00:54:44Z" level=info msg="event interval matches VsphereConfigurationTestsRollOutTooOften" locator="{Kind map[deployment:aws-ebs-csi-driver-operator hmsg:4b9015bace namespace:openshift-cluster-csi-drivers]}" message="{CSIDriverUpdated Updated CSIDriver.storage.k8s.io/ebs.csi.aws.com because it changed map[firstTimestamp:2026-01-29T00:54:44Z lastTimestamp:2026-01-29T00:54:44Z reason:CSIDriverUpdated]}"while the scaled down finished at I0129 00:54:37.666658 7927 mutable_csinode_allocatable.go:581] Successfully scaled down aws-ebs-csi-driver-operator
the ebs csi driver operator actually does not scale down successfully so the nodeAllocatableUpdatePeriodSeconds reconcile back to 600 . So the next step the CSINodeAllocatableCount does not update in time …