Kubelet failed to publish CSINode

4.12.61    

Randomly    

The issue could not be managed reproduced I found it in CI debugging. It seems after the cluster install finished the csinode "ip-10-0-55-251" always failed to be published.   

Jul 22 09:27:48.381: INFO: At 2024-07-22 09:20:47 +0000 UTC - event for my-pvc-mk6scdgy: {ebs.csi.aws.com_aws-ebs-csi-driver-controller-755d94b6b-dpp28_69694037-9530-49a5-960f-9131846bded9 } Provisioning: External provisioner is provisioning volume for claim "e2e-test-storage-general-csi-2f7nw/my-pvc-mk6scdgy"
Jul 22 09:27:48.381: INFO: At 2024-07-22 09:20:47 +0000 UTC - event for my-pvc-mk6scdgy: {ebs.csi.aws.com_aws-ebs-csi-driver-controller-755d94b6b-dpp28_69694037-9530-49a5-960f-9131846bded9 } ProvisioningFailed: failed to provision volume with StorageClass "gp3-csi": error generating accessibility requirements: error getting CSINode for selected node "ip-10-0-55-251.ec2.internal": csinode.storage.k8s.io "ip-10-0-55-251.ec2.internal" not found


# Checkout the system log on issued node, csinode cannot be published due to connectivity and permission issues

tcp 10.0.67.215:6443: connect: connection refused
Jul 22 05:45:43.867690 ip-10-0-55-251 kubenswrapper[2162]: I0722 05:45:43.867654    2162 csi_plugin.go:993] Failed to contact API server when waiting for CSINode publishing: Get "https://api-int.ci-op-nx1wbwjz-c0363.qe.devcluster.openshift.com:6443/apis/storage.k8s.io/v1/csinodes/ip-10-0-55-251.ec2.internal": dial tcp 10.0.48.55:6443: connect: connection refused
Jul 22 05:45:49.631628 ip-10-0-55-251 kubenswrapper[2162]: I0722 05:45:49.631603    2162 csi_plugin.go:993] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-55-251.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 22 05:46:08.262381 ip-10-0-55-251 kubenswrapper[2162]: I0722 05:46:08.262353    2162 nodeinfomanager.go:401] Failed to publish CSINode: nodes "ip-10-0-55-251.ec2.internal" not found
Jul 22 05:46:08.262381 ip-10-0-55-251 kubenswrapper[2162]: E0722 05:46:08.262381    2162 csi_plugin.go:295] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-55-251.ec2.internal" not found
  

One of the csinode "ip-10-0-55-251" always failed to be published.      

 When cluster install successfully all the csinode should be published.