Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-44723

aws-efs-csi-driver-controller-metrics/aws-efs-csi-driver-controller-metrics target down with OpenShift Container Platform 4.17

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • 4.17
    • Storage
    • Important
    • None
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      After the upgrade to OpenShift Container Platform 4.17, it's being observed that aws-efs-csi-driver-controller-metrics/aws-efs-csi-driver-controller-metrics is reporting target down state. When checking the newly created Container one can find the below logs, that may explain the effect seen/reported.
      
      $ oc logs aws-efs-csi-driver-controller-5b8d5cfdf4-zwh67 -c kube-rbac-proxy-8211
      W1119 07:53:10.249934       1 deprecated.go:66] 
      ==== Removed Flag Warning ======================
      
      logtostderr is removed in the k8s upstream and has no effect any more.
      
      ===============================================
      		
      I1119 07:53:10.250382       1 kube-rbac-proxy.go:233] Valid token audiences: 
      I1119 07:53:10.250431       1 kube-rbac-proxy.go:347] Reading certificate files
      I1119 07:53:10.250645       1 kube-rbac-proxy.go:395] Starting TCP socket on 0.0.0.0:9211
      I1119 07:53:10.250944       1 kube-rbac-proxy.go:402] Listening securely on 0.0.0.0:9211
      I1119 07:54:01.440714       1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused
      I1119 07:54:19.860038       1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused
      I1119 07:54:31.432943       1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused
      I1119 07:54:49.852801       1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused
      I1119 07:55:01.433635       1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused
      I1119 07:55:19.853259       1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused
      I1119 07:55:31.432722       1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused
      I1119 07:55:49.852606       1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused
      I1119 07:56:01.432707       1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused
      I1119 07:56:19.853137       1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused
      I1119 07:56:31.440223       1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused
      I1119 07:56:49.856349       1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused
      I1119 07:57:01.432528       1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused
      I1119 07:57:19.853132       1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused
      I1119 07:57:31.433104       1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused
      I1119 07:57:49.852859       1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused
      I1119 07:58:01.433321       1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused
      I1119 07:58:19.853612       1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused
      

      Version-Release number of selected component (if applicable):

      OpenShift Container Platform 4.17
      

      How reproducible:

      Always
      

      Steps to Reproduce:

      1. Install OpenShift Container Platform 4.17
      2. Install aws-efs-csi-driver-operator
      3. Create efs.csi.aws.com CSIDriver object and wait for the aws-efs-csi-driver-controller to roll out.
      

      Actual results:

      The below Target Down Alert is being raised
      
      50% of the aws-efs-csi-driver-controller-metrics/aws-efs-csi-driver-controller-metrics targets in Namespace openshift-cluster-csi-drivers namespace have been unreachable for more than 15 minutes. This may be a symptom of network connectivity issues, down nodes, or failures within these components. Assess the health of the infrastructure and nodes running these targets and then contact support.
      

      Expected results:

      The ServiceMonitor endpoint should be reachable and properly responding with the desired information to monitor the health of the component.
      

      Additional info:

      
      

              rbednar@redhat.com Roman Bednar
              rhn-support-sreber Simon Reber
              Rohit Patil Rohit Patil
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated: