Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-56898

openstack-cinder-csi-driver-controller-metrics prometheus target showing down for 9202 port

XMLWordPrintable

    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • ShiftStack Sprint 271, ShiftStack Sprint 272, ShiftStack Sprint 273
    • 3
    • -
    • In Progress
    • Release Note Not Required
    • None
    • None
    • None
    • None
    • None

      This is a clone of issue OCPBUGS-54975. The following is the description of the original issue:

      Description of problem:

      Cinder CSI driver controller targets showing down for prometheus:
      ~~~
      openstack-cinder-csi-driver-controller-854d987bc9-n4ztn         https://10.129.0.48:9202/metrics                 down     server returned HTTP status 502 Bad Gateway   
      openstack-cinder-csi-driver-controller-854d987bc9-qkzlx         https://10.130.0.38:9202/metrics                 down     server returned HTTP status 502 Bad Gateway
      ~~~
       
      Servicemonitor is scraping metrics on driver-m port:
      ~~~
      apiVersion: monitoring.coreos.com/v1
      kind: ServiceMonitor
        name: openstack-cinder-csi-driver-controller-monitor
        namespace: openshift-cluster-csi-drivers
      spec:
        endpoints:
          interval: 30s
          path: /metrics
          port: driver-m
          scheme: https
      ~~~
      
      When the endpoints yaml are checked, driver-m port is mapped to 9202 port.
      ~~~
      apiVersion: v1
      kind: Endpoints
        name: openstack-cinder-csi-driver-controller-metrics
        namespace: openshift-cluster-csi-drivers
      subsets:
      - addresses:
        - ip: 10.129.0.48
          nodeName: master-xvpnp-0
          targetRef:
            kind: Pod
            name: openstack-cinder-csi-driver-controller-854d987bc9-n4ztn
            namespace: openshift-cluster-csi-drivers
      
        - ip: 10.130.0.38
          nodeName: master-fd4q2-1
          targetRef:
            kind: Pod
            name: openstack-cinder-csi-driver-controller-854d987bc9-qkzlx
            namespace: openshift-cluster-csi-drivers
      
        ports:
        - name: driver-m
          port: 9202
          protocol: TCP
      ~~~
      
      From pod yaml, 9202 port is mapped to container: kube-rbac-proxy-8202
      ~~~
      apiVersion: v1
      kind: Pod  name: openstack-cinder-csi-driver-controller-854d987bc9-n4ztn
        namespace: openshift-cluster-csi-drivers
      spec:
        containers:
        - args:
          - --secure-listen-address=0.0.0.0:9202
          - --upstream=http://127.0.0.1:8202/    name: kube-rbac-proxy-8202
          ports:
          - containerPort: 9202
            name: driver-m
            protocol: TCP
      ~~~
      
      This container is showing below logs:
      ~~~
      2025-04-10T08:27:22.374262574Z I0410 08:27:22.374224       1 kube-rbac-proxy.go:395] Starting TCP socket on 0.0.0.0:9202
      2025-04-10T08:27:22.374502030Z I0410 08:27:22.374471       1 kube-rbac-proxy.go:402] Listening securely on 0.0.0.0:9202
      2025-04-10T08:27:29.071617312Z I0410 08:27:29.071576       1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8202: connect: connection refused
      2025-04-10T08:27:46.671558739Z I0410 08:27:46.668164       1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8202: connect: connection refused2025-04-11T09:26:59.064050080Z I0411 09:26:59.064000       1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8202: connect: connection refused
      2025-04-11T09:27:16.661091884Z I0411 09:27:16.659746       1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8202: connect: connection refused
      ~~~
      
      Other targets for same pod are showing up:
      ~~~
      openstack-cinder-csi-driver-controller-854d987bc9-qkzlx         https://10.130.0.38:9203/metrics                 up                                                     
      openstack-cinder-csi-driver-controller-854d987bc9-n4ztn         https://10.129.0.48:9203/metrics                 up                                                     
      openstack-cinder-csi-driver-controller-854d987bc9-n4ztn         https://10.129.0.48:9204/metrics                 up                                                     
      openstack-cinder-csi-driver-controller-854d987bc9-qkzlx         https://10.130.0.38:9204/metrics                 up                                                     
      openstack-cinder-csi-driver-controller-854d987bc9-qkzlx         https://10.130.0.38:9205/metrics                 up                                                     
      openstack-cinder-csi-driver-controller-854d987bc9-n4ztn         https://10.129.0.48:9205/metrics                 up                                                     
      openstack-cinder-csi-driver-controller-854d987bc9-n4ztn         https://10.129.0.48:9206/metrics                 up                                                     
      openstack-cinder-csi-driver-controller-854d987bc9-qkzlx         https://10.130.0.38:9206/metrics                 up
      ~~~

      Version-Release number of selected component (if applicable):

          OCP 4.18, 4.17 ,4.16 running Cinder CSI driver on OpenStack ENV

      How reproducible:

          

      Steps to Reproduce:

          1.
          2.
          3.
          

      Actual results:

      openstack-cinder-csi-driver-controller target is showing as down for prometheus. 

      Expected results:

      openstack-cinder-csi-driver-controller target should be up for prometheus.     

      Additional info:

          

              sfinucan@redhat.com Stephen Finucane
              openshift-crt-jira-prow OpenShift Prow Bot
              None
              None
              Itshak Brown Itshak Brown
              None
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated: