Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-23443

Manila deployed without metrics endpoints

XMLWordPrintable

    • ?
    • Moderate
    • No
    • ShiftStack Sprint 245, ShiftStack Sprint 246, ShiftStack Sprint 247, ShiftStack Sprint 248, ShiftStack Sprint 249, ShiftStack Sprint 250
    • 6
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, the `manila-csi-driver-controller-metrics` service had empty endpoints due to an incorrect name for the app selector. With this release the app selector name is changed to `openstack-manila-csi` and the issue is fixed.(link:https://issues.redhat.com/browse/OCPBUGS-23443[*OCPBUGS-23443*])
      Show
      * Previously, the `manila-csi-driver-controller-metrics` service had empty endpoints due to an incorrect name for the app selector. With this release the app selector name is changed to `openstack-manila-csi` and the issue is fixed.(link: https://issues.redhat.com/browse/OCPBUGS-23443 [* OCPBUGS-23443 *])
    • Bug Fix
    • Done

      This is a clone of issue OCPBUGS-9331. The following is the description of the original issue:

      Version: 4.11.0-0.nightly-2022-06-22-015220

      $ openshift-install version
      openshift-install 4.11.0-0.nightly-2022-06-22-015220
      built from commit f912534f12491721e3874e2bf64f7fa8d44aa7f5
      release image registry.ci.openshift.org/ocp/release@sha256:9c2e9cafaaf48464a0d27652088d8fb3b2336008a615868aadf8223202bdc082
      release architecture amd64

      Platform: OSP 16.1.8 with manila service

      Please specify:

      • IPI

      What happened?

      In a fresh 4.11 cluster (with Kuryr, but shouldn't be related to the issue), there are not endpoints
      for manila metrics:

      > $ oc -n openshift-manila-csi-driver get endpoints
      NAME ENDPOINTS AGE
      manila-csi-driver-controller-metrics <none> 3h7m

      > $ oc -n openshift-manila-csi-driver describe endpoints
      Name: manila-csi-driver-controller-metrics
      Namespace: openshift-manila-csi-driver
      Labels: app=manila-csi-driver-controller-metrics
      Annotations: endpoints.kubernetes.io/last-change-trigger-time: 2022-06-22T10:30:06Z
      Subsets:
      Events: <none>

      > $ oc -n openshift-manila-csi-driver get all
      NAME READY STATUS RESTARTS AGE
      pod/csi-nodeplugin-nfsplugin-4mqgx 1/1 Running 0 3h7m
      pod/csi-nodeplugin-nfsplugin-555ns 1/1 Running 0 3h2m
      pod/csi-nodeplugin-nfsplugin-bn26j 1/1 Running 0 3h7m
      pod/csi-nodeplugin-nfsplugin-lfsm7 1/1 Running 0 3h1m
      pod/csi-nodeplugin-nfsplugin-xwxnz 1/1 Running 0 3h1m
      pod/csi-nodeplugin-nfsplugin-zqnkt 1/1 Running 0 3h7m
      pod/openstack-manila-csi-controllerplugin-7fc4b4f56d-ddn25 6/6 Running 2 (158m ago) 3h7m
      pod/openstack-manila-csi-controllerplugin-7fc4b4f56d-p9jss 6/6 Running 0 3h6m
      pod/openstack-manila-csi-nodeplugin-6w426 2/2 Running 0 3h2m
      pod/openstack-manila-csi-nodeplugin-fvsjr 2/2 Running 0 3h7m
      pod/openstack-manila-csi-nodeplugin-g9x4t 2/2 Running 0 3h1m
      pod/openstack-manila-csi-nodeplugin-gp76x 2/2 Running 0 3h7m
      pod/openstack-manila-csi-nodeplugin-n9v9t 2/2 Running 0 3h7m
      pod/openstack-manila-csi-nodeplugin-s6srv 2/2 Running 0 3h1m

      NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
      service/manila-csi-driver-controller-metrics ClusterIP 172.30.118.232 <none> 443/TCP,444/TCP 3h7m

      NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
      daemonset.apps/csi-nodeplugin-nfsplugin 6 6 6 6 6 <none> 3h7m
      daemonset.apps/openstack-manila-csi-nodeplugin 6 6 6 6 6 <none> 3h7m

      NAME READY UP-TO-DATE AVAILABLE AGE
      deployment.apps/openstack-manila-csi-controllerplugin 2/2 2 2 3h7m

      NAME DESIRED CURRENT READY AGE
      replicaset.apps/openstack-manila-csi-controllerplugin-5697ccfcbf 0 0 0 3h7m
      replicaset.apps/openstack-manila-csi-controllerplugin-7fc4b4f56d 2 2 2 3h7m

      This can lead to not being able to retrieve manila metrics.

      openshift_install.log: http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/j2pg/DFG-osasinfra-shiftstack_periodic_subjob-ocp_install-4.11-kuryr-ipi/15/undercloud-0/home/stack/ostest/.openshift_install.log.gz

      must_gather: http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/j2pg/DFG-osasinfra-shiftstack_periodic_subjob-ocp_install-4.11-kuryr-ipi/15/infrared/.workspaces/workspace_2022-06-22_09-58-34/must-gather-install.tar.gz/

      cinder-csi for example is configured with such endpoints:

      > $ oc -n openshift-cluster-csi-drivers get endpoints
      NAME ENDPOINTS AGE
      openstack-cinder-csi-driver-controller-metrics 10.196.1.100:9203,10.196.2.82:9203,10.196.1.100:9205 + 5 more... 3h15m

      > $ oc -n openshift-cluster-csi-drivers describe endpoints
      Name: openstack-cinder-csi-driver-controller-metrics
      Namespace: openshift-cluster-csi-drivers
      Labels: app=openstack-cinder-csi-driver-controller-metrics
      Annotations: endpoints.kubernetes.io/last-change-trigger-time: 2022-06-22T10:58:57Z
      Subsets:
      Addresses: 10.196.1.100,10.196.2.82
      NotReadyAddresses: <none>
      Ports:
      Name Port Protocol
      ---- ---- --------
      attacher-m 9203 TCP
      snapshotter-m 9205 TCP
      provisioner-m 9202 TCP
      resizer-m 9204 TCP

      Events: <none>

              maandre@redhat.com Martin André
              openshift-crt-jira-prow OpenShift Prow Bot
              Jon Uriarte Jon Uriarte
              Darragh Fitzmaurice Darragh Fitzmaurice
              Red Hat Employee
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated:
                Resolved: