Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-19367

The console handler panics on baremetal 4.14.0-rc.0 ipv6 sno cluster

XMLWordPrintable

    • None
    • No
    • True
    • NA
    • Release Note Not Required
    • In Progress

      Description of problem:

      baremetal 4.14.0-rc.0 ipv6 sno cluster, login as admin user to admin console, there is not Observe menu on the left navigation bar, see picture, https://drive.google.com/file/d/13RAXPxtKhAElN9xf8bAmLJa0GI8pP0fH/view?usp=sharing, monitoring-plugin status is Failed, see: https://drive.google.com/file/d/1YsSaGdLT4bMn-6E-WyFWbOpwvDY4t6na/view?usp=sharing, error is

      Failed to get a valid plugin manifest from /api/plugins/monitoring-plugin/
      r: Bad Gateway 

      checked console logs, 9443: connect: connection refused

      $ oc -n openshift-console logs console-6869f8f4f4-56mbj
      ...
      E0915 12:50:15.498589       1 handlers.go:164] GET request for "monitoring-plugin" plugin failed: Get "https://monitoring-plugin.openshift-monitoring.svc.cluster.local:9443/plugin-manifest.json": dial tcp [fd02::f735]:9443: connect: connection refused
      2023/09/15 12:50:15 http: panic serving [fd01:0:0:1::2]:39156: runtime error: invalid memory address or nil pointer dereference
      goroutine 183760 [running]:
      net/http.(*conn).serve.func1()
          /usr/lib/golang/src/net/http/server.go:1854 +0xbf
      panic({0x3259140, 0x4fcc150})
          /usr/lib/golang/src/runtime/panic.go:890 +0x263
      github.com/openshift/console/pkg/plugins.(*PluginsHandler).proxyPluginRequest(0xc0003b5760, 0x2?, {0xc0009bc7d1, 0x11}, {0x3a41fa0, 0xc0002f6c40}, 0xb?)
          /go/src/github.com/openshift/console/pkg/plugins/handlers.go:165 +0x582
      github.com/openshift/console/pkg/plugins.(*PluginsHandler).HandlePluginAssets(0xaa00000000000010?, {0x3a41fa0, 0xc0002f6c40}, 0xc0001f7500)
          /go/src/github.com/openshift/console/pkg/plugins/handlers.go:147 +0x26d
      github.com/openshift/console/pkg/server.(*Server).HTTPHandler.func23({0x3a41fa0?, 0xc0002f6c40?}, 0x7?)
          /go/src/github.com/openshift/console/pkg/server/server.go:604 +0x33
      net/http.HandlerFunc.ServeHTTP(...)
          /usr/lib/golang/src/net/http/server.go:2122
      github.com/openshift/console/pkg/server.authMiddleware.func1(0xc0001f7500?, {0x3a41fa0?, 0xc0002f6c40?}, 0xd?)
          /go/src/github.com/openshift/console/pkg/server/middleware.go:25 +0x31
      github.com/openshift/console/pkg/server.authMiddlewareWithUser.func1({0x3a41fa0, 0xc0002f6c40}, 0xc0001f7500)
          /go/src/github.com/openshift/console/pkg/server/middleware.go:81 +0x46c
      net/http.HandlerFunc.ServeHTTP(0x5120938?, {0x3a41fa0?, 0xc0002f6c40?}, 0x7ffb6ea27f18?)
          /usr/lib/golang/src/net/http/server.go:2122 +0x2f
      net/http.StripPrefix.func1({0x3a41fa0, 0xc0002f6c40}, 0xc0001f7400)
          /usr/lib/golang/src/net/http/server.go:2165 +0x332
      net/http.HandlerFunc.ServeHTTP(0xc001102c00?, {0x3a41fa0?, 0xc0002f6c40?}, 0xc000655a00?)
          /usr/lib/golang/src/net/http/server.go:2122 +0x2f
      net/http.(*ServeMux).ServeHTTP(0x34025e0?, {0x3a41fa0, 0xc0002f6c40}, 0xc0001f7400)
          /usr/lib/golang/src/net/http/server.go:2500 +0x149
      github.com/openshift/console/pkg/server.securityHeadersMiddleware.func1({0x3a41fa0, 0xc0002f6c40}, 0x3305040?)
          /go/src/github.com/openshift/console/pkg/server/middleware.go:128 +0x3af
      net/http.HandlerFunc.ServeHTTP(0x0?, {0x3a41fa0?, 0xc0002f6c40?}, 0x11db52e?)
          /usr/lib/golang/src/net/http/server.go:2122 +0x2f
      net/http.serverHandler.ServeHTTP({0xc0008201e0?}, {0x3a41fa0, 0xc0002f6c40}, 0xc0001f7400)
          /usr/lib/golang/src/net/http/server.go:2936 +0x316
      net/http.(*conn).serve(0xc0009b4120, {0x3a43e70, 0xc001223500})
          /usr/lib/golang/src/net/http/server.go:1995 +0x612
      created by net/http.(*Server).Serve
          /usr/lib/golang/src/net/http/server.go:3089 +0x5ed
      I0915 12:50:24.267777       1 handlers.go:118] User settings ConfigMap "user-settings-4b4c2f4d-159c-4358-bba3-3d87f113cd9b" already exist, will return existing data.
      I0915 12:50:24.267813       1 handlers.go:118] User settings ConfigMap "user-settings-4b4c2f4d-159c-4358-bba3-3d87f113cd9b" already exist, will return existing data.
      E0915 12:50:30.155515       1 handlers.go:164] GET request for "monitoring-plugin" plugin failed: Get "https://monitoring-plugin.openshift-monitoring.svc.cluster.local:9443/plugin-manifest.json": dial tcp [fd02::f735]:9443: connect: connection refused
      2023/09/15 12:50:30 http: panic serving [fd01:0:0:1::2]:42990: runtime error: invalid memory address or nil pointer dereference 

      9443 port is Connection refused

      $ oc -n openshift-monitoring get pod -o wide
      NAME                                                     READY   STATUS    RESTARTS   AGE     IP                  NODE    NOMINATED NODE   READINESS GATES
      alertmanager-main-0                                      6/6     Running   6          3d22h   fd01:0:0:1::564     sno-2   <none>           <none>
      cluster-monitoring-operator-6cb777d488-nnpmx             1/1     Running   4          7d16h   fd01:0:0:1::12      sno-2   <none>           <none>
      kube-state-metrics-dc5f769bc-p97m7                       3/3     Running   12         7d16h   fd01:0:0:1::3b      sno-2   <none>           <none>
      monitoring-plugin-85bfb98485-d4g5x                       1/1     Running   4          7d16h   fd01:0:0:1::55      sno-2   <none>           <none>
      node-exporter-ndnnj                                      2/2     Running   8          7d16h   2620:52:0:165::41   sno-2   <none>           <none>
      openshift-state-metrics-78df59b4d5-j6r5s                 3/3     Running   12         7d16h   fd01:0:0:1::3a      sno-2   <none>           <none>
      prometheus-adapter-6f86f7d8f5-ttflf                      1/1     Running   0          4h23m   fd01:0:0:1::b10c    sno-2   <none>           <none>
      prometheus-k8s-0                                         6/6     Running   6          3d22h   fd01:0:0:1::566     sno-2   <none>           <none>
      prometheus-operator-7c94855989-csts2                     2/2     Running   8          7d16h   fd01:0:0:1::39      sno-2   <none>           <none>
      prometheus-operator-admission-webhook-7bb64b88cd-bvq8m   1/1     Running   4          7d16h   fd01:0:0:1::37      sno-2   <none>           <none>
      thanos-querier-5bbb764599-vlztq                          6/6     Running   6          3d22h   fd01:0:0:1::56a     sno-2   <none>           <none>
      
      $  oc -n openshift-monitoring get svc monitoring-plugin
      NAME                TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
      monitoring-plugin   ClusterIP   fd02::f735   <none>        9443/TCP   7d16h
      
      
      $ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -v 'https://monitoring-plugin.openshift-monitoring.svc.cluster.local:9443/plugin-manifest.json' | jq
      *   Trying fd02::f735...
      * TCP_NODELAY set
      * connect to fd02::f735 port 9443 failed: Connection refused
      * Failed to connect to monitoring-plugin.openshift-monitoring.svc.cluster.local port 9443: Connection refused
      * Closing connection 0
      curl: (7) Failed to connect to monitoring-plugin.openshift-monitoring.svc.cluster.local port 9443: Connection refused
      command terminated with exit code 7
      

      no such issue in other 4.14.0-rc.0 ipv4 cluster, but issue reproduced on other 4.14.0-rc.0 ipv6 cluster.
      4.14.0-rc.0 ipv4 cluster,

      $ oc get clusterversion
      NAME      VERSION       AVAILABLE   PROGRESSING   SINCE   STATUS
      version   4.14.0-rc.0   True        False         20m     Cluster version is 4.14.0-rc.0
      
      $ oc -n openshift-monitoring get pod -o wide | grep monitoring-plugin
      monitoring-plugin-85bfb98485-nh428                       1/1     Running   0          4m      10.128.0.107   ci-ln-pby4bj2-72292-l5q8v-master-0   <none>           <none>
      
      $ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k  'https://monitoring-plugin.openshift-monitoring.svc.cluster.local:9443/plugin-manifest.json' | jq
      ...
      {
        "name": "monitoring-plugin",
        "version": "1.0.0",
        "displayName": "OpenShift console monitoring plugin",
        "description": "This plugin adds the monitoring UI to the OpenShift web console",
        "dependencies": {
          "@console/pluginAPI": "*"
        },
        "extensions": [
          {
            "type": "console.page/route",
            "properties": {
              "exact": true,
              "path": "/monitoring",
              "component": {
                "$codeRef": "MonitoringUI"
              }
            }
          },
      ...

      meet issue "9443: Connection refused" in 4.14.0-rc.0 ipv6 cluster(launched cluster-bot cluster: launch 4.14.0-rc.0 metal,ipv6) and login console

      $ oc get clusterversion
      NAME      VERSION       AVAILABLE   PROGRESSING   SINCE   STATUS
      version   4.14.0-rc.0   True        False         44m     Cluster version is 4.14.0-rc.0
      $ oc -n openshift-monitoring get pod -o wide | grep monitoring-plugin
      monitoring-plugin-bd6ffdb5d-b5csk                        1/1     Running   0          53m   fd01:0:0:4::b             worker-0.ostest.test.metalkube.org   <none>           <none>
      monitoring-plugin-bd6ffdb5d-vhtpf                        1/1     Running   0          53m   fd01:0:0:5::9             worker-2.ostest.test.metalkube.org   <none>           <none>
      $ oc -n openshift-monitoring get svc monitoring-plugin
      NAME                TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
      monitoring-plugin   ClusterIP   fd02::402d   <none>        9443/TCP   59m
      
      $ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -v 'https://monitoring-plugin.openshift-monitoring.svc.cluster.local:9443/plugin-manifest.json' | jq
      *   Trying fd02::402d...
      * TCP_NODELAY set
      * connect to fd02::402d port 9443 failed: Connection refused
      * Failed to connect to monitoring-plugin.openshift-monitoring.svc.cluster.local port 9443: Connection refused
      * Closing connection 0
      curl: (7) Failed to connect to monitoring-plugin.openshift-monitoring.svc.cluster.local port 9443: Connection refused
      command terminated with exit code 7$ oc -n openshift-console get pod | grep console
      console-5cffbc7964-7ljft     1/1     Running   0          56m
      console-5cffbc7964-d864q     1/1     Running   0          56m$ oc -n openshift-console logs console-5cffbc7964-7ljft
      ...
      E0916 14:34:16.330117       1 handlers.go:164] GET request for "monitoring-plugin" plugin failed: Get "https://monitoring-plugin.openshift-monitoring.svc.cluster.local:9443/plugin-manifest.json": dial tcp [fd02::402d]:9443: connect: connection refused
      2023/09/16 14:34:16 http: panic serving [fd01:0:0:4::2]:37680: runtime error: invalid memory address or nil pointer dereference
      goroutine 3985 [running]:
      net/http.(*conn).serve.func1()
          /usr/lib/golang/src/net/http/server.go:1854 +0xbf
      panic({0x3259140, 0x4fcc150})
          /usr/lib/golang/src/runtime/panic.go:890 +0x263
      github.com/openshift/console/pkg/plugins.(*PluginsHandler).proxyPluginRequest(0xc0008f6780, 0x2?, {0xc000665211, 0x11}, {0x3a41fa0, 0xc0009221c0}, 0xb?)
          /go/src/github.com/openshift/console/pkg/plugins/handlers.go:165 +0x582
      github.com/openshift/console/pkg/plugins.(*PluginsHandler).HandlePluginAssets(0xfe00000000000010?, {0x3a41fa0, 0xc0009221c0}, 0xc000d8d600)
          /go/src/github.com/openshift/console/pkg/plugins/handlers.go:147 +0x26d
      github.com/openshift/console/pkg/server.(*Server).HTTPHandler.func23({0x3a41fa0?, 0xc0009221c0?}, 0x7?)
          /go/src/github.com/openshift/console/pkg/server/server.go:604 +0x33
      net/http.HandlerFunc.ServeHTTP(...)
          /usr/lib/golang/src/net/http/server.go:2122
      github.com/openshift/console/pkg/server.authMiddleware.func1(0xc000d8d600?, {0x3a41fa0?, 0xc0009221c0?}, 0xd?)
          /go/src/github.com/openshift/console/pkg/server/middleware.go:25 +0x31
      github.com/openshift/console/pkg/server.authMiddlewareWithUser.func1({0x3a41fa0, 0xc0009221c0}, 0xc000d8d600)
          /go/src/github.com/openshift/console/pkg/server/middleware.go:81 +0x46c
      net/http.HandlerFunc.ServeHTTP(0xc000653830?, {0x3a41fa0?, 0xc0009221c0?}, 0x7f824506bf18?)
          /usr/lib/golang/src/net/http/server.go:2122 +0x2f
      net/http.StripPrefix.func1({0x3a41fa0, 0xc0009221c0}, 0xc000d8d500)
          /usr/lib/golang/src/net/http/server.go:2165 +0x332
      net/http.HandlerFunc.ServeHTTP(0xc00007e800?, {0x3a41fa0?, 0xc0009221c0?}, 0xc000b2da00?)
          /usr/lib/golang/src/net/http/server.go:2122 +0x2f
      net/http.(*ServeMux).ServeHTTP(0x34025e0?, {0x3a41fa0, 0xc0009221c0}, 0xc000d8d500)
          /usr/lib/golang/src/net/http/server.go:2500 +0x149
      github.com/openshift/console/pkg/server.securityHeadersMiddleware.func1({0x3a41fa0, 0xc0009221c0}, 0x3305040?)
          /go/src/github.com/openshift/console/pkg/server/middleware.go:128 +0x3af
      net/http.HandlerFunc.ServeHTTP(0x0?, {0x3a41fa0?, 0xc0009221c0?}, 0x11db52e?)
          /usr/lib/golang/src/net/http/server.go:2122 +0x2f
      net/http.serverHandler.ServeHTTP({0xc000db9b00?}, {0x3a41fa0, 0xc0009221c0}, 0xc000d8d500)
          /usr/lib/golang/src/net/http/server.go:2936 +0x316
      net/http.(*conn).serve(0xc000653680, {0x3a43e70, 0xc000676f30})
          /usr/lib/golang/src/net/http/server.go:1995 +0x612
      created by net/http.(*Server).Serve
          /usr/lib/golang/src/net/http/server.go:3089 +0x5ed 

      Version-Release number of selected component (if applicable):

      baremetal 4.14.0-rc.0 ipv6 sno cluster,
      $ token=`oc create token prometheus-k8s -n openshift-monitoring`
      $ $ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://thanos-querier.openshift-monitoring.svc:9091/api/v1/query?' --data-urlencode 'query=virt_platform'  | jq
      {
        "status": "success",
        "data": {
          "resultType": "vector",
          "result": [
            {
              "metric": {
                "__name__": "virt_platform",
                "baseboard_manufacturer": "Dell Inc.",
                "baseboard_product_name": "01J4WF",
                "bios_vendor": "Dell Inc.",
                "bios_version": "1.10.2",
                "container": "kube-rbac-proxy",
                "endpoint": "https",
                "instance": "sno-2",
                "job": "node-exporter",
                "namespace": "openshift-monitoring",
                "pod": "node-exporter-ndnnj",
                "prometheus": "openshift-monitoring/k8s",
                "service": "node-exporter",
                "system_manufacturer": "Dell Inc.",
                "system_product_name": "PowerEdge R750",
                "system_version": "Not Specified",
                "type": "none"
              },
              "value": [
                1694785092.664,
                "1"
              ]
            }
          ]
        }
      }

      How reproducible:

      only seen on this cluster

      Steps to Reproduce:

      1. see the description
      2.
      3.
      

      Actual results:

      no Observe menu on admin console, monitoring-plugin is failed

      Expected results:

      no error

            jhadvig@redhat.com Jakub Hadvig
            juzhao@redhat.com Junqi Zhao
            Tai Gao Tai Gao
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: