-
Bug
-
Resolution: Done-Errata
-
Undefined
-
4.14.0
-
None
-
No
-
False
-
-
Release Note Not Required
-
In Progress
This is a clone of issue OCPBUGS-19367. The following is the description of the original issue:
—
Description of problem:
baremetal 4.14.0-rc.0 ipv6 sno cluster, login as admin user to admin console, there is not Observe menu on the left navigation bar, see picture, https://drive.google.com/file/d/13RAXPxtKhAElN9xf8bAmLJa0GI8pP0fH/view?usp=sharing, monitoring-plugin status is Failed, see: https://drive.google.com/file/d/1YsSaGdLT4bMn-6E-WyFWbOpwvDY4t6na/view?usp=sharing, error is
Failed to get a valid plugin manifest from /api/plugins/monitoring-plugin/ r: Bad Gateway
checked console logs, 9443: connect: connection refused
$ oc -n openshift-console logs console-6869f8f4f4-56mbj ... E0915 12:50:15.498589 1 handlers.go:164] GET request for "monitoring-plugin" plugin failed: Get "https://monitoring-plugin.openshift-monitoring.svc.cluster.local:9443/plugin-manifest.json": dial tcp [fd02::f735]:9443: connect: connection refused 2023/09/15 12:50:15 http: panic serving [fd01:0:0:1::2]:39156: runtime error: invalid memory address or nil pointer dereference goroutine 183760 [running]: net/http.(*conn).serve.func1() /usr/lib/golang/src/net/http/server.go:1854 +0xbf panic({0x3259140, 0x4fcc150}) /usr/lib/golang/src/runtime/panic.go:890 +0x263 github.com/openshift/console/pkg/plugins.(*PluginsHandler).proxyPluginRequest(0xc0003b5760, 0x2?, {0xc0009bc7d1, 0x11}, {0x3a41fa0, 0xc0002f6c40}, 0xb?) /go/src/github.com/openshift/console/pkg/plugins/handlers.go:165 +0x582 github.com/openshift/console/pkg/plugins.(*PluginsHandler).HandlePluginAssets(0xaa00000000000010?, {0x3a41fa0, 0xc0002f6c40}, 0xc0001f7500) /go/src/github.com/openshift/console/pkg/plugins/handlers.go:147 +0x26d github.com/openshift/console/pkg/server.(*Server).HTTPHandler.func23({0x3a41fa0?, 0xc0002f6c40?}, 0x7?) /go/src/github.com/openshift/console/pkg/server/server.go:604 +0x33 net/http.HandlerFunc.ServeHTTP(...) /usr/lib/golang/src/net/http/server.go:2122 github.com/openshift/console/pkg/server.authMiddleware.func1(0xc0001f7500?, {0x3a41fa0?, 0xc0002f6c40?}, 0xd?) /go/src/github.com/openshift/console/pkg/server/middleware.go:25 +0x31 github.com/openshift/console/pkg/server.authMiddlewareWithUser.func1({0x3a41fa0, 0xc0002f6c40}, 0xc0001f7500) /go/src/github.com/openshift/console/pkg/server/middleware.go:81 +0x46c net/http.HandlerFunc.ServeHTTP(0x5120938?, {0x3a41fa0?, 0xc0002f6c40?}, 0x7ffb6ea27f18?) /usr/lib/golang/src/net/http/server.go:2122 +0x2f net/http.StripPrefix.func1({0x3a41fa0, 0xc0002f6c40}, 0xc0001f7400) /usr/lib/golang/src/net/http/server.go:2165 +0x332 net/http.HandlerFunc.ServeHTTP(0xc001102c00?, {0x3a41fa0?, 0xc0002f6c40?}, 0xc000655a00?) /usr/lib/golang/src/net/http/server.go:2122 +0x2f net/http.(*ServeMux).ServeHTTP(0x34025e0?, {0x3a41fa0, 0xc0002f6c40}, 0xc0001f7400) /usr/lib/golang/src/net/http/server.go:2500 +0x149 github.com/openshift/console/pkg/server.securityHeadersMiddleware.func1({0x3a41fa0, 0xc0002f6c40}, 0x3305040?) /go/src/github.com/openshift/console/pkg/server/middleware.go:128 +0x3af net/http.HandlerFunc.ServeHTTP(0x0?, {0x3a41fa0?, 0xc0002f6c40?}, 0x11db52e?) /usr/lib/golang/src/net/http/server.go:2122 +0x2f net/http.serverHandler.ServeHTTP({0xc0008201e0?}, {0x3a41fa0, 0xc0002f6c40}, 0xc0001f7400) /usr/lib/golang/src/net/http/server.go:2936 +0x316 net/http.(*conn).serve(0xc0009b4120, {0x3a43e70, 0xc001223500}) /usr/lib/golang/src/net/http/server.go:1995 +0x612 created by net/http.(*Server).Serve /usr/lib/golang/src/net/http/server.go:3089 +0x5ed I0915 12:50:24.267777 1 handlers.go:118] User settings ConfigMap "user-settings-4b4c2f4d-159c-4358-bba3-3d87f113cd9b" already exist, will return existing data. I0915 12:50:24.267813 1 handlers.go:118] User settings ConfigMap "user-settings-4b4c2f4d-159c-4358-bba3-3d87f113cd9b" already exist, will return existing data. E0915 12:50:30.155515 1 handlers.go:164] GET request for "monitoring-plugin" plugin failed: Get "https://monitoring-plugin.openshift-monitoring.svc.cluster.local:9443/plugin-manifest.json": dial tcp [fd02::f735]:9443: connect: connection refused 2023/09/15 12:50:30 http: panic serving [fd01:0:0:1::2]:42990: runtime error: invalid memory address or nil pointer dereference
9443 port is Connection refused
$ oc -n openshift-monitoring get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES alertmanager-main-0 6/6 Running 6 3d22h fd01:0:0:1::564 sno-2 <none> <none> cluster-monitoring-operator-6cb777d488-nnpmx 1/1 Running 4 7d16h fd01:0:0:1::12 sno-2 <none> <none> kube-state-metrics-dc5f769bc-p97m7 3/3 Running 12 7d16h fd01:0:0:1::3b sno-2 <none> <none> monitoring-plugin-85bfb98485-d4g5x 1/1 Running 4 7d16h fd01:0:0:1::55 sno-2 <none> <none> node-exporter-ndnnj 2/2 Running 8 7d16h 2620:52:0:165::41 sno-2 <none> <none> openshift-state-metrics-78df59b4d5-j6r5s 3/3 Running 12 7d16h fd01:0:0:1::3a sno-2 <none> <none> prometheus-adapter-6f86f7d8f5-ttflf 1/1 Running 0 4h23m fd01:0:0:1::b10c sno-2 <none> <none> prometheus-k8s-0 6/6 Running 6 3d22h fd01:0:0:1::566 sno-2 <none> <none> prometheus-operator-7c94855989-csts2 2/2 Running 8 7d16h fd01:0:0:1::39 sno-2 <none> <none> prometheus-operator-admission-webhook-7bb64b88cd-bvq8m 1/1 Running 4 7d16h fd01:0:0:1::37 sno-2 <none> <none> thanos-querier-5bbb764599-vlztq 6/6 Running 6 3d22h fd01:0:0:1::56a sno-2 <none> <none> $ oc -n openshift-monitoring get svc monitoring-plugin NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE monitoring-plugin ClusterIP fd02::f735 <none> 9443/TCP 7d16h $ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -v 'https://monitoring-plugin.openshift-monitoring.svc.cluster.local:9443/plugin-manifest.json' | jq * Trying fd02::f735... * TCP_NODELAY set * connect to fd02::f735 port 9443 failed: Connection refused * Failed to connect to monitoring-plugin.openshift-monitoring.svc.cluster.local port 9443: Connection refused * Closing connection 0 curl: (7) Failed to connect to monitoring-plugin.openshift-monitoring.svc.cluster.local port 9443: Connection refused command terminated with exit code 7
no such issue in other 4.14.0-rc.0 ipv4 cluster, but issue reproduced on other 4.14.0-rc.0 ipv6 cluster.
4.14.0-rc.0 ipv4 cluster,
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.14.0-rc.0 True False 20m Cluster version is 4.14.0-rc.0
$ oc -n openshift-monitoring get pod -o wide | grep monitoring-plugin
monitoring-plugin-85bfb98485-nh428 1/1 Running 0 4m 10.128.0.107 ci-ln-pby4bj2-72292-l5q8v-master-0 <none> <none>
$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k 'https://monitoring-plugin.openshift-monitoring.svc.cluster.local:9443/plugin-manifest.json' | jq
...
{
"name": "monitoring-plugin",
"version": "1.0.0",
"displayName": "OpenShift console monitoring plugin",
"description": "This plugin adds the monitoring UI to the OpenShift web console",
"dependencies": {
"@console/pluginAPI": "*"
},
"extensions": [
{
"type": "console.page/route",
"properties": {
"exact": true,
"path": "/monitoring",
"component": {
"$codeRef": "MonitoringUI"
}
}
},
...
meet issue "9443: Connection refused" in 4.14.0-rc.0 ipv6 cluster(launched cluster-bot cluster: launch 4.14.0-rc.0 metal,ipv6) and login console
$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.14.0-rc.0 True False 44m Cluster version is 4.14.0-rc.0 $ oc -n openshift-monitoring get pod -o wide | grep monitoring-plugin monitoring-plugin-bd6ffdb5d-b5csk 1/1 Running 0 53m fd01:0:0:4::b worker-0.ostest.test.metalkube.org <none> <none> monitoring-plugin-bd6ffdb5d-vhtpf 1/1 Running 0 53m fd01:0:0:5::9 worker-2.ostest.test.metalkube.org <none> <none> $ oc -n openshift-monitoring get svc monitoring-plugin NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE monitoring-plugin ClusterIP fd02::402d <none> 9443/TCP 59m $ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -v 'https://monitoring-plugin.openshift-monitoring.svc.cluster.local:9443/plugin-manifest.json' | jq * Trying fd02::402d... * TCP_NODELAY set * connect to fd02::402d port 9443 failed: Connection refused * Failed to connect to monitoring-plugin.openshift-monitoring.svc.cluster.local port 9443: Connection refused * Closing connection 0 curl: (7) Failed to connect to monitoring-plugin.openshift-monitoring.svc.cluster.local port 9443: Connection refused command terminated with exit code 7$ oc -n openshift-console get pod | grep console console-5cffbc7964-7ljft 1/1 Running 0 56m console-5cffbc7964-d864q 1/1 Running 0 56m$ oc -n openshift-console logs console-5cffbc7964-7ljft ... E0916 14:34:16.330117 1 handlers.go:164] GET request for "monitoring-plugin" plugin failed: Get "https://monitoring-plugin.openshift-monitoring.svc.cluster.local:9443/plugin-manifest.json": dial tcp [fd02::402d]:9443: connect: connection refused 2023/09/16 14:34:16 http: panic serving [fd01:0:0:4::2]:37680: runtime error: invalid memory address or nil pointer dereference goroutine 3985 [running]: net/http.(*conn).serve.func1() /usr/lib/golang/src/net/http/server.go:1854 +0xbf panic({0x3259140, 0x4fcc150}) /usr/lib/golang/src/runtime/panic.go:890 +0x263 github.com/openshift/console/pkg/plugins.(*PluginsHandler).proxyPluginRequest(0xc0008f6780, 0x2?, {0xc000665211, 0x11}, {0x3a41fa0, 0xc0009221c0}, 0xb?) /go/src/github.com/openshift/console/pkg/plugins/handlers.go:165 +0x582 github.com/openshift/console/pkg/plugins.(*PluginsHandler).HandlePluginAssets(0xfe00000000000010?, {0x3a41fa0, 0xc0009221c0}, 0xc000d8d600) /go/src/github.com/openshift/console/pkg/plugins/handlers.go:147 +0x26d github.com/openshift/console/pkg/server.(*Server).HTTPHandler.func23({0x3a41fa0?, 0xc0009221c0?}, 0x7?) /go/src/github.com/openshift/console/pkg/server/server.go:604 +0x33 net/http.HandlerFunc.ServeHTTP(...) /usr/lib/golang/src/net/http/server.go:2122 github.com/openshift/console/pkg/server.authMiddleware.func1(0xc000d8d600?, {0x3a41fa0?, 0xc0009221c0?}, 0xd?) /go/src/github.com/openshift/console/pkg/server/middleware.go:25 +0x31 github.com/openshift/console/pkg/server.authMiddlewareWithUser.func1({0x3a41fa0, 0xc0009221c0}, 0xc000d8d600) /go/src/github.com/openshift/console/pkg/server/middleware.go:81 +0x46c net/http.HandlerFunc.ServeHTTP(0xc000653830?, {0x3a41fa0?, 0xc0009221c0?}, 0x7f824506bf18?) /usr/lib/golang/src/net/http/server.go:2122 +0x2f net/http.StripPrefix.func1({0x3a41fa0, 0xc0009221c0}, 0xc000d8d500) /usr/lib/golang/src/net/http/server.go:2165 +0x332 net/http.HandlerFunc.ServeHTTP(0xc00007e800?, {0x3a41fa0?, 0xc0009221c0?}, 0xc000b2da00?) /usr/lib/golang/src/net/http/server.go:2122 +0x2f net/http.(*ServeMux).ServeHTTP(0x34025e0?, {0x3a41fa0, 0xc0009221c0}, 0xc000d8d500) /usr/lib/golang/src/net/http/server.go:2500 +0x149 github.com/openshift/console/pkg/server.securityHeadersMiddleware.func1({0x3a41fa0, 0xc0009221c0}, 0x3305040?) /go/src/github.com/openshift/console/pkg/server/middleware.go:128 +0x3af net/http.HandlerFunc.ServeHTTP(0x0?, {0x3a41fa0?, 0xc0009221c0?}, 0x11db52e?) /usr/lib/golang/src/net/http/server.go:2122 +0x2f net/http.serverHandler.ServeHTTP({0xc000db9b00?}, {0x3a41fa0, 0xc0009221c0}, 0xc000d8d500) /usr/lib/golang/src/net/http/server.go:2936 +0x316 net/http.(*conn).serve(0xc000653680, {0x3a43e70, 0xc000676f30}) /usr/lib/golang/src/net/http/server.go:1995 +0x612 created by net/http.(*Server).Serve /usr/lib/golang/src/net/http/server.go:3089 +0x5ed
Version-Release number of selected component (if applicable):
baremetal 4.14.0-rc.0 ipv6 sno cluster,
$ token=`oc create token prometheus-k8s -n openshift-monitoring`
$ $ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://thanos-querier.openshift-monitoring.svc:9091/api/v1/query?' --data-urlencode 'query=virt_platform' | jq
{
"status": "success",
"data": {
"resultType": "vector",
"result": [
{
"metric": {
"__name__": "virt_platform",
"baseboard_manufacturer": "Dell Inc.",
"baseboard_product_name": "01J4WF",
"bios_vendor": "Dell Inc.",
"bios_version": "1.10.2",
"container": "kube-rbac-proxy",
"endpoint": "https",
"instance": "sno-2",
"job": "node-exporter",
"namespace": "openshift-monitoring",
"pod": "node-exporter-ndnnj",
"prometheus": "openshift-monitoring/k8s",
"service": "node-exporter",
"system_manufacturer": "Dell Inc.",
"system_product_name": "PowerEdge R750",
"system_version": "Not Specified",
"type": "none"
},
"value": [
1694785092.664,
"1"
]
}
]
}
}
How reproducible:
only seen on this cluster
Steps to Reproduce:
1. see the description 2. 3.
Actual results:
no Observe menu on admin console, monitoring-plugin is failed
Expected results:
no error
- clones
-
-
- Closed
-
- is blocked by
-
-
- Closed
-
- links to
-
RHBA-2024:2668
OpenShift Container Platform 4.14.z bug fix update