-
Bug
-
Resolution: Not a Bug
-
Undefined
-
None
-
4.17
-
None
-
Quality / Stability / Reliability
-
False
-
-
None
-
None
-
No
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
We are trying to deploy Multicluster Engine 2.6.0-121, but it failes because a the openshift-console-operator service webhook is missing
Version-Release number of selected component (if applicable):
ocp-4.17.0-0.nightly-2024-06-13-010514 and 4.17-ec0
How reproducible:
Always
Steps to Reproduce:
1. Deploy an OCP cluster v4.17x 2. Try to deploy ACM v2.11.0-111 + MCE 2.6.0-121 3. The MCE CR never completes the installation because the webhook service is not available. * The same operators versions can be installed in OCP 4.16 which has the webhook endpoint available * The cluster operators are not degraded
Actual results:
MCE fails during its installation
Expected results:
MCE to deploy sucessfully The openshift-console-operator webhook should be present
Additional info:
$ oc describe multiclusterengines.multicluster.openshift.io multiclusterengine | tail -20
Last Transition Time: 2024-06-11T08:36:49Z
Last Update Time: 2024-06-11T08:36:49Z
Message: Setting the operator
Reason: WaitingForResource
Status: True
Type: Progressing
Last Transition Time: 2024-06-11T08:38:48Z
Last Update Time: 2024-06-11T08:38:48Z
Reason: ComponentsAvailable
Status: True
Type: Available
Last Transition Time: 2024-06-11T12:26:58Z
Last Update Time: 2024-06-11T12:26:58Z
Message: error applying object Name: mce Kind: ConsolePlugin Error: conversion webhook for console.openshift.io/v1alpha1, Kind=ConsolePlugin failed: Post "https://webhook.openshift-console-operator.svc:9443/crdconvert?timeout=30s": service "webhook" not found
Reason: FailedApplyingComponent
Status: True
Type: ComponentFailure: mce (Kind:ConsolePlugin)
Desired Version: 2.6.0-121
Phase: Error
Events: <none>
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.17.0-0.nightly-2024-06-13-010514 True False 178m Cluster version is 4.17.0-0.nightly-2024-06-13-010514$ oc get co |grep console
console 4.17.0-0.nightly-2024-06-13-010514 True False False 102m
$ oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication 4.17.0-0.nightly-2024-06-13-010514 True False False 3h52m
baremetal 4.17.0-0.nightly-2024-06-13-010514 True False False 4h37m
cloud-controller-manager 4.17.0-0.nightly-2024-06-13-010514 True False False 4h38m
cloud-credential 4.17.0-0.nightly-2024-06-13-010514 True False False 5h8m
cluster-autoscaler 4.17.0-0.nightly-2024-06-13-010514 True False False 4h37m
config-operator 4.17.0-0.nightly-2024-06-13-010514 True False False 4h37m
console 4.17.0-0.nightly-2024-06-13-010514 True False False 103m
control-plane-machine-set 4.17.0-0.nightly-2024-06-13-010514 True False False 4h37m
csi-snapshot-controller 4.17.0-0.nightly-2024-06-13-010514 True False False 4h37m
dns 4.17.0-0.nightly-2024-06-13-010514 True False False 4h36m
etcd 4.17.0-0.nightly-2024-06-13-010514 True False False 4h35m
image-registry 4.17.0-0.nightly-2024-06-13-010514 True False False 4h12m
ingress 4.17.0-0.nightly-2024-06-13-010514 True False False 4h1m
insights 4.17.0-0.nightly-2024-06-13-010514 True False False 4h30m
kube-apiserver 4.17.0-0.nightly-2024-06-13-010514 True False False 4h22m
kube-controller-manager 4.17.0-0.nightly-2024-06-13-010514 True False False 4h34m
kube-scheduler 4.17.0-0.nightly-2024-06-13-010514 True False False 4h34m
kube-storage-version-migrator 4.17.0-0.nightly-2024-06-13-010514 True False False 4h37m
machine-api 4.17.0-0.nightly-2024-06-13-010514 True False False 4h1m
machine-approver 4.17.0-0.nightly-2024-06-13-010514 True False False 4h37m
machine-config 4.17.0-0.nightly-2024-06-13-010514 True False False 4h36m
marketplace 4.17.0-0.nightly-2024-06-13-010514 True False False 4h37m
monitoring 4.17.0-0.nightly-2024-06-13-010514 True False False 3h59m
network 4.17.0-0.nightly-2024-06-13-010514 True False False 4h37m
node-tuning 4.17.0-0.nightly-2024-06-13-010514 True False False 4h1m
openshift-apiserver 4.17.0-0.nightly-2024-06-13-010514 True False False 4h21m
openshift-controller-manager 4.17.0-0.nightly-2024-06-13-010514 True False False 4h33m
openshift-samples 4.17.0-0.nightly-2024-06-13-010514 True False False 4h14m
operator-lifecycle-manager 4.17.0-0.nightly-2024-06-13-010514 True False False 4h36m
operator-lifecycle-manager-catalog 4.17.0-0.nightly-2024-06-13-010514 True False False 4h36m
operator-lifecycle-manager-packageserver 4.17.0-0.nightly-2024-06-13-010514 True False False 4h29m
service-ca 4.17.0-0.nightly-2024-06-13-010514 True False False 4h37m
storage 4.17.0-0.nightly-2024-06-13-010514 True False False 4h37m
$ oc -n openshift-console-operator get ep webhook
Error from server (NotFound): endpoints "webhook" not found$ oc -n openshift-console-operator get ep
NAME ENDPOINTS AGE
metrics 10.129.0.76:8443 159m
$ oc logs -n openshift-console-operator console-operator-74986bc5bd-lqcwq
E0613 20:06:35.149800 1 sync_v400.go:640] failed to get "logging-view-plugin" plugin: consoleplugin.console.openshift.io "logging-view-plugin" not found
I0613 20:06:35.152760 1 status_controller.go:218] clusteroperator/console diff {"status":{"conditions":[{"lastTransitionTime":"2024-06-13T19:09:45Z","message":"All is well","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2024-06-13T20:06:24Z","message":"SyncLoopRefreshProgressing: working toward v
ersion 4.17.0-0.nightly-2024-06-13-010514, 2 replicas available","reason":"SyncLoopRefresh_InProgress","status":"True","type":"Progressing"},{"lastTransitionTime":"2024-06-13T19:09:45Z","message":"All is well","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2024-06-13T19:06:36Z","message":"All is well",
"reason":"AsExpected","status":"True","type":"Upgradeable"},{"lastTransitionTime":"2024-06-13T18:47:04Z","reason":"NoData","status":"Unknown","type":"EvaluationConditionsDetected"}]}}
I0613 20:06:35.164372 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"a73074dc-1a4f-4ebb-b603-35a48fd649c2", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/console
changed: Progressing message changed from "SyncLoopRefreshProgressing: working toward version 4.17.0-0.nightly-2024-06-13-010514, 1 replicas available" to "SyncLoopRefreshProgressing: working toward version 4.17.0-0.nightly-2024-06-13-010514, 2 replicas available"
I0613 20:06:35.170238 1 helpers.go:201] Operator status changed: &v1.OperatorStatus{
ObservedGeneration: 4,
Conditions: []v1.OperatorCondition{
... // 30 identical elements
{Type: "RedirectServiceSyncProgressing", Status: "False", LastTransitionTime: {Time: s"2024-06-13 18:47:25 +0000 UTC"}},
{Type: "SyncLoopRefreshDegraded", Status: "False", LastTransitionTime: {Time: s"2024-06-13 19:06:36 +0000 UTC"}},
- {
- Type: "SyncLoopRefreshProgressing",
- Status: "True",
- LastTransitionTime: s"2024-06-13 20:06:24 +0000 UTC",
- Reason: "InProgress",
- Message: "working toward version 4.17.0-0.nightly-2024-06-13-010514, 2 replicas available",
- },
+ {
+ Type: "SyncLoopRefreshProgressing",
+ Status: "False",
+ LastTransitionTime: s"2024-06-13 20:06:35.168914426 +0000 UTC m=+4771.552484965",
+ },
{Type: "DownloadsDefaultRouteSyncUpgradeable", Status: "True", LastTransitionTime: {Time: s"2024-06-13 19:06:36 +0000 UTC"}},
{Type: "DownloadsCustomRouteSyncDegraded", Status: "False", LastTransitionTime: {Time: s"2024-06-13 18:47:26 +0000 UTC"}},
... // 27 identical elements
},
Version: "",
- ReadyReplicas: 2,
+ ReadyReplicas: 1,
Generations: {{Group: "apps", Resource: "deployments", Namespace: "openshift-console", Name: "downloads", ...}, {Group: "apps", Resource: "deployments", Namespace: "openshift-console", Name: "console", ...}},
}
E0613 20:06:35.181964 1 sync_v400.go:640] failed to get "logging-view-plugin" plugin: consoleplugin.console.openshift.io "logging-view-plugin" not found
I0613 20:06:35.333132 1 status_controller.go:218] clusteroperator/console diff {"status":{"conditions":[{"lastTransitionTime":"2024-06-13T19:09:45Z","message":"All is well","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2024-06-13T20:06:35Z","message":"All is well","reason":"AsExpected","status":
"False","type":"Progressing"},{"lastTransitionTime":"2024-06-13T19:09:45Z","message":"All is well","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2024-06-13T19:06:36Z","message":"All is well","reason":"AsExpected","status":"True","type":"Upgradeable"},{"lastTransitionTime":"2024-06-13T18:47:04Z","reaso
n":"NoData","status":"Unknown","type":"EvaluationConditionsDetected"}]}}
I0613 20:06:35.337473 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"a73074dc-1a4f-4ebb-b603-35a48fd649c2", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/console
changed: Progressing changed from True to False ("All is well")
E0613 20:06:35.531567 1 sync_v400.go:640] failed to get "logging-view-plugin" plugin: consoleplugin.console.openshift.io "logging-view-plugin" not found
E0613 20:06:41.249565 1 sync_v400.go:640] failed to get "logging-view-plugin" plugin: consoleplugin.console.openshift.io "logging-view-plugin" not found
E0613 20:06:47.929447 1 sync_v400.go:640] failed to get "logging-view-plugin" plugin: consoleplugin.console.openshift.io "logging-view-plugin" not found
E0613 20:06:51.157118 1 sync_v400.go:640] failed to get "logging-view-plugin" plugin: consoleplugin.console.openshift.io "logging-view-plugin" not found
E0613 20:07:04.094468 1 sync_v400.go:640] failed to get "logging-view-plugin" plugin: consoleplugin.console.openshift.io "logging-view-plugin" not found
E0613 20:07:04.124574 1 sync_v400.go:640] failed to get "logging-view-plugin" plugin: consoleplugin.console.openshift.io "logging-view-plugin" not found
E0613 20:07:04.135965 1 sync_v400.go:640] failed to get "logging-view-plugin" plugin: consoleplugin.console.openshift.io "logging-view-plugin" not found
E0613 20:07:04.294687 1 sync_v400.go:640] failed to get "logging-view-plugin" plugin: consoleplugin.console.openshift.io "logging-view-plugin" not found
I0613 20:07:04.898541 1 helpers.go:201] Operator status changed: &v1.OperatorStatus{
ObservedGeneration: 4,
Conditions: {{Type: "ConsoleNotificationSyncDegraded", Status: "False", LastTransitionTime: {Time: s"2024-06-13 18:47:04 +0000 UTC"}}, {Type: "ConsoleNotificationSyncProgressing", Status: "False", LastTransitionTime: {Time: s"2024-06-13 18:47:04 +0000 UTC"}}, {Type: "OAuthClientsControllerDegraded", Status: "False", Las
tTransitionTime: {Time: s"2024-06-13 19:06:36 +0000 UTC"}, Reason: "AsExpected", ...}, {Type: "PDBSyncProgressing", Status: "False", LastTransitionTime: {Time: s"2024-06-13 18:47:04 +0000 UTC"}}, ...},
Version: "",
- ReadyReplicas: 1,
+ ReadyReplicas: 2,
Generations: {{Group: "apps", Resource: "deployments", Namespace: "openshift-console", Name: "downloads", ...}, {Group: "apps", Resource: "deployments", Namespace: "openshift-console", Name: "console", ...}},
}
E0613 20:07:04.915150 1 sync_v400.go:640] failed to get "logging-view-plugin" plugin: consoleplugin.console.openshift.io "logging-view-plugin" not found
E0613 20:07:05.898422 1 sync_v400.go:640] failed to get "logging-view-plugin" plugin: consoleplugin.console.openshift.io "logging-view-plugin" not found
E0613 20:07:11.735341 1 sync_v400.go:640] failed to get "logging-view-plugin" plugin: consoleplugin.console.openshift.io "logging-view-plugin" not found
E0613 20:07:36.088993 1 sync_v400.go:640] failed to get "logging-view-plugin" plugin: consoleplugin.console.openshift.io "logging-view-plugin" not found