Loading...

Type: Bug
Resolution: Won't Do
Priority: Undefined
Fix Version/s: None
Affects Version/s: 0.3.0
Component/s: None
Labels:
None

Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Intelligence Requested:
Market:

Regression:
None

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

After applying the example from https://docs.openshift.com/container-platform/4.16/observability/cluster_observability_operator/configuring-the-cluster-observability-operator-to-monitor-a-service.html, the observability-operator pod starts to crashloop with the following errors.

W0709 15:59:26.186117       1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.1/tools/cache/reflector.go:232: failed to list *v1alpha1.ConsolePlugin: conversion webhook for console.openshift.io/v1, Kind=ConsolePlugin failed: Post "https://webhook.openshift-console-operator.svc:9443/crdconvert?timeout=30s": service "webhook" not found
E0709 15:59:26.186149       1 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.1/tools/cache/reflector.go:232: Failed to watch *v1alpha1.ConsolePlugin: failed to list *v1alpha1.ConsolePlugin: conversion webhook for console.openshift.io/v1, Kind=ConsolePlugin failed: Post "https://webhook.openshift-console-operator.svc:9443/crdconvert?timeout=30s": service "webhook" not found
2024-07-09T15:59:26Z	INFO	thanos-querier	watched MonitoringStack changed, checking for matching querier	 {"Monitoring Stack": "obo-test/example-coo-monitoring-stack"}
I0709 15:59:27.085910       1 trace.go:236] Trace[853298249]: "DeltaFIFO Pop Process" ID:openshift-console/console-operator,Depth:266,Reason:slow event handlers blocking the queue (09-Jul-2024 15:59:26.886) (total time: 199ms):
Trace[853298249]: [199.295885ms] [199.295885ms] END
I0709 15:59:27.387593       1 trace.go:236] Trace[817858043]: "DeltaFIFO Pop Process" ID:openshift-infra/system:deployers,Depth:226,Reason:slow event handlers blocking the queue (09-Jul-2024 15:59:27.086) (total time: 301ms):
Trace[817858043]: [301.504545ms] [301.504545ms] END
W0709 15:59:27.390290       1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.1/tools/cache/reflector.go:232: failed to list *v1alpha1.ConsolePlugin: conversion webhook for console.openshift.io/v1, Kind=ConsolePlugin failed: Post "https://webhook.openshift-console-operator.svc:9443/crdconvert?timeout=30s": service "webhook" not found
E0709 15:59:27.390306       1 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.1/tools/cache/reflector.go:232: Failed to watch *v1alpha1.ConsolePlugin: failed to list *v1alpha1.ConsolePlugin: conversion webhook for console.openshift.io/v1, Kind=ConsolePlugin failed: Post "https://webhook.openshift-console-operator.svc:9443/crdconvert?timeout=30s": service "webhook" not found
W0709 15:59:29.473627       1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.1/tools/cache/reflector.go:232: failed to list *v1alpha1.ConsolePlugin: conversion webhook for console.openshift.io/v1, Kind=ConsolePlugin failed: Post "https://webhook.openshift-console-operator.svc:9443/crdconvert?timeout=30s": service "webhook" not found
E0709 15:59:29.473659       1 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.1/tools/cache/reflector.go:232: Failed to watch *v1alpha1.ConsolePlugin: failed to list *v1alpha1.ConsolePlugin: conversion webhook for console.openshift.io/v1, Kind=ConsolePlugin failed: Post "https://webhook.openshift-console-operator.svc:9443/crdconvert?timeout=30s": service "webhook" not found
W0709 15:59:35.192013       1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.1/tools/cache/reflector.go:232: failed to list *v1alpha1.ConsolePlugin: conversion webhook for console.openshift.io/v1, Kind=ConsolePlugin failed: Post "https://webhook.openshift-console-operator.svc:9443/crdconvert?timeout=30s": service "webhook" not found
E0709 15:59:35.192039       1 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.1/tools/cache/reflector.go:232: Failed to watch *v1alpha1.ConsolePlugin: failed to list *v1alpha1.ConsolePlugin: conversion webhook for console.openshift.io/v1, Kind=ConsolePlugin failed: Post "https://webhook.openshift-console-operator.svc:9443/crdconvert?timeout=30s": service "webhook" not found
W0709 15:59:44.926859       1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.1/tools/cache/reflector.go:232: failed to list *v1alpha1.ConsolePlugin: conversion webhook for console.openshift.io/v1, Kind=ConsolePlugin failed: Post "https://webhook.openshift-console-operator.svc:9443/crdconvert?timeout=30s": service "webhook" not found
E0709 15:59:44.926884       1 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.1/tools/cache/reflector.go:232: Failed to watch *v1alpha1.ConsolePlugin: failed to list *v1alpha1.ConsolePlugin: conversion webhook for console.openshift.io/v1, Kind=ConsolePlugin failed: Post "https://webhook.openshift-console-operator.svc:9443/crdconvert?timeout=30s": service "webhook" not found
W0709 15:59:59.204677       1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.1/tools/cache/reflector.go:232: failed to list *v1alpha1.ConsolePlugin: conversion webhook for console.openshift.io/v1, Kind=ConsolePlugin failed: Post "https://webhook.openshift-console-operator.svc:9443/crdconvert?timeout=30s": service "webhook" not found
E0709 15:59:59.204704       1 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.1/tools/cache/reflector.go:232: Failed to watch *v1alpha1.ConsolePlugin: failed to list *v1alpha1.ConsolePlugin: conversion webhook for console.openshift.io/v1, Kind=ConsolePlugin failed: Post "https://webhook.openshift-console-operator.svc:9443/crdconvert?timeout=30s": service "webhook" not found
W0709 16:00:31.533868       1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.1/tools/cache/reflector.go:232: failed to list *v1alpha1.ConsolePlugin: conversion webhook for console.openshift.io/v1, Kind=ConsolePlugin failed: Post "https://webhook.openshift-console-operator.svc:9443/crdconvert?timeout=30s": service "webhook" not found
E0709 16:00:31.533906       1 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.1/tools/cache/reflector.go:232: Failed to watch *v1alpha1.ConsolePlugin: failed to list *v1alpha1.ConsolePlugin: conversion webhook for console.openshift.io/v1, Kind=ConsolePlugin failed: Post "https://webhook.openshift-console-operator.svc:9443/crdconvert?timeout=30s": service "webhook" not found
W0709 16:01:07.059424       1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.1/tools/cache/reflector.go:232: failed to list *v1alpha1.ConsolePlugin: conversion webhook for console.openshift.io/v1, Kind=ConsolePlugin failed: Post "https://webhook.openshift-console-operator.svc:9443/crdconvert?timeout=30s": service "webhook" not found
E0709 16:01:07.059461       1 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.1/tools/cache/reflector.go:232: Failed to watch *v1alpha1.ConsolePlugin: failed to list *v1alpha1.ConsolePlugin: conversion webhook for console.openshift.io/v1, Kind=ConsolePlugin failed: Post "https://webhook.openshift-console-operator.svc:9443/crdconvert?timeout=30s": service "webhook" not found
2024-07-09T16:01:26Z	ERROR	Could not wait for Cache to sync	{"controller": "monitoringstack", "controllerGroup": "monitoring.rhobs", "controllerKind": "MonitoringStack", "error": "failed to wait for monitoringstack caches to sync: timed out waiting for cache to be synced for Kind *v1alpha1.MonitoringStack"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.1
	/remote-source/observability-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.2/pkg/internal/controller/controller.go:198
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
	/remote-source/observability-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.2/pkg/internal/controller/controller.go:203
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
	/remote-source/observability-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.2/pkg/internal/controller/controller.go:229
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
	/remote-source/observability-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.2/pkg/manager/runnable_group.go:226
2024-07-09T16:01:26Z	ERROR	Could not wait for Cache to sync	{"controller": "uiplugin", "controllerGroup": "observability.openshift.io", "controllerKind": "UIPlugin", "error": "failed to wait for uiplugin caches to sync: timed out waiting for cache to be synced for Kind *v1alpha1.UIPlugin"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.1
	/remote-source/observability-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.2/pkg/internal/controller/controller.go:198
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
	/remote-source/observability-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.2/pkg/internal/controller/controller.go:203
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
	/remote-source/observability-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.2/pkg/internal/controller/controller.go:229
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
	/remote-source/observability-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.2/pkg/manager/runnable_group.go:226
2024-07-09T16:01:26Z	ERROR	Could not wait for Cache to sync	{"controller": "thanosquerier", "controllerGroup": "monitoring.rhobs", "controllerKind": "ThanosQuerier", "error": "failed to wait for thanosquerier caches to sync: timed out waiting for cache to be synced for Kind *v1alpha1.ThanosQuerier"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.1
	/remote-source/observability-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.2/pkg/internal/controller/controller.go:198
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
	/remote-source/observability-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.2/pkg/internal/controller/controller.go:203
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
	/remote-source/observability-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.2/pkg/internal/controller/controller.go:229
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
	/remote-source/observability-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.2/pkg/manager/runnable_group.go:226
2024-07-09T16:01:26Z	INFO	Stopping and waiting for non leader election runnables
2024-07-09T16:01:26Z	INFO	Stopping and waiting for leader election runnables
2024-07-09T16:01:26Z	ERROR	error received after stop sequence was engaged	{"error": "failed to wait for uiplugin caches to sync: timed out waiting for cache to be synced for Kind *v1alpha1.UIPlugin"}
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).engageStopProcedure.func1
	/remote-source/observability-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.2/pkg/manager/internal.go:499
2024-07-09T16:01:26Z	ERROR	error received after stop sequence was engaged	{"error": "failed to wait for thanosquerier caches to sync: timed out waiting for cache to be synced for Kind *v1alpha1.ThanosQuerier"}
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).engageStopProcedure.func1
	/remote-source/observability-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.2/pkg/manager/internal.go:499
2024-07-09T16:01:26Z	INFO	Stopping and waiting for caches
2024-07-09T16:01:26Z	ERROR	controller-runtime.source.EventHandler	failed to get informer from cache	{"error": "Timeout: failed waiting for *v1alpha1.ConsolePlugin Informer to sync"}
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1
	/remote-source/observability-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.2/pkg/internal/source/kind.go:76
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func1
	/remote-source/observability-operator/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.30.1/pkg/util/wait/loop.go:53
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
	/remote-source/observability-operator/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.30.1/pkg/util/wait/loop.go:54
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
	/remote-source/observability-operator/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.30.1/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1
	/remote-source/observability-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.2/pkg/internal/source/kind.go:64
2024-07-09T16:01:26Z	INFO	Stopping and waiting for webhooks
2024-07-09T16:01:26Z	INFO	Stopping and waiting for HTTP servers
2024-07-09T16:01:26Z	INFO	controller-runtime.metrics	Shutting down metrics server with timeout of 1 minute
2024-07-09T16:01:26Z	INFO	shutting down server	{"name": "health probe", "addr": "[::]:8081"}
2024-07-09T16:01:26Z	INFO	Wait completed, proceeding to shutdown the manager
2024-07-09T16:01:26Z	ERROR	setup	terminating	{"error": "unable to start manager: failed to wait for monitoringstack caches to sync: timed out waiting for cache to be synced for Kind *v1alpha1.MonitoringStack"}
main.main
	/remote-source/observability-operator/app/cmd/operator/main.go:138
runtime.main
	/usr/lib/golang/src/runtime/proc.go:271

Tested on OCP 4.16 and OCP 4.17 nightly.

duplicates

COO-225 observability-operator pod failed to to list *v1alpha1.ConsolePlugin on OCP4.17

Closed

Details

Description

Attachments

Issue Links

Activity

People

Dates

PagerDuty