Loading...

Type: Bug
Resolution: Done
Priority: Undefined
Fix Version/s: 1.0.0
Affects Version/s: None
Component/s: None
Labels:
None

Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Intelligence Requested:
Market:

Sprint:
MON Sprint 262

Regression:
None

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Version of components:

OpenShift version: 4.16.0-0.nightly-2024-11-05-155328

Operator bundle: quay.io/rhobs/observability-operator-bundle:0.4.3-241105174753

Description of the issue:

When the operator flag

-openshift.enabled

is set in the operator CSV, the observability-operator pod fails to start with the following error:

% oc logs observability-operator-ddb44d588-9pdkg
2024-11-06T05:00:00Z INFO setup running with arguments {"namespace": "openshift-operators", "metrics-bind-address": ":8080", "images": "alertmanager=quay.io/prometheus/alertmanager:v0.26.0,prometheus=quay.io/prometheus/prometheus:v2.49.1,thanos=quay.io/thanos/thanos:v0.33.0", "openshift.enabled": true}
I1106 05:00:00.882367       1 event.go:389] "Event occurred" object="serving-cert::/etc/tls/private/tls.crt::/etc/tls/private/tls.key" fieldPath="" kind="" apiVersion="" type="Warning" reason="TLSConfigChanged" message="loaded serving cert [\"serving-cert::/etc/tls/private/tls.crt::/etc/tls/private/tls.key\"]: \"observability-operator.openshift-operators.svc\" [serving] validServingFor=[observability-operator.openshift-operators.svc,observability-operator.openshift-operators.svc.cluster.local] issuer=\"openshift-service-serving-signer@1730865408\" (2024-11-06 04:55:17 +0000 UTC to 2026-11-06 04:55:18 +0000 UTC (now=2024-11-06 05:00:00.88221642 +0000 UTC))"
E1106 05:00:00.980696       1 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:serviceaccount:openshift-operators:observability-operator-sa\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{serving-cert::/etc/tls/private/tls.crt::/etc/tls/private/tls.key.1805485e1f4014e3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:,Namespace:,Name:serving-cert::/etc/tls/private/tls.crt::/etc/tls/private/tls.key,UID:,APIVersion:,ResourceVersion:,FieldPath:,},Reason:TLSConfigChanged,Message:loaded serving cert [\"serving-cert::/etc/tls/private/tls.crt::/etc/tls/private/tls.key\"]: \"observability-operator.openshift-operators.svc\" [serving] validServingFor=[observability-operator.openshift-operators.svc,observability-operator.openshift-operators.svc.cluster.local] issuer=\"openshift-service-serving-signer@1730865408\" (2024-11-06 04:55:17 +0000 UTC to 2026-11-06 04:55:18 +0000 UTC (now=2024-11-06 05:00:00.88221642 +0000 UTC)),Source:EventSource{Component:cluster-observability-operator,Host:,},FirstTimestamp:2024-11-06 05:00:00.882242787 +0000 UTC m=+0.488516298,LastTimestamp:2024-11-06 05:00:00.882242787 +0000 UTC m=+0.488516298,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:cluster-observability-operator,ReportingInstance:,}"
2024-11-06T05:00:01Z ERROR setup cannot create a new operator {"error": "unable to register operator controller: one of For() or Named() must be called"}
main.main
 /workspace/cmd/operator/main.go:138
runtime.main
 /usr/local/go/src/runtime/proc.go:271

Steps to reproduce the issue:

1.Install the operator build off the latest upstream branch.

% operator-sdk run bundle \                  
    quay.io/rhobs/observability-operator-bundle:0.4.3-241105174753 \
    --install-mode AllNamespaces \
    --namespace openshift-operators

2.The operator will install successfully.

3.Set the openshift.enabled feature flag in the operator CSV.

              containers:
              - args:
                - --namespace=$(NAMESPACE)
                - --images=alertmanager=quay.io/prometheus/alertmanager:v0.26.0
                - --images=prometheus=quay.io/prometheus/prometheus:v2.49.1
                - --images=thanos=quay.io/thanos/thanos:v0.33.0
                - -openshift.enabled
                env:
                - name: NAMESPACE
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.namespace
                image: quay.io/rhobs/observability-operator:0.4.3-241105174753
                imagePullPolicy: Always
                livenessProbe:
                  httpGet:
                    path: /healthz
                    port: 8081
                name: operator

4.The operator pod fails to start.

% oc get pods
NAME                                                              READY   STATUS             RESTARTS        AGE
2ff553674ee74072cb97ef495e3f3ce4646270b66432bb25e98547aaf82f5x8   0/1     Completed          0               14m
amq-streams-cluster-operator-v2.7.0-7-65b9f44b7b-5ml94            1/1     Running            0               32m
istio-operator-5c676cd6d9-sd4nn                                   1/1     Running            0               32m
kiali-operator-5449f54985-bgrf8                                   1/1     Running            0               32m
obo-prometheus-operator-5bfc49689d-m8clt                          1/1     Running            0               14m
obo-prometheus-operator-admission-webhook-6968675b4c-9p84s        1/1     Running            0               14m
obo-prometheus-operator-admission-webhook-6968675b4c-gwgmk        1/1     Running            0               14m
observability-operator-66b7c4ff7d-d9xdb                           1/1     Running            0               14m
observability-operator-ddb44d588-9pdkg                            0/1     CrashLoopBackOff   6 (3m51s ago)   9m53s
quay-io-rhobs-observability-operator-bundle-0-4-3-241105174753    1/1     Running            0               14m
 
% oc logs observability-operator-ddb44d588-9pdkg
2024-11-06T05:05:36Z INFO setup running with arguments {"namespace": "openshift-operators", "metrics-bind-address": ":8080", "images": "alertmanager=quay.io/prometheus/alertmanager:v0.26.0,prometheus=quay.io/prometheus/prometheus:v2.49.1,thanos=quay.io/thanos/thanos:v0.33.0", "openshift.enabled": true}
I1106 05:05:36.382410       1 event.go:389] "Event occurred" object="serving-cert::/etc/tls/private/tls.crt::/etc/tls/private/tls.key" fieldPath="" kind="" apiVersion="" type="Warning" reason="TLSConfigChanged" message="loaded serving cert [\"serving-cert::/etc/tls/private/tls.crt::/etc/tls/private/tls.key\"]: \"observability-operator.openshift-operators.svc\" [serving] validServingFor=[observability-operator.openshift-operators.svc,observability-operator.openshift-operators.svc.cluster.local] issuer=\"openshift-service-serving-signer@1730865408\" (2024-11-06 04:55:17 +0000 UTC to 2026-11-06 04:55:18 +0000 UTC (now=2024-11-06 05:05:36.381724252 +0000 UTC))"
E1106 05:05:36.585918       1 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:serviceaccount:openshift-operators:observability-operator-sa\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{serving-cert::/etc/tls/private/tls.crt::/etc/tls/private/tls.key.180548ac3c940554  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:,Namespace:,Name:serving-cert::/etc/tls/private/tls.crt::/etc/tls/private/tls.key,UID:,APIVersion:,ResourceVersion:,FieldPath:,},Reason:TLSConfigChanged,Message:loaded serving cert [\"serving-cert::/etc/tls/private/tls.crt::/etc/tls/private/tls.key\"]: \"observability-operator.openshift-operators.svc\" [serving] validServingFor=[observability-operator.openshift-operators.svc,observability-operator.openshift-operators.svc.cluster.local] issuer=\"openshift-service-serving-signer@1730865408\" (2024-11-06 04:55:17 +0000 UTC to 2026-11-06 04:55:18 +0000 UTC (now=2024-11-06 05:05:36.381724252 +0000 UTC)),Source:EventSource{Component:cluster-observability-operator,Host:,},FirstTimestamp:2024-11-06 05:05:36.38173218 +0000 UTC m=+0.595819936,LastTimestamp:2024-11-06 05:05:36.38173218 +0000 UTC m=+0.595819936,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:cluster-observability-operator,ReportingInstance:,}"
2024-11-06T05:05:36Z ERROR setup cannot create a new operator {"error": "unable to register operator controller: one of For() or Named() must be called"}
main.main
 /workspace/cmd/operator/main.go:138
runtime.main
 /usr/local/go/src/runtime/proc.go:271

links to

rhobs/observability-operator#618: fix: give operator controller a name

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates

PagerDuty