Loading...

XML

Word

Printable

Type: Bug
Resolution: Unresolved
Priority: Critical
Fix Version/s: None
Affects Version/s: odf-4.16
Component/s: odf-dr/multicluster-orchestrator
Labels:
None

Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
BZ Internal Whiteboard:
4.16.4
Bugzilla Bug:
RHBZ: 2293611
Dev Approval:
Committed
PM Approval:
Targeted
QE Approval:
?
Release Note Text:

Hide
.Instability of the token-exchange-agent pod after upgrade

The token-exchange-agent pod on the managed cluster is unstable as the old deployment resources are not cleaned up properly. This might cause application failover action to fail.

Workaround: Refer the knowledgebase article, "token-exchange-agent" pod on managed cluster is unstable after upgrade to ODF 4.16.0.

Result: If the workaround is followed, "token-exchange-agent" pod is stabilized and failover action works as expected.

Show
.Instability of the token-exchange-agent pod after upgrade The token-exchange-agent pod on the managed cluster is unstable as the old deployment resources are not cleaned up properly. This might cause application failover action to fail. Workaround: Refer the knowledgebase article, "token-exchange-agent" pod on managed cluster is unstable after upgrade to ODF 4.16.0. Result: If the workaround is followed, "token-exchange-agent" pod is stabilized and failover action works as expected.
Release Note Type:
Known Issue
Target Release:

odf-4.16.4
Intelligence Requested:
Market:

Regression:
None

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Description of problem (please be detailed as possible and provide log
snippests):
On a RDR setup after upgrading the cluster from 4.15.3 to 4.16.0-130, it has been observed that the "token-exchange-agent" pod on managed cluster is unstable. The pod crashes few times, with a new pod spinning up to replace it. Sometimes there is no crash, but still new pod comes up. This cycle continues indefinitely.

> Case 1 (pod logs in Additional Info section):
token-exchange-agent-6d994bf479-7w545 0/1 CrashLoopBackOff 3 (39s ago) 9m25s
–
token-exchange-agent-6d994bf479-7w545 0/1 Running 4 (71s ago) 9m57s
token-exchange-agent-76cf9f4f9b-47mcm 0/1 ContainerCreating 0 1s
–
token-exchange-agent-6d994bf479-7w545 0/1 Running 4 (75s ago) 10m
token-exchange-agent-76cf9f4f9b-47mcm 0/1 Running 0 5s
–
token-exchange-agent-76cf9f4f9b-47mcm 1/1 Running 0 119s

> Case 2:

token-exchange-agent-5c59bb74cb-g2hjs 1/1 Running 0 11m
token-exchange-agent-5d559f946c-x77jh 0/1 Running 0 59s
–
token-exchange-agent-5d559f946c-x77jh 1/1 Running 0 3m26s

Version of all relevant components (if applicable):
ODF: 4.16.0-130.stable
OCP: 4.16.0-0.nightly-2024-06-19-080432
ACM: 2.10.3
Submariner: 0.17.1
Gitops: 1.12.3
Volsync: 0.9.1
OADP: 1.3.2

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?

Is there any workaround available to the best of your knowledge?

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
2

Can this issue reproducible?

Can this issue reproduce from the UI?

If this is a regression, please provide more details to justify this:

Steps to Reproduce:
1. Configure RDR setup using OCP 4.15 and ODF 4.15
2. Perform upgrade
3. Observe the token-exchange-agent pod on managed clusters

Actual results:
Pod crashes and restarts

Expected results:
Pod should be stable and doesn't crash or restart indefinitely

Additional Info:

$ oc get csv -n openshift-storage
NAME DISPLAY VERSION REPLACES PHASE
mcg-operator.v4.16.0-130.stable NooBaa Operator 4.16.0-130.stable mcg-operator.v4.15.3-rhodf Succeeded
ocs-client-operator.v4.16.0-130.stable OpenShift Data Foundation Client 4.16.0-130.stable Succeeded
ocs-operator.v4.16.0-130.stable OpenShift Container Storage 4.16.0-130.stable ocs-operator.v4.15.3-rhodf Succeeded
odf-csi-addons-operator.v4.16.0-130.stable CSI Addons 4.16.0-130.stable odf-csi-addons-operator.v4.15.3-rhodf Succeeded
odf-operator.v4.16.0-130.stable OpenShift Data Foundation 4.16.0-130.stable odf-operator.v4.15.3-rhodf Succeeded
odf-prometheus-operator.v4.16.0-130.stable Prometheus Operator 4.16.0-130.stable Succeeded
odr-cluster-operator.v4.16.0-130.stable Openshift DR Cluster Operator 4.16.0-130.stable odr-cluster-operator.v4.15.3-rhodf Succeeded
openshift-gitops-operator.v1.12.3 Red Hat OpenShift GitOps 1.12.3 openshift-gitops-operator.v1.12.2 Succeeded
recipe.v4.16.0-130.stable Recipe 4.16.0-130.stable Succeeded
rook-ceph-operator.v4.16.0-130.stable Rook-Ceph 4.16.0-130.stable Succeeded
volsync-product.v0.9.1 VolSync 0.9.1 volsync-product.v0.8.2 Succeeded

token-exchange-agent Pod logs when there is a crash (case 1):

I0621 08:37:09.389256 I0621 08:37:09.389652 I0621 08:37:09.391274 I0621 08:37:09.391347 I0621 08:37:09.391397 I0621 08:37:09.391555 I0621 08:37:09.404192 W0621 08:37:09.418144 E0621 08:37:09.418172 W0621 08:37:09.420068 E0621 08:37:09.420104 I0621 08:37:09.604552 I0621 08:37:09.605690 I0621 08:37:09.810062 I0621 08:37:09.820636 I0621 08:37:09.820735 I0621 08:37:09.832760 I0621 08:37:09.832933 I0621 08:37:09.855429 I0621 08:37:09.855558 I0621 08:37:09.897718 I0621 08:37:09.897832 I0621 08:37:09.915274 I0621 08:37:09.927227 I0621 08:37:09.949149 I0621 08:37:09.980000 I0621 08:37:09.980141 I0621 08:37:09.991036 I0621 08:37:10.073710 I0621 08:37:10.142209 I0621 08:37:10.142389 I0621 08:37:10.236610 W0621 08:37:10.340115 E0621 08:37:10.340142 I0621 08:37:10.465114 I0621 08:37:10.465250 I0621 08:37:10.559231 W0621 08:37:10.769452 E0621 08:37:10.769483 I0621 08:37:11.107172 I0621 08:37:11.107294 I0621 08:37:11.201198 I0621 08:37:12.389177 I0621 08:37:12.389363 I0621 08:37:12.483282 W0621 08:37:13.540994 E0621 08:37:13.541096 W0621 08:37:13.562591 E0621 08:37:13.562620 I0621 08:37:14.952517 I0621 08:37:14.952670 I0621 08:37:15.046223 W0621 08:37:18.724804 E0621 08:37:18.724835 W0621 08:37:19.468938 E0621 08:37:19.468987 I0621 08:37:20.076072 I0621 08:37:20.076194 I0621 08:37:20.168655 W0621 08:37:28.616033 E0621 08:37:28.616074 I0621 08:37:30.319051 I0621 08:37:30.319208 I0621 08:37:30.411292 W0621 08:37:30.572544 E0621 08:37:30.572578 W0621 08:37:47.003710 E0621 08:37:47.003739 W0621 08:37:47.713311 E0621 08:37:47.713344 & Detected at:
> goroutine > runtime/debug.Stack()
> /usr/lib/gola > sigs.k8s.io/co > /remote-sourc > sigs.k8s.io/co 1 manager.go:112] Starting addon agents.
1 common.go:40] Health probes server is running.
1 manager.go:196] starting hub controller manager
1 manager.go:309] starting spoke controller manager
1 manager.go:240] Waiting for MaintenanceMode CRD to be created. MaintenanceMode controller is not running yet.
1 manager.go:275] Starting lease updater.
1 manager.go:262] MaintenanceMode CRD exists. MaintenanceMode controller is now running.
1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: Failed to watch *v1alpha1.MirrorPeer: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
1 rook_secret_handler.go:67] detected cluster type: Converged
1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
1 rook_secret_handler.go:67] detected cluster type: Converged
1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
1 rook_secret_handler.go:67] detected cluster type: Converged
1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
1 rook_secret_handler.go:67] detected cluster type: Converged
1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
1 rook_secret_handler.go:67] detected cluster type: Converged
1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
1 rook_secret_handler.go:67] detected cluster type: Converged
1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
1 rook_secret_handler.go:67] detected cluster type: Converged
1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: Failed to watch *v1alpha1.MirrorPeer: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
1 rook_secret_handler.go:67] detected cluster type: Converged
1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
1 rook_secret_handler.go:67] detected cluster type: Converged
1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
1 rook_secret_handler.go:67] detected cluster type: Converged
1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: Failed to watch *v1alpha1.MirrorPeer: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
1 rook_secret_handler.go:67] detected cluster type: Converged
1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: Failed to watch *v1alpha1.MirrorPeer: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
1 rook_secret_handler.go:67] detected cluster type: Converged
1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
1 rook_secret_handler.go:67] detected cluster type: Converged
1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: Failed to watch *v1alpha1.MirrorPeer: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: Failed to watch *v1alpha1.MirrorPeer: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
#91;controller-runtime] log.SetLogger(...) was never called; logs will not be displayed.
155 [running]:
ng/src/runtime/debug/stack.go:24 +0x5e
ntroller-runtime/pkg/log.eventuallyFulfillRoot()
e/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/log/log.go:60 +0xcd
ntroller-runtime/pkg/log.(*delegatingLogSink).WithValues(0xc000128200,

{0xc000d163c0, 0x4, 0x4}

)
> /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/log/deleg.go:168 +0x49
> github.com/go-logr/logr.Logger.WithValues(...)
> /remote-source/deps/gomod/pkg/mod/github.com/go-logr/logr@v1.4.1/logr.go:332
> sigs.k8s.io/controller-runtime/pkg/builder.(*Builder).doController.func1(0xc000a839a0)
> /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/builder/controller.go:402 +0x2ba
> sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000280f00,

{0x27802a0, 0xc0006e4370}, {0x212b280?, 0xc000a83140?})
> /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:306 +0x16a
> sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000280f00, {0x27802a0, 0xc0006e4370}

)
> /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266 +0x1af
> sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
> /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227 +0x79
> created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 147
> /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:223 +0x565
I0621 08:37:50.800948 1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
I0621 08:37:50.801066 1 rook_secret_handler.go:67] detected cluster type: Converged
I0621 08:37:50.893760 1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
W0621 08:38:26.849200 1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
E0621 08:38:26.849235 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: Failed to watch *v1alpha1.MirrorPeer: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
W0621 08:38:30.087863 1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
E0621 08:38:30.087895 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
I0621 08:38:31.763595 1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
I0621 08:38:31.763732 1 rook_secret_handler.go:67] detected cluster type: Converged
I0621 08:38:31.856349 1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
W0621 08:39:02.893120 1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
E0621 08:39:02.893155 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
W0621 08:39:03.565503 1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
E0621 08:39:03.565532 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.2/tools/cache/reflector.go:229: Failed to watch *v1alpha1.MirrorPeer: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
E0621 08:39:09.392659 1 manager.go:198] failed to wait for greensecret_controller caches to sync: timed out waiting for cache to be synced for Kind *v1.Secretproblem running hub controller manager

Assignee:: Umanga Chapagain

Reporter:: Sidhant Agrawal

QA Contact:: Sidhant Agrawal

Votes:: 0 Vote for this issue

Watchers:: 17 Start watching this issue

Created:: 2024/06/21 8:50 AM

Updated:: 2024/11/25 2:31 PM

Details

Description

Attachments

Easy Agile Planning Poker

Activity

People

Dates

PagerDuty