Loading...

XML

Word

Printable

Type: Bug
Resolution: Done-Errata
Priority: Critical
Fix Version/s: None
Affects Version/s: 4.16, 4.16.0, 4.17, 4.16.z
Component/s: openshift-controller-manager / controller-manager
Labels:

Severity:
Critical
Regression:
Yes
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Target Version:

4.17.0
Target Backport Versions:

4.16.z

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

PX Impact Score:
PX Scheduling Request:
PX Priority Data:
PX Impact Range:
PX Review Complete:
PX Technical Impact:

Description of problem:

In 4,16 OCP starts to place an annotation on service accounts when it creates a dockercfg secret. Some operators/reconciliation loops (incorrectly) will then try to set the annotation on the SA back to exactly what they wanted. OCP will annotate again and create a new secret. Operators sets it back without annotation. Rinse Repeat.

Eventually etcd will get completely overloaded with secrets, will start to OOM, and the entire cluster will come down.

There is belief that at least otel, tempo, acm, odf/ocs, strymzi, elasticsearch and possibly other operators reconciled the annoations on the SA by setting them back exactly how they wanted them set.

These seem to be related (but no complete)

https://issues.redhat.com/browse/LOG-5776

https://issues.redhat.com/browse/ENTMQST-6129

https://issues.redhat.com/browse/TRACING-4435

https://issues.redhat.com/browse/ACM-10987

blocks

OCPBUGS-36862 4.16 "Bad" reconciliation loops can cause unbounded dockercfg secret creation