Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-36862

4.16 "Bad" reconciliation loops can cause unbounded dockercfg secret creation

XMLWordPrintable

    • Critical
    • Yes
    • Proposed
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, if the `openshift.io/internal-registry-pull-secret-ref` annotation was removed from a `ServiceAccount`, {product-title} recreated the deleted annotation and created a new managed image pull `Secret`. This contention could cause the cluster to get overloaded with image pull `Secrets`. With this release, {product-title} attempts to reclaim managed image pull `Secrets` that were previously referenced and deletes managed image pull `Secrets` that remain orphaned after reconciliation. (link:https://issues.redhat.com/browse/OCPBUGS-36862[*OCPBUGS-36862*])
      ------------------------------------------------------

      Previously, if the `openshift.io/internal-registry-pull-secret-ref` annotation was removed from a `ServiceAccount`, usually by a misbehaving operator or reconciliation loop, OpenShift Container Platform would recreate the deleted annotation and create a new managed image pull `Secret`. This contention could cause the cluster to get overloaded with image pull `Secrets`. With this release, OpenShift Container Platform attempts to reclaim managed image pull `Secrets` previously referenced by the annotation when attempting to re-create the annotation after it has been deleted. Additionally, managed image pull `Secrets` that remain orphaned after reconciliation will be deleted.
      Show
      * Previously, if the `openshift.io/internal-registry-pull-secret-ref` annotation was removed from a `ServiceAccount`, {product-title} recreated the deleted annotation and created a new managed image pull `Secret`. This contention could cause the cluster to get overloaded with image pull `Secrets`. With this release, {product-title} attempts to reclaim managed image pull `Secrets` that were previously referenced and deletes managed image pull `Secrets` that remain orphaned after reconciliation. (link: https://issues.redhat.com/browse/OCPBUGS-36862 [* OCPBUGS-36862 *]) ------------------------------------------------------ Previously, if the `openshift.io/internal-registry-pull-secret-ref` annotation was removed from a `ServiceAccount`, usually by a misbehaving operator or reconciliation loop, OpenShift Container Platform would recreate the deleted annotation and create a new managed image pull `Secret`. This contention could cause the cluster to get overloaded with image pull `Secrets`. With this release, OpenShift Container Platform attempts to reclaim managed image pull `Secrets` previously referenced by the annotation when attempting to re-create the annotation after it has been deleted. Additionally, managed image pull `Secrets` that remain orphaned after reconciliation will be deleted.
    • Bug Fix
    • In Progress

      This is a clone of issue OCPBUGS-36833. The following is the description of the original issue:
      —
      Description of problem:

      In 4,16 OCP starts to place an annotation on service accounts when it creates a dockercfg secret. Some operators/reconciliation loops (incorrectly) will then try to set the annotation on the SA back to exactly what they wanted. OCP will annotate again and create a new secret. Operators sets it back without annotation. Rinse Repeat.

Eventually etcd will get completely overloaded with secrets, will start to OOM, and the entire cluster will come down.

       

      There is belief that at least otel, tempo, acm, odf/ocs, strymzi, elasticsearch and possibly other operators reconciled the annoations on the SA by setting them back exactly how they wanted them set.

       

      These seem to be related (but no complete)

      https://issues.redhat.com/browse/LOG-5776

      https://issues.redhat.com/browse/ENTMQST-6129

      https://issues.redhat.com/browse/TRACING-4435

      https://issues.redhat.com/browse/ACM-10987

              rh-ee-sabiswas Sayan Biswas
              openshift-crt-jira-prow OpenShift Prow Bot
              Ke Wang Ke Wang
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

                Created:
                Updated:
                Resolved: