Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-61667

ignition-server pods are frequently restarting

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Moderate
    • None
    • None
    • None
    • None
    • Proposed
    • Bug Fix
    • Hide
      Before this update, the ignition server deployment used a global `mirroredReleaseImage` state that could be modified by concurrent image lookup operations from different reconciliation processes, causing race conditions. As a consequence, the `MIRRORED_RELEASE_IMAGE` environment variable alternated between the original image and its mirror registry, triggering constant deployment regenerations. In this release, the global mirror state is replaced with image-specific lookup logic that deterministically resolves mirrors based on the specific control plane release image and registry policies, with defensive filtering for empty registry entries. As a result, the ignition server deployments remain stable with consistent `MIRRORED_RELEASE_IMAGE` values, eliminating unnecessary pod restarts and deployment churn.
      Show
      Before this update, the ignition server deployment used a global `mirroredReleaseImage` state that could be modified by concurrent image lookup operations from different reconciliation processes, causing race conditions. As a consequence, the `MIRRORED_RELEASE_IMAGE` environment variable alternated between the original image and its mirror registry, triggering constant deployment regenerations. In this release, the global mirror state is replaced with image-specific lookup logic that deterministically resolves mirrors based on the specific control plane release image and registry policies, with defensive filtering for empty registry entries. As a result, the ignition server deployments remain stable with consistent `MIRRORED_RELEASE_IMAGE` values, eliminating unnecessary pod restarts and deployment churn.
    • None
    • None
    • None
    • None

      This is a clone of issue OCPBUGS-60185. The following is the description of the original issue:
      —
      Issue or Concern :

      ignition-server pods are frequently restarting

      Steps Action Taken :

      • The deployment for the ignition-server for the hosted-cluster is getting updated frequently and the status.conditions are flapping
      • The env for the ignition-server container is getting updated frequently: 
                  value: quay.io/openshift-release-dev/ocp-release:4.18.18-x86_64@sha256:eca2e3f7de2bd92b18f69547c8f0ad842fdb83f0821f76b8692f2716a86b0bde

        and the imageDigestMirrors for quay.io 

                  value: pull-through-proxy-quay-io.apps.osv1.example.com/openshift-release-dev/ocp-release:4.18.18-x86_64@sha256:eca2e3f7de2bd92b18f69547c8f0ad842fdb83f0821f76b8692f2716a86b0bde
      • In the managedFields we can see that the hypershift-controlplane-manager is updating the Deployment ignition-server around 20 times in 5 minutes, which is leading to continuous restarts

              jparrill@redhat.com Juan Manuel Parrilla Madrid
              rhn-support-anowak Andreas Nowak
              None
              None
              XiuJuan Wang XiuJuan Wang
              None
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated: