Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-61677

ignition-server pods are frequently restarting

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • None
    • 4.18.z
    • HyperShift
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Moderate
    • None
    • None
    • None
    • None
    • Done
    • Bug Fix
    • Hide
      Before this update, the ignition server deployment used a global `mirroredReleaseImage` state that could be modified by concurrent image lookup operations, which caused race conditions. As a consequence, the MIRRORED_RELEASE_IMAGE environment variable flipped between the original image and its mirror registry, which triggered constant deployment regenerations. With this release, the global mirror state is replaced with an image-specific lookup logic, which ensures deterministic mirror resolution and eliminates defensive filtering for empty registry entries. As a result, ignition server deployments remain stable with consistent MIRRORED_RELEASE_IMAGE values, which eliminates unnecessary pod restarts and deployment churn. (link:https://issues.redhat.com/browse/OCPBUGS-61677[OCPBUGS-61677])
      Show
      Before this update, the ignition server deployment used a global `mirroredReleaseImage` state that could be modified by concurrent image lookup operations, which caused race conditions. As a consequence, the MIRRORED_RELEASE_IMAGE environment variable flipped between the original image and its mirror registry, which triggered constant deployment regenerations. With this release, the global mirror state is replaced with an image-specific lookup logic, which ensures deterministic mirror resolution and eliminates defensive filtering for empty registry entries. As a result, ignition server deployments remain stable with consistent MIRRORED_RELEASE_IMAGE values, which eliminates unnecessary pod restarts and deployment churn. (link: https://issues.redhat.com/browse/OCPBUGS-61677 [ OCPBUGS-61677 ])
    • None
    • None
    • None
    • None

      This is a clone of issue OCPBUGS-61667. The following is the description of the original issue:
      —
      This is a clone of issue OCPBUGS-60185. The following is the description of the original issue:
      —
      Issue or Concern :

      ignition-server pods are frequently restarting

      Steps Action Taken :

      • The deployment for the ignition-server for the hosted-cluster is getting updated frequently and the status.conditions are flapping
      • The env for the ignition-server container is getting updated frequently: 
                  value: quay.io/openshift-release-dev/ocp-release:4.18.18-x86_64@sha256:eca2e3f7de2bd92b18f69547c8f0ad842fdb83f0821f76b8692f2716a86b0bde

        and the imageDigestMirrors for quay.io 

                  value: pull-through-proxy-quay-io.apps.osv1.example.com/openshift-release-dev/ocp-release:4.18.18-x86_64@sha256:eca2e3f7de2bd92b18f69547c8f0ad842fdb83f0821f76b8692f2716a86b0bde
      • In the managedFields we can see that the hypershift-controlplane-manager is updating the Deployment ignition-server around 20 times in 5 minutes, which is leading to continuous restarts

              jparrill@redhat.com Juan Manuel Parrilla Madrid
              rhn-support-anowak Andreas Nowak
              None
              None
              XiuJuan Wang XiuJuan Wang
              None
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: