Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-29003

Default Internal Registry cleans custom images stored on it from 4.13 to 4.14

    XMLWordPrintable

Details

    • Important
    • Yes
    • False
    • Hide

      None

      Show
      None
    • Release Note Not Required
    • In Progress

    Description

      Description of problem:

      After upgrade from 4.13.x to 4.14.10, the workload images that the customer stored inside the internal registry are lost, resulting the applications pods into error 
"Back-off pulling image".

Even when manually pulling with podman, it fails then with "manifest unknown" because the image cannot be found in the registry anymore.


- This behavior was found and reproduced 100% on ARO clusters, where the internal registry is by default backed up by the Storage Account created by the ARO RP service principal, which is the Containers blob service.

- I do not know if in non-managed Azure clusters or any other architecture the same behavior is found.

      Version-Release number of selected component (if applicable):

      4.14.10

      How reproducible:

      100% with an ARO cluster (Managed cluster)

      Steps to Reproduce:  Attached.

      The workaround found so far is to rebuild the apps or re-import the images. But those tasks are lengthy and costly specially if it is a production cluster.

      Attachments

        Issue Links

          blocks

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-29525 Default Internal Registry cleans custom images stored on it from 4.13 to 4.14

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed
          is blocked by

          Spike - Represents a research-related task. IR-461 Impact assesment for OCPBUGS-29003: Default Internal Registry deletes custom images

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed
          is cloned by

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-29525 Default Internal Registry cleans custom images stored on it from 4.13 to 4.14

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed
          is related to

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-32328 Azure upgrades to 4.14.15+ fail with UPI storage account

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Verified
          links to

          Web Link KCS: Pods fails with ImagePullBackOff after upgrade to 4.14.10

          GitHub openshift/cluster-image-registry-operator#998: WIP: OCPBUGS-29003: move azure storage blobs from `docker` back into `/docker`

          GitHub openshift/docker-distribution#40: OCPBUGS-29003: fix azure storage path to use leading slash

          GitHub openshift/image-registry#392: OCPBUGS-29003: update distribution with bugfix

          Web Link RHEA-2024:0041 OpenShift Container Platform 4.16.z bug fix update

          Activity

            People

              fmissi Flavian Missi
              rhn-support-hgomes Hevellyn Gomes
              Wen Wang Wen Wang
              Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

                Created:
                Updated: