Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-29604

Default Internal Registry cleans custom images stored on it from 4.13 to 4.14

XMLWordPrintable

    • Important
    • Yes
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, the {product-title} {product-version} release introduced a change that gave users the perception that their images were lost when updating from {product-title} version 4.13 to 4.14. A change to the default internal registry caused the registry to use an incorrect path when using the Microsoft Azure object storage. With this release, the correct path is used and a job has been added to the registry operator that moves any blobs pushed to the registry that used the wrong storage path into the correct storage path, which effectively merges the two distinct storage paths into a single path. (link:https://issues.redhat.com/browse/OCPBUGS-29604[*OCPBUGS-29604*])
      +
      [NOTE]
      ====
      This fix does *not* work on Azure Stack Hub (ASH). ASH users who used OCP versions 4.14.0 through 4.14.13 when upgrading to 4.14.14+ will need to execute manual steps to move their blobs to the correct storage path. See link:https://access.redhat.com/solutions/7057297[Red Hat 7057297 Knowledgebase article].
      ====
      Show
      * Previously, the {product-title} {product-version} release introduced a change that gave users the perception that their images were lost when updating from {product-title} version 4.13 to 4.14. A change to the default internal registry caused the registry to use an incorrect path when using the Microsoft Azure object storage. With this release, the correct path is used and a job has been added to the registry operator that moves any blobs pushed to the registry that used the wrong storage path into the correct storage path, which effectively merges the two distinct storage paths into a single path. (link: https://issues.redhat.com/browse/OCPBUGS-29604 [* OCPBUGS-29604 *]) + [NOTE] ==== This fix does *not* work on Azure Stack Hub (ASH). ASH users who used OCP versions 4.14.0 through 4.14.13 when upgrading to 4.14.14+ will need to execute manual steps to move their blobs to the correct storage path. See link: https://access.redhat.com/solutions/7057297 [Red Hat 7057297 Knowledgebase article]. ====
    • Bug Fix
    • Done

      This is a clone of issue OCPBUGS-29525. The following is the description of the original issue:
      —
      This is a clone of issue OCPBUGS-29003. The following is the description of the original issue:
      —
      Description of problem:

      After upgrade from 4.13.x to 4.14.10, the workload images that the customer stored inside the internal registry are lost, resulting the applications pods into error 
"Back-off pulling image".

Even when manually pulling with podman, it fails then with "manifest unknown" because the image cannot be found in the registry anymore.


- This behavior was found and reproduced 100% on ARO clusters, where the internal registry is by default backed up by the Storage Account created by the ARO RP service principal, which is the Containers blob service.

- I do not know if in non-managed Azure clusters or any other architecture the same behavior is found.

      Version-Release number of selected component (if applicable):

      4.14.10

      How reproducible:

      100% with an ARO cluster (Managed cluster)

      Steps to Reproduce:  Attached.

      The workaround found so far is to rebuild the apps or re-import the images. But those tasks are lengthy and costly specially if it is a production cluster.

            fmissi Flavian Missi
            openshift-crt-jira-prow OpenShift Prow Bot
            xiujuan wang xiujuan wang
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: