Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-29525

Default Internal Registry cleans custom images stored on it from 4.13 to 4.14

XMLWordPrintable

    • Important
    • Yes
    • False
    • Hide

      None

      Show
      None
    • Hide
      Wordsmithed:

      Previously, the {product-title} 4.14 release introduced a change that gave users the perception that their images were lost when updating from {product-title} version 4.13 to 4.14. A change to the default internal registry caused the registry to use an incorrect path when using the Microsoft Azure object storage. With this release, the correct path is used and a job has been added to the registry operator that moves any blobs pushed to the registry that used the wrong storage path into the correct storage path, which effectively merges the two distinct storage paths into a single path.
      +
      [NOTE]
      ====
      This fix does *not* work on Azure Stack Hub (ASH). ASH users who used OCP versions 4.14.0 through 4.14.13 when upgrading to 4.14.14+ will need to execute manual steps to move their blobs to the correct storage path.
      ====

      ----------------------------------------
      OLD RN TEXT:

      4.14 introduced a change to the registry that made it use the wrong path when Azure object storage was used, giving users the perception their images was lost when updating from 4.13 to 4.14.
      The fix not only ensures that the correct path is used, but also adds a job to the registry operator that moves any blobs pushed to the registry while it used the wrong storage path into the correct storage path, effectively merging the two distinct storage paths into a single one. Note that the path fix job *does not work on Azure Stack hub*! ASH users that used OCP versions 4.14.0 through 4.14.13, when upgrading to 4.14.14+ will need to execute manual steps to move their blobs to the correct storage path (from docker into /docker).
      Show
      Wordsmithed: Previously, the {product-title} 4.14 release introduced a change that gave users the perception that their images were lost when updating from {product-title} version 4.13 to 4.14. A change to the default internal registry caused the registry to use an incorrect path when using the Microsoft Azure object storage. With this release, the correct path is used and a job has been added to the registry operator that moves any blobs pushed to the registry that used the wrong storage path into the correct storage path, which effectively merges the two distinct storage paths into a single path. + [NOTE] ==== This fix does *not* work on Azure Stack Hub (ASH). ASH users who used OCP versions 4.14.0 through 4.14.13 when upgrading to 4.14.14+ will need to execute manual steps to move their blobs to the correct storage path. ==== ---------------------------------------- OLD RN TEXT: 4.14 introduced a change to the registry that made it use the wrong path when Azure object storage was used, giving users the perception their images was lost when updating from 4.13 to 4.14. The fix not only ensures that the correct path is used, but also adds a job to the registry operator that moves any blobs pushed to the registry while it used the wrong storage path into the correct storage path, effectively merging the two distinct storage paths into a single one. Note that the path fix job *does not work on Azure Stack hub*! ASH users that used OCP versions 4.14.0 through 4.14.13, when upgrading to 4.14.14+ will need to execute manual steps to move their blobs to the correct storage path (from docker into /docker).
    • Bug Fix
    • In Progress

      This is a clone of issue OCPBUGS-29003. The following is the description of the original issue:
      —
      Description of problem:

      After upgrade from 4.13.x to 4.14.10, the workload images that the customer stored inside the internal registry are lost, resulting the applications pods into error 
"Back-off pulling image".

Even when manually pulling with podman, it fails then with "manifest unknown" because the image cannot be found in the registry anymore.


- This behavior was found and reproduced 100% on ARO clusters, where the internal registry is by default backed up by the Storage Account created by the ARO RP service principal, which is the Containers blob service.

- I do not know if in non-managed Azure clusters or any other architecture the same behavior is found.

      Version-Release number of selected component (if applicable):

      4.14.10

      How reproducible:

      100% with an ARO cluster (Managed cluster)

      Steps to Reproduce:  Attached.

      The workaround found so far is to rebuild the apps or re-import the images. But those tasks are lengthy and costly specially if it is a production cluster.

            fmissi Flavian Missi
            openshift-crt-jira-prow OpenShift Prow Bot
            Wen Wang Wen Wang
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: