Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-58186

oc-mirror v2 doesn't retry when timeout at mirroring a image

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Moderate
    • None
    • None
    • CLID Sprint 274
    • 1
    • In Progress
    • Release Note Not Required
    • N/A
    • None
    • None
    • None
    • None

      Description of problem:

      According to the docs [1], the default number of times to retry for oc-mirror v2 is "2".
      However, it seems that oc-mirror v2 doesn't retry when timeout at mirroring a image.
      It doesn't retry even when we specified --retry-times flag.

      It's common that retry after timeout. So we think this is a bug.

      [1] https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/disconnected_environments/mirroring-in-disconnected-environments#oc-mirror-command-reference-v2_about-installing-oc-mirror-v2

      Version-Release number of selected component (if applicable):

      • oc-mirror v4.18.16

      How reproducible:

      Always

      Steps to Reproduce:

      Step1. To intentionally cause timeout, we exclude the access to quay.io from proxy.

      $ export http_proxy=http://<proxy url>
      $ export https_proxy=http://<proxy url>
      $ export no_proxy=quay.io
      

      Step2. Create imageset-config.yaml file as following:

      kind: ImageSetConfiguration
      apiVersion: mirror.openshift.io/v2alpha1
      mirror:
        platform:
          architectures:
            - amd64
          channels:
          - name: stable-4.18
            type: ocp
            minVersion: 4.18.9
            maxVersion: 4.18.9
      

      Step3. Run oc-mirror v2 command.

      $ oc-mirror -c ./imageset-config.yaml file:///home/user/data/metadata --v2 --log-level trace
      

      If you add --retry-times flag:

      $ oc-mirror -c ./imageset-config.yaml file:///home/user/data/metadata --v2 --retry-times 4 --log-level trace
      

      Actual results:

      oc-mirror v2 doesn't retry even once when timeout at mirroring a image.

      ....
      2025/06/11 17:32:49  [DEBUG]  : image found : quay.io/openshift-release-dev/ocp-release:4.18.9-x86_64
      2025/06/11 17:32:49  [DEBUG]  : [ReleaseImageCollector] copying  release image quay.io/openshift-release-dev/ocp-release:4.18.9-x86_64 
      2025/06/11 17:35:51  [INFO]   : mirror time     : 3m4.526618307s
      2025/06/11 17:35:51  [INFO]   : 👋 Goodbye, thank you for using oc-mirror
      2025/06/11 17:35:51  [ERROR]  : collection error: [ReleaseImageCollector] copy release index image: initializing source docker://quay.io/openshift-release-dev/ocp-release:4.18.9-x86_64: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp 52.54.188.161:443: i/o timeout 
      

      Expected results:

      oc-mirror v2 retries the number of times specified by user when timeout at mirroring a image.

      Additional information:

      oc-mirror v2 calls the retry function, and the handling of errors is defined by IsErrorRetryable [2][3].
      In the default IsErrorRetryable, it handle context.DeadlineExceeded error as not-retryable.
      Caller should make its own IsErrorRetryable instead if necessary.
      The oc-mirror v2 doesn't make its own IsErrorRetryable to handle context.DeadlineExceeded error [4].
      We think this is the cause of the issue.

      [2] https://github.com/containers/common/blob/main/pkg/retry/retry.go#L69
      [3] https://pkg.go.dev/github.com/containers/common/pkg/retry#IsErrorRetryable
      [4] https://github.com/openshift/oc-mirror/blob/be3d7693429ba713a42a169ea3b9eee44bcd4834/v2/internal/pkg/mirror/options.go#L273

              rdossant Rafael Fonseca dos Santos
              rhn-support-kahara Kazuhisa Hara
              None
              None
              Nidan Gavali Nidan Gavali
              None
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: