Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-3393

oc image mirror to file system is not working as expected when HTTP 429 error is received from registry with empty response

XMLWordPrintable

    • Important
    • Rejected
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      While mirror to filesystem, if 429 error is received from registry, the layer is incorrectly flagged as having been mirrored & therefore not picked up by subsequent mirror re-run requests. It gives the impression as mirror to file system in second attempt is successful. However, causing issue while mirroring from filesystem to target registry (Due to missing files)

      Version-Release number of selected component (if applicable):

      oc version
      Client Version: 4.8.42
      Server Version: 4.8.14
      Kubernetes Version: v1.21.1+a620f50

      How reproducible:

      When 429 occurs while mirror to file system

      Steps to Reproduce:

      1. Run mirror to filesystem command : oc image mirror -f mirror-to-filesystem.txt --filter-by-os '.*' -a $REGISTRY_AUTH_FILE --insecure --skip-multiple-scopes --max-per-registry=1 --continue-on-error=true --dir "$LOCAL_DIR_PATH"  
      
      Output: 
      info: Mirroring completed in 2h19m24.14s (25.75MB/s)
      error: one or more errors occurred 
      E.g
      error: unable to push <registry>/namespace/<image-name>: failed to retrieve blob <image-digest>: error parsing HTTP 429 response body: unexpected end of JSON input: ""
      
      
      2. Re Run mirror to filesystem command : oc image mirror -f mirror-to-filesystem.txt --filter-by-os '.*' -a $REGISTRY_AUTH_FILE --insecure --skip-multiple-scopes --max-per-registry=1 --continue-on-error=true --dir "$LOCAL_DIR_PATH"
      
      Output:
      info: Mirroring completed in 480ms (0B/s)
      
      
      3. Run mirror from filesystem command : oc image mirror -f mirror-from-filesystem.txt -a $REGISTRY_AUTH_FILE --from-dir "$LOCAL_DIR_PATH" --filter-by-os '.*' --insecure --skip-multiple-scopes --max-per-registry=1 --continue-on-error=true
      
      Output: 
      info: Mirroring completed in 53m5.21s (67.61MB/s)
      error: one or more errors occurred
      E.g
      error: unable to push file://local/namespace/<image-name>: failed to retrieve blob <image-digest>: open /root/local/namespace/<image-name>/blobs/<image-digest>: no such file or directory

       

      Actual results:

      1) mirror to filesystem first attempt: 
      
      info: Mirroring completed in 2h19m24.14s (25.75MB/s) 
      error: one or more errors occurred 
      E.g
      error: unable to push <registry>/namespace/<image-name>: failed to retrieve blob <image-digest>: error parsing HTTP 429 response body: unexpected end of JSON input: ""
      
      2) mirror to filesystem second attempt: 
      
      info: Mirroring completed in 480ms (0B/s)
      
       
      3) mirror from filesystem to target registry:  
      
      info: Mirroring completed in 53m5.21s (67.61MB/s) 
      error: one or more errors occurred 
      E.g 
      error: unable to push file://local/namespace/<image-name>: failed to retrieve blob <image-digest>: open /root/local/namespace/<image-name>/blobs/<image-digest>: no such file or directory

      Expected results:

      source image mirror -> to file system and image mirror from file system -> target registry should complete successfully

      Additional info:

       

            aguclu@redhat.com Arda Guclu
            manoj.paladiya Manoj Paladiya
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: