imagestream import failed with remote image with error "Bad Gateway" in aws-rosa-hcp-private-proxy cluster

    4.16.0-0.nightly-2024-10-08-205135

   always(since Mid-September)

    1.Import an image
    2.
    3.
    

   # oc import-image test --from=quay.io/openshifttest/hello-openshift@sha256:56c354e7885051b6bb4263f9faa58b2c292d44790599b7dde0e49e7c466cf339 --loglevel=10 --confirm
I1009 15:23:54.826542   98204 loader.go:395] Config loaded from file:  /Users/xiuwang/kubec/hcp
I1009 15:23:54.827152   98204 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: oc/4.18.0 (darwin/amd64) kubernetes/e3584b9" -H "Authorization: Bearer <masked>" 'https://api.ci-rosa-h-cjge.flfc.i3.devshift.org:443/apis/image.openshift.io/v1/namespaces/xiuwang/imagestreams/test'
I1009 15:23:54.830338   98204 round_trippers.go:495] HTTP Trace: DNS Lookup for ec2-34-213-241-195.us-west-2.compute.amazonaws.com resolved to [{34.213.241.195 }]
I1009 15:23:55.024352   98204 round_trippers.go:510] HTTP Trace: Dial to tcp:34.213.241.195:3128 succeed
I1009 15:23:55.661364   98204 round_trippers.go:553] GET https://api.ci-rosa-h-cjge.flfc.i3.devshift.org:443/apis/image.openshift.io/v1/namespaces/xiuwang/imagestreams/test 200 OK in 834 milliseconds
I1009 15:23:55.661456   98204 round_trippers.go:570] HTTP Statistics: DNSLookup 2 ms Dial 193 ms TLSHandshake 194 ms ServerProcessing 215 ms Duration 834 ms
I1009 15:23:55.661498   98204 round_trippers.go:577] Response Headers:
I1009 15:23:55.661524   98204 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: e67d28a3-8c0b-4b7d-a333-a960ae0fea62
I1009 15:23:55.661542   98204 round_trippers.go:580]     Cache-Control: no-cache, private
I1009 15:23:55.661565   98204 round_trippers.go:580]     Cache-Control: no-store
I1009 15:23:55.661582   98204 round_trippers.go:580]     Content-Type: application/json
I1009 15:23:55.661595   98204 round_trippers.go:580]     Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
I1009 15:23:55.661612   98204 round_trippers.go:580]     Content-Length: 1113
I1009 15:23:55.661627   98204 round_trippers.go:580]     Audit-Id: 9606f29d-dccd-4699-9366-36fc081fe620
I1009 15:23:55.661643   98204 round_trippers.go:580]     Audit-Id: 9606f29d-dccd-4699-9366-36fc081fe620
I1009 15:23:55.661656   98204 round_trippers.go:580]     Date: Wed, 09 Oct 2024 07:23:55 GMT
I1009 15:23:55.661668   98204 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: d5b62b2b-3e70-4f54-8bfb-671c9c0f5170
I1009 15:23:55.661807   98204 request.go:1212] Response Body: {"kind":"ImageStream","apiVersion":"image.openshift.io/v1","metadata":{"name":"test","namespace":"xiuwang","uid":"59bcbaa6-92c2-412d-9415-e91e4e7bd8cf","resourceVersion":"52745","generation":1,"creationTimestamp":"2024-10-09T06:49:51Z","annotations":{"openshift.io/image.dockerRepositoryCheck":"2024-10-09T06:49:51Z"}},"spec":{"lookupPolicy":{"local":false},"tags":[{"name":"latest","annotations":null,"from":{"kind":"DockerImage","name":"quay.io/openshifttest/hello-openshift@sha256:56c354e7885051b6bb4263f9faa58b2c292d44790599b7dde0e49e7c466cf339"},"generation":1,"importPolicy":{"importMode":"Legacy"},"referencePolicy":{"type":"Source"}}]},"status":{"dockerImageRepository":"image-registry.openshift-image-registry.svc:5000/xiuwang/test","tags":[{"tag":"latest","items":null,"conditions":[{"type":"ImportSuccess","status":"False","lastTransitionTime":"2024-10-09T06:49:51Z","reason":"InternalError","message":"Internal error occurred: quay.io/openshifttest/hello-openshift@sha256:56c354e7885051b6bb4263f9faa58b2c292d44790599b7dde0e49e7c466cf339: Get \"https://quay.io/v2/\": Bad Gateway","generation":1}]}]}}
I1009 15:23:55.664747   98204 request.go:1212] Request Body: {"kind":"ImageStreamImport","apiVersion":"image.openshift.io/v1","metadata":{"name":"test","namespace":"xiuwang","resourceVersion":"52745","creationTimestamp":null},"spec":{"import":true,"images":[{"from":{"kind":"DockerImage","name":"quay.io/openshifttest/hello-openshift@sha256:56c354e7885051b6bb4263f9faa58b2c292d44790599b7dde0e49e7c466cf339"},"to":{"name":"latest"},"importPolicy":{"importMode":"Legacy"},"referencePolicy":{"type":"Source"}}]},"status":{}}
I1009 15:23:55.664896   98204 round_trippers.go:466] curl -v -XPOST  -H "User-Agent: oc/4.18.0 (darwin/amd64) kubernetes/e3584b9" -H "Authorization: Bearer <masked>" -H "Accept: application/json, */*" -H "Content-Type: application/json" 'https://api.ci-rosa-h-cjge.flfc.i3.devshift.org:443/apis/image.openshift.io/v1/namespaces/xiuwang/imagestreamimports'
I1009 15:23:55.912043   98204 round_trippers.go:553] POST https://api.ci-rosa-h-cjge.flfc.i3.devshift.org:443/apis/image.openshift.io/v1/namespaces/xiuwang/imagestreamimports 201 Created in 247 milliseconds
I1009 15:23:55.912129   98204 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 246 ms Duration 247 ms
I1009 15:23:55.912156   98204 round_trippers.go:577] Response Headers:
I1009 15:23:55.912184   98204 round_trippers.go:580]     Content-Length: 2257
I1009 15:23:55.912204   98204 round_trippers.go:580]     Audit-Id: 521f29a1-6fff-45bc-a7a4-3be26950b9a4
I1009 15:23:55.912225   98204 round_trippers.go:580]     Audit-Id: 521f29a1-6fff-45bc-a7a4-3be26950b9a4
I1009 15:23:55.912241   98204 round_trippers.go:580]     Cache-Control: no-cache, private
I1009 15:23:55.912262   98204 round_trippers.go:580]     Cache-Control: no-store
I1009 15:23:55.912278   98204 round_trippers.go:580]     Content-Type: application/json
I1009 15:23:55.912297   98204 round_trippers.go:580]     Date: Wed, 09 Oct 2024 07:23:55 GMT
I1009 15:23:55.912316   98204 round_trippers.go:580]     Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
I1009 15:23:55.912438   98204 request.go:1212] Response Body: {"kind":"ImageStreamImport","apiVersion":"image.openshift.io/v1","metadata":{"name":"test","namespace":"xiuwang","uid":"cf7e6bbc-3443-49f6-a7ff-4b09a2cf0431","resourceVersion":"52745","creationTimestamp":"2024-10-09T07:23:55Z","managedFields":[{"manager":"oc","operation":"Update","apiVersion":"image.openshift.io/v1","time":"2024-10-09T07:23:55Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:images":{},"f:import":{}}}}]},"spec":{"import":true,"images":[{"from":{"kind":"DockerImage","name":"quay.io/openshifttest/hello-openshift@sha256:56c354e7885051b6bb4263f9faa58b2c292d44790599b7dde0e49e7c466cf339"},"to":{"name":"latest"},"importPolicy":{"importMode":"Legacy"},"referencePolicy":{"type":"Source"}}]},"status":{"import":{"metadata":{"name":"test","namespace":"xiuwang","uid":"59bcbaa6-92c2-412d-9415-e91e4e7bd8cf","resourceVersion":"52745","generation":1,"creationTimestamp":"2024-10-09T06:49:51Z","annotations":{"openshift.io/image.dockerRepositoryCheck":"2024-10-09T06:49:51Z"}},"spec":{"lookupPolicy":{"local":false},"tags":[{"name":"latest","annotations":null,"from":{"kind":"DockerImage","name":"quay.io/openshifttest/hello-openshift@sha256:56c354e7885051b6bb4263f9faa58b2c292d44790599b7dde0e49e7c466cf339"},"generation":1,"importPolicy":{"importMode":"Legacy"},"referencePolicy":{"type":"Source"}}]},"status":{"dockerImageRepository":"image-registry.openshift-image-registry.svc:5000/xiuwang/test","tags":[{"tag":"latest","items":null,"conditions":[{"type":"ImportSuccess","status":"False","lastTransitionTime":"2024-10-09T06:49:51Z","reason":"InternalError","message":"Internal error occurred: quay.io/openshifttest/hello-openshift@sha256:56c354e7885051b6bb4263f9faa58b2c292d44790599b7dde0e49e7c466cf339: Get \"https://quay.io/v2/\": Bad Gateway","generation":1}]}]}},"images":[{"status":{"metadata":{},"status":"Failure","message":"Internal error occurred: quay.io/openshifttest/hello-openshift@sha256:56c354e7885051b6bb4263f9faa58b2c292d44790599b7dde0e49e7c466cf339: Get \"https://quay.io/v2/\": Bad Gateway","reason":"InternalError","details":{"causes":[{"message":"quay.io/openshifttest/hello-openshift@sha256:56c354e7885051b6bb4263f9faa58b2c292d44790599b7dde0e49e7c466cf339: Get \"https://quay.io/v2/\": Bad Gateway"}]},"code":500}}]}}
error: tag  failed: Internal error occurred: quay.io/openshifttest/hello-openshift@sha256:56c354e7885051b6bb4263f9faa58b2c292d44790599b7dde0e49e7c466cf339: Get "https://quay.io/v2/": Bad Gateway
imagestream.image.openshift.io/test imported with errorsName:            test
Namespace:        xiuwang
Created:        34 minutes ago
Labels:            <none>
Annotations:        openshift.io/image.dockerRepositoryCheck=2024-10-09T06:49:51Z
Image Repository:    image-registry.openshift-image-registry.svc:5000/xiuwang/test
Image Lookup:        local=false
Unique Images:        0
Tags:            1latest
  tagged from quay.io/openshifttest/hello-openshift@sha256:56c354e7885051b6bb4263f9faa58b2c292d44790599b7dde0e49e7c466cf339  ! error: Import failed (InternalError): Internal error occurred: quay.io/openshifttest/hello-openshift@sha256:56c354e7885051b6bb4263f9faa58b2c292d44790599b7dde0e49e7c466cf339: Get "https://quay.io/v2/": Bad Gateway
      34 minutes ago
error: imported completed with errors

    Could import image successfully

    https://gcsweb-qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.16-amd64-nightly-aws-rosa-hcp-private-proxy-int-full-f14/1843676847938736128/artifacts/aws-rosa-hcp-private-proxy-int-full-f14/gather-must-gather/artifacts/must-gather.tar