-
Bug
-
Resolution: Unresolved
-
Normal
-
None
-
4.17.z, 4.16.z, 4.18
-
None
-
Critical
-
Yes
-
Rejected
-
False
-
Description of problem:
Can't reuse blob to push aws c2s bucket with x509 error since Dec. And on prow ci result, it failed in pull image from internal registry
Version-Release number of selected component (if applicable):
4.18
How reproducible:
always
Steps to Reproduce:
1.Set up an aws c2s cluster with 4.18 using prow
https://gcsweb-qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.18-amd64-nightly-aws-sc2s-ipi-disc-priv-fips-f2/1865304153111138304/artifacts/aws-sc2s-ipi-disc-priv-fips-f2/openshift-extended-test/build-log.txt
build logs : 2024-12-07T11:25:33.257156973Z Replaced Dockerfile FROM image quay.io/openshifttest/busybox@sha256:c5439d7db88ab5423999530349d327b04279ad3161d7596d2126dfb5b02bfd1f
2024-12-07T11:25:35.613930151Z time="2024-12-07T11:25:35Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"
2024-12-07T11:25:35.614776370Z I1207 11:25:35.614748 1 defaults.go:112] Defaulting to storage driver "overlay" with options [mountopt=metacopy=on].
2024-12-07T11:25:35.629453348Z Caching blobs under "/var/cache/blobs".
2024-12-07T11:25:35.632143910Z
2024-12-07T11:25:35.632143910Z Pulling image image-registry.openshift-image-registry.svc:5000/e2e-test-default-image-registry-zgmf2/busybox@sha256:c5439d7db88ab5423999530349d327b04279ad3161d7596d2126dfb5b02bfd1f ...
2024-12-07T11:25:35.636791870Z Trying to pull image-registry.openshift-image-registry.svc:5000/e2e-test-default-image-registry-zgmf2/busybox@sha256:c5439d7db88ab5423999530349d327b04279ad3161d7596d2126dfb5b02bfd1f...
2024-12-07T11:25:36.695243084Z Warning: Pull failed, retrying in 5s ...
2024-12-07T11:25:41.698639710Z Trying to pull image-registry.openshift-image-registry.svc:5000/e2e-test-default-image-registry-zgmf2/busybox@sha256:c5439d7db88ab5423999530349d327b04279ad3161d7596d2126dfb5b02bfd1f...
2024-12-07T11:25:41.848354202Z Warning: Pull failed, retrying in 5s ...
2024-12-07T11:25:46.852090834Z Trying to pull image-registry.openshift-image-registry.svc:5000/e2e-test-default-image-registry-zgmf2/busybox@sha256:c5439d7db88ab5423999530349d327b04279ad3161d7596d2126dfb5b02bfd1f...
2024-12-07T11:25:47.406620628Z Warning: Pull failed, retrying in 5s ...
2024-12-07T11:25:52.439067513Z error: build error: failed to pull image: After retrying 2 times, Pull image still failed due to error: copying system image from manifest list: parsing image configuration: Get "https://ci-op-5gnlt4mn-c6dd8-6xdhm-image-registry-us-isob-east-1-sxuku.s3.us-isob-east-1.sc2s.sgov.gov/docker/registry/v2/blobs/sha256/b9/b97242f89c8a29d13aea12843a08441a4bbfc33528f55b60366c1d8f6923d0d4/data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAUW2DH7NUFSKUEAEQ%2F20241207%2Fus-isob-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241207T112547Z&X-Amz-Expires=1200&X-Amz-SignedHeaders=host&X-Amz-Signature=6aba8b40b4b4f43e49074fcdae2db73c10c648c93ce9fb4d15bcd114925ef713": tls: failed to verify certificate: x509: certificate signed by unknown authority
2. But when reproduce it on prow ci cluster. we could push image at first try as always, but failed to push with reuse blob.
oc get builds
NAME TYPE FROM STATUS STARTED DURATION
test-registry-1 Docker Dockerfile Complete 2 hours ago 15s
test-registry-2 Docker Dockerfile Failed (PushImageToRegistryFailed) 2 hours ago 21s
oc logs -f test-registry-2-build
error: build error: Failed to push image: trying to reuse blob sha256:530afca65e2ea04227630ae746e0c85b2bd1a179379cbf2b6501b49c4cab2ccc at destination: Head "https://ci-op-qf55lf0q-3c6ac-j65q8-image-registry-us-iso-east-1-soajkc.s3.us-iso-east-1.c2s.ic.gov/docker/registry/v2/blobs/sha256/53/530afca65e2ea04227630ae746e0c85b2bd1a179379cbf2b6501b49c4cab2ccc/data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAUW2DH7NUKYUBPOG2%2F20241210%2Fus-iso-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241210T070804Z&X-Amz-Expires=1200&X-Amz-SignedHeaders=host&X-Amz-Signature=f5a44e20c1f5548e8cbec55f0faa0677fea306ecc8f1008a298a6171b8969a79": tls: failed to verify certificate: x509: certificate signed by unknown authority
3. Can't push a existing image to s3 bucket using podman
podman push image-registry.openshift-image-registry.svc:5000/wxj/test:latest
Getting image source signatures
Copying blob 1a8c6bfa0a12 done |
Copying blob 42e40303859e done |
Copying blob 9c6be2aa7dfd done |
Copying blob b80a8be6331b done |
Copying config cda9cd0379 done |
Writing manifest to image destination
sh-5.1# podman push image-registry.openshift-image-registry.svc:5000/wxj/test:latest
Getting image source signatures
Copying blob 9c6be2aa7dfd done |
Copying blob 42e40303859e done |
Copying blob 1a8c6bfa0a12 done |
Copying blob b80a8be6331b done |
Copying config cda9cd0379 [--------------------------------------] 8.0b / 20.7KiB | 7.8 MiB/s
Error: writing blob: checking whether a blob sha256:cda9cd0379eae8c992bf505926f7a9ea95f93b876cb55b15c9345b0d41a45388 exists in image-registry.openshift-image-registry.svc:5000/wxj/test: StatusCode: 403, ""
Actual results:
Failed to pull or push reuse blob from/to image registry.
Expected results:
Shouldn't fail to pull/push image from internal registry.
Additional info:
must gather log https://gcsweb-qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.18-amd64-nightly-aws-sc2s-ipi-disc-priv-fips-f2/1865304153111138304/artifacts/aws-sc2s-ipi-disc-priv-fips-f2/gather-must-gather/
