-
Bug
-
Resolution: Done-Errata
-
Critical
-
4.14, 4.15, 4.16, 4.17
This is a clone of issue OCPBUGS-31446. The following is the description of the original issue:
—
Description of problem:
imagesStreams on hosted-clusters pointing to image on private registries are failing due to tls verification although the registry is correctly trusted. example: $ oc create namespace e2e-test $ oc --namespace=e2e-test tag virthost.ostest.test.metalkube.org:5000/localimages/local-test-image:e2e-7-registry-k8s-io-e2e-test-images-busybox-1-29-4-4zE9mRvED4RQoUxQ busybox:latest $ oc --namespace=e2e-test set image-lookup busybox stirabos@t14s:~$ oc get imagestream -n e2e-test NAME IMAGE REPOSITORY TAGS UPDATED busybox image-registry.openshift-image-registry.svc:5000/e2e-test/busybox latest stirabos@t14s:~$ oc get imagestream -n e2e-test busybox -o yaml apiVersion: image.openshift.io/v1 kind: ImageStream metadata: annotations: openshift.io/image.dockerRepositoryCheck: "2024-03-27T12:43:56Z" creationTimestamp: "2024-03-27T12:43:56Z" generation: 3 name: busybox namespace: e2e-test resourceVersion: "49021" uid: 847281e7-e307-4057-ab57-ccb7bfc49327 spec: lookupPolicy: local: true tags: - annotations: null from: kind: DockerImage name: virthost.ostest.test.metalkube.org:5000/localimages/local-test-image:e2e-7-registry-k8s-io-e2e-test-images-busybox-1-29-4-4zE9mRvED4RQoUxQ generation: 2 importPolicy: importMode: Legacy name: latest referencePolicy: type: Source status: dockerImageRepository: image-registry.openshift-image-registry.svc:5000/e2e-test/busybox tags: - conditions: - generation: 2 lastTransitionTime: "2024-03-27T12:43:56Z" message: 'Internal error occurred: virthost.ostest.test.metalkube.org:5000/localimages/local-test-image:e2e-7-registry-k8s-io-e2e-test-images-busybox-1-29-4-4zE9mRvED4RQoUxQ: Get "https://virthost.ostest.test.metalkube.org:5000/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority' reason: InternalError status: "False" type: ImportSuccess items: null tag: latest While image virthost.ostest.test.metalkube.org:5000/localimages/local-test-image:e2e-7-registry-k8s-io-e2e-test-images-busybox-1-29-4-4zE9mRvED4RQoUxQ can be properly consumed if directly used for a container on a pod on the same cluster. user-ca-bundle config map is properly propagated from hypershift: $ oc get configmap -n openshift-config user-ca-bundle NAME DATA AGE user-ca-bundle 1 3h32m $ openssl x509 -text -noout -in <(oc get cm -n openshift-config user-ca-bundle -o json | jq -r '.data["ca-bundle.crt"]') Certificate: Data: Version: 3 (0x2) Serial Number: 11:3f:15:23:97:ac:c2:d5:f6:54:06:1a:9a:22:f2:b5:bf:0c:5a:00 Signature Algorithm: sha256WithRSAEncryption Issuer: C = US, ST = NC, L = Raleigh, O = Test Company, OU = Testing, CN = test.metalkube.org Validity Not Before: Mar 27 08:28:07 2024 GMT Not After : Mar 27 08:28:07 2025 GMT Subject: C = US, ST = NC, L = Raleigh, O = Test Company, OU = Testing, CN = test.metalkube.org Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:c1:49:1f:18:d2:12:49:da:76:05:36:3e:6b:1a: 82:a7:22:0d:be:f5:66:dc:97:44:c7:ca:31:4d:f3: 7f:0a:d3:de:df:f2:b6:23:f9:09:b1:7a:3f:19:cc: 22:c9:70:90:30:a7:eb:49:28:b6:d1:e0:5a:14:42: 02:93:c4:ac:cc:da:b1:5a:8f:9c:af:60:19:1a:e3: b1:34:c2:b6:2f:78:ec:9f:fe:38:75:91:0f:a6:09: 78:28:36:9e:ab:1c:0d:22:74:d5:52:fe:0a:fc:db: 5a:7c:30:9d:84:7d:f7:6a:46:fe:c5:6f:50:86:98: cc:35:1f:6c:b0:e6:21:fc:a5:87:da:81:2c:7b:e4: 4e:20:bb:35:cc:6c:81:db:b3:95:51:cf:ff:9f:ed: 00:78:28:1d:cd:41:1d:03:45:26:45:d4:36:98:bd: bf:5c:78:0f:c7:23:5c:44:5d:a6:ae:85:2b:99:25: ae:c0:73:b1:d2:87:64:3e:15:31:8e:63:dc:be:5c: ed:e3:fe:97:29:10:fb:5c:43:2f:3a:c2:e4:1a:af: 80:18:55:bc:40:0f:12:26:6b:f9:41:da:e2:a4:6b: fd:66:ae:bc:9c:e8:2a:5a:3b:e7:2b:fc:a6:f6:e2: 73:9b:79:ee:0c:86:97:ab:2e:cc:47:e7:1b:e5:be: 0c:9f Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Basic Constraints: CA:TRUE, pathlen:0 X509v3 Subject Alternative Name: DNS:virthost.ostest.test.metalkube.org Signature Algorithm: sha256WithRSAEncryption Signature Value: 58:d2:da:f9:2a:c0:2d:7a:d9:9f:1f:97:e1:fd:36:a7:32:d3: ab:3f:15:cd:68:8e:be:7c:11:ec:5e:45:50:c4:ec:d8:d3:c5: 22:3c:79:5a:01:63:9e:5a:bd:02:0c:87:69:c6:ff:a2:38:05: 21:e4:96:78:40:db:52:c8:08:44:9a:96:6a:70:1e:1e:ae:74: e2:2d:fa:76:86:4d:06:b1:cf:d5:5c:94:40:17:5d:9f:84:2c: 8b:65:ca:48:2b:2d:00:3b:42:b9:3c:08:1b:c5:5d:d2:9c:e9: bc:df:9a:7c:db:30:07:be:33:2a:bb:2d:69:72:b8:dc:f4:0e: 62:08:49:93:d5:0f:db:35:98:18:df:e6:87:11:ce:65:5b:dc: 6f:f7:f0:1c:b0:23:40:1e:e3:45:17:04:1a:bc:d1:57:d7:0d: c8:26:6d:99:fe:28:52:fe:ba:6a:a1:b8:d1:d1:50:a9:fa:03: bb:b7:ad:0e:82:d2:e8:34:91:fa:b4:f9:81:d1:9b:6d:0f:a3: 8c:9d:c4:4a:1e:08:26:71:b9:1a:e8:49:96:0f:db:5c:76:db: ae:c7:6b:2e:ea:89:5d:7f:a3:ba:ea:7e:12:97:12:bc:1e:7f: 49:09:d4:08:a6:4a:34:73:51:9e:a2:9a:ec:2a:f7:fc:b5:5c: f8:20:95:ad This is probably a side effect of https://issues.redhat.com/browse/RFE-3093 - imagestream to trust CA added during the installation, that is also affecting imagestreams that requires a CA cert injected by hypershift during hosted-cluster creation in the disconnected use case.
Version-Release number of selected component (if applicable):
v4.14, v4.15, v4.16
How reproducible:
100%
Steps to Reproduce:
once connected to a disconnected hosted cluster, create an image stream pointing to an image on the internal mirror registry:
1. $ oc --namespace=e2e-test tag virthost.ostest.test.metalkube.org:5000/localimages/local-test-image:e2e-7-registry-k8s-io-e2e-test-images-busybox-1-29-4-4zE9mRvED4RQoUxQ busybox:latest
2. $ oc --namespace=e2e-test set image-lookup busybox
3. then check the image stream
Actual results:
status: dockerImageRepository: image-registry.openshift-image-registry.svc:5000/e2e-test/busybox tags: - conditions: - generation: 2 lastTransitionTime: "2024-03-27T12:43:56Z" message: 'Internal error occurred: virthost.ostest.test.metalkube.org:5000/localimages/local-test-image:e2e-7-registry-k8s-io-e2e-test-images-busybox-1-29-4-4zE9mRvED4RQoUxQ: Get "https://virthost.ostest.test.metalkube.org:5000/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority' although the same image can be directly consumed by a pod on the same cluster
Expected results:
status: dockerImageRepository: image-registry.openshift-image-registry.svc:5000/e2e-test/busybox tags: - conditions: - generation: 8 lastTransitionTime: "2024-03-27T13:30:46Z" message: dockerimage.image.openshift.io "virthost.ostest.test.metalkube.org:5000/localimages/local-test-image:e2e-7-registry-k8s-io-e2e-test-images-busybox-1-29-4-4zE9mRvED4RQoUxQ" not found reason: NotFound status: "False" type: ImportSuccess
Additional info:
This is probably a side effect of https://issues.redhat.com/browse/RFE-3093 Marking the imagestream as: importPolicy: importMode: Legacy insecure: true is enough to workaround this.
- blocks
-
-
- Closed
-
- clones
-
-
- Closed
-
- is blocked by
-
-
- Closed
-
- is cloned by
-
-
- Closed
-
- links to
-
RHEA-2024:0041
OpenShift Container Platform 4.16.z bug fix update
(1 links to)
[OCPBUGS-34390] HCP: imagesStreams on hosted-clusters pointing to image on private registries are failing due to tls verification although the registry is correctly trusted
➜ export KUBECONFIG=kubeconfig ➜ oc get clusterversion version oc NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.16.0-0.ci-2024-06-10-215047 True False 93m Error while reconciling 4.16.0-0.ci-2024-06-10-215047: the cluster operator image-registry is degraded ➜ oc get multiclusterengines multiclusterengine-sample -ojsonpath="{.status.currentVersion}" 2.6.0-121% ➜ oc get hostedcluster -A NAMESPACE NAME VERSION KUBECONFIG PROGRESS AVAILABLE PROGRESSING MESSAGE local-cluster 151215bb3ba087f834a2 4.16.0-0.ci-2024-06-10-215047 151215bb3ba087f834a2-admin-kubeconfig Completed True False The hosted control plane is available ➜ export KUBECONFIG=hostedcluster.kubeconfig ➜ oc get node NAME STATUS ROLES AGE VERSION ostest-extraworker-0 Ready worker 54m v1.29.5+b2f5b11 ostest-extraworker-1 Ready worker 54m v1.29.5+b2f5b11 ostest-extraworker-2 Ready worker 54m v1.29.5+b2f5b11 ➜ oc create namespace liangli-test namespace/liangli-test created ➜ oc --namespace=liangli-test tag virthost.ostest.test.metalkube.org:5000/localimages/local-test-image:e2e-7-registry-k8s-io-e2e-test-images-busybox-1-29-2-ZYWRth-o9U_JR2ZE busybox:latest Tag busybox:latest set to virthost.ostest.test.metalkube.org:5000/localimages/local-test-image:e2e-7-registry-k8s-io-e2e-test-images-busybox-1-29-2-ZYWRth-o9U_JR2ZE. ➜ oc set image-lookup busybox --namespace=liangli-test imagestream.image.openshift.io/busybox image lookup updated ➜ oc get imagestream -n liangli-test NAME IMAGE REPOSITORY TAGS UPDATED busybox image-registry.openshift-image-registry.svc:5000/liangli-test/busybox latest 24 seconds ago ➜ oc get imagestream -n liangli-test -o yaml apiVersion: v1 items: - apiVersion: image.openshift.io/v1 kind: ImageStream metadata: annotations: openshift.io/image.dockerRepositoryCheck: "2024-06-11T14:39:01Z" creationTimestamp: "2024-06-11T14:39:01Z" generation: 3 name: busybox namespace: liangli-test resourceVersion: "209756" uid: 9990c6bb-0b10-481c-a9d7-45ec653798b1 spec: lookupPolicy: local: true tags: - annotations: null from: kind: DockerImage name: virthost.ostest.test.metalkube.org:5000/localimages/local-test-image:e2e-7-registry-k8s-io-e2e-test-images-busybox-1-29-2-ZYWRth-o9U_JR2ZE generation: 2 importPolicy: importMode: Legacy name: latest referencePolicy: type: Source status: dockerImageRepository: image-registry.openshift-image-registry.svc:5000/liangli-test/busybox tags: - items: - created: "2024-06-11T14:39:01Z" dockerImageReference: virthost.ostest.test.metalkube.org:5000/localimages/local-test-image@sha256:f1fa97a4e1dab565da2790e6c76f8aecf87c44c9ac7ba53a9c65af96ec472c4a generation: 2 image: sha256:f1fa97a4e1dab565da2790e6c76f8aecf87c44c9ac7ba53a9c65af96ec472c4a tag: latest kind: List metadata: resourceVersion: ""
Looks good, so move to verified.
Since the problem described in this issue should be resolved in a recent advisory, it has been closed.
For information on the advisory (Critical: OpenShift Container Platform 4.16.0 bug fix and security update), and where to find the updated files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2024:0041