-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
4.17
-
None
-
None
-
False
-
Description of problem:
Configured an OpenShift Mirror Installation Registry with MicroShift images, as described by product docs, but MicroShift does not fully start: pods from the LVM Storage operator fail signature checks.
Version-Release number of selected component (if applicable):
How reproducible:
Always. Did it twice on local VMs with same results.
Steps to Reproduce:
1. Configure a mirror registry, as described by https://docs.openshift.com/container-platform/4.17/disconnected/mirroring/installing-mirroring-creating-registry.html
2.Extract an image list from the MicroShift release package, and use that list to populare the mirror registry, as described by https://docs.redhat.com/en/documentation/red_hat_build_of_microshift/4.17/html/embedding_in_a_rhel_for_edge_image/microshift-deploy-with-mirror-registry
3. Install the microshift package and start the microshift service
Actual results:
lvms-operator pods do not start: $ oc get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system csi-snapshot-controller-69ddff88c8-hnl7v 1/1 Running 1 3d16h kube-system csi-snapshot-webhook-74dc497864-2sljq 1/1 Running 1 3d16h openshift-dns dns-default-vtfsb 2/2 Running 3 3d16h openshift-dns node-resolver-lnplm 1/1 Running 1 3d16h openshift-ingress router-default-575b4fc7-g64sd 1/1 Running 1 3d16h openshift-ovn-kubernetes ovnkube-master-5vvbq 4/4 Running 5 3d16h openshift-ovn-kubernetes ovnkube-node-rp645 1/1 Running 2 3d16h openshift-service-ca service-ca-9db855698-ccbvj 1/1 Running 1 3d16h openshift-storage lvms-operator-7f544467bc-gfkj6 0/1 ImagePullBackOff 0 3d16h $ oc describe pod -n openshift-storage lvms-operator-7f544467bc-gfkj6 ... Normal Pulling 50m kubelet Pulling image "registry.redhat.io/lvms4/lvms-rhel9-operator@sha256:bd6dc4d6e90fdbcdb844759e203c9c591abc5ac29a956257a90bda101a37b76e" Warning Failed 50m kubelet Failed to pull image "registry.redhat.io/lvms4/lvms-rhel9-operator@sha256:bd6dc4d6e90fdbcdb844759e203c9c591abc5ac29a956257a90bda101a37b76e": SignatureValidationFailed: copying system image from manifest list: Source image rejected: A signature was required, but no signature exists Warning Failed 50m kubelet Error: SignatureValidationFailed Warning Failed 53m kubelet Error: ImagePullBackOff Normal BackOff 4m3s (x217 over 53m) kubelet Back-off pulling image "registry.redhat.io/lvms4/lvms-rhel9-operator@sha256:bd6dc4d6e90fdbcdb844759e203c9c591abc5ac29a956257a90bda101a37b76e"
Expected results:
All pods start and run fine: $ oc get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system csi-snapshot-controller-69ddff88c8-9llvn 1/1 Running 0 5m46s kube-system csi-snapshot-webhook-74dc497864-2l6w7 1/1 Running 0 5m52s openshift-dns dns-default-m6sck 2/2 Running 0 4m58s openshift-dns node-resolver-tfjwm 1/1 Running 0 5m49s openshift-ingress router-default-575b4fc7-nh67m 1/1 Running 0 5m47s openshift-ovn-kubernetes ovnkube-master-xhw2k 4/4 Running 1 (5m ago) 5m49s openshift-ovn-kubernetes ovnkube-node-8n5xv 1/1 Running 1 (5m1s ago) 5m49s openshift-service-ca service-ca-9db855698-78zf8 1/1 Running 0 5m46s openshift-storage lvms-operator-7f544467bc-r7sqq 1/1 Running 0 5m50s openshift-storage vg-manager-w5j2x 1/1 Running 0 80s
Additional info:
If I replace the container engine policy.json with one that does not require signature validation anywhere, then I can restart microshoft and the LVM Storage operator pods start fine
$ cat /etc/containers/policy.json
{
"default": [
{
"type": "insecureAcceptAnything"
}
],
"transports":
{
"docker-daemon":
{
"": [{"type":"insecureAcceptAnything"}]
}
}
}
But we want customers to deploy and run with signature checks enabled, else they are vulnerable to supply chain attacks.
I think we need a documented and tested proccess to mirror MicroShift images (and also images from selected operators, such as gitops and multus, that you can install from MicroShift RPMS) which also copies their signatures. I have no idea if the mirror registry supports that or if we need additional infrastructure.