Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-44695

Cannot deploy MicroShift air-gapped unless I disable container image signature checks

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • 4.17
    • MicroShift
    • None
    • None
    • 3
    • uShift Sprint 264
    • 1
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      Configured an OpenShift Mirror Installation Registry with MicroShift images, as described by product docs, but MicroShift does not fully start: pods from the LVM Storage operator fail signature checks.

      Version-Release number of selected component (if applicable):

          

      How reproducible:

      Always. Did it twice on local VMs with same results. 

      Steps to Reproduce:

          1. Configure a mirror registry, as described by https://docs.openshift.com/container-platform/4.17/disconnected/mirroring/installing-mirroring-creating-registry.html
          2.Extract an image list from the MicroShift release package, and use that list to populare the mirror registry, as described by https://docs.redhat.com/en/documentation/red_hat_build_of_microshift/4.17/html/embedding_in_a_rhel_for_edge_image/microshift-deploy-with-mirror-registry         
          3. Install the microshift package and start the microshift service
          

      Actual results:

      lvms-operator pods do not start:
      
      $ oc get pod -A
      NAMESPACE                  NAME                                       READY   STATUS             RESTARTS   AGE
      kube-system                csi-snapshot-controller-69ddff88c8-hnl7v   1/1     Running            1          3d16h
      kube-system                csi-snapshot-webhook-74dc497864-2sljq      1/1     Running            1          3d16h
      openshift-dns              dns-default-vtfsb                          2/2     Running            3          3d16h
      openshift-dns              node-resolver-lnplm                        1/1     Running            1          3d16h
      openshift-ingress          router-default-575b4fc7-g64sd              1/1     Running            1          3d16h
      openshift-ovn-kubernetes   ovnkube-master-5vvbq                       4/4     Running            5          3d16h
      openshift-ovn-kubernetes   ovnkube-node-rp645                         1/1     Running            2          3d16h
      openshift-service-ca       service-ca-9db855698-ccbvj                 1/1     Running            1          3d16h
      openshift-storage          lvms-operator-7f544467bc-gfkj6             0/1     ImagePullBackOff   0          3d16h
      $ oc describe pod -n openshift-storage lvms-operator-7f544467bc-gfkj6 
      ...
        Normal   Pulling           50m                     kubelet            Pulling image "registry.redhat.io/lvms4/lvms-rhel9-operator@sha256:bd6dc4d6e90fdbcdb844759e203c9c591abc5ac29a956257a90bda101a37b76e"
        Warning  Failed            50m                     kubelet            Failed to pull image "registry.redhat.io/lvms4/lvms-rhel9-operator@sha256:bd6dc4d6e90fdbcdb844759e203c9c591abc5ac29a956257a90bda101a37b76e": SignatureValidationFailed: copying system image from manifest list: Source image rejected: A signature was required, but no signature exists
        Warning  Failed            50m                     kubelet            Error: SignatureValidationFailed
        Warning  Failed            53m                     kubelet            Error: ImagePullBackOff
        Normal   BackOff           4m3s (x217 over 53m)    kubelet            Back-off pulling image "registry.redhat.io/lvms4/lvms-rhel9-operator@sha256:bd6dc4d6e90fdbcdb844759e203c9c591abc5ac29a956257a90bda101a37b76e"

      Expected results:

      All pods start and run fine:
      
      $ oc get pod -A
      NAMESPACE                  NAME                                       READY   STATUS    RESTARTS       AGE
      kube-system                csi-snapshot-controller-69ddff88c8-9llvn   1/1     Running   0              5m46s
      kube-system                csi-snapshot-webhook-74dc497864-2l6w7      1/1     Running   0              5m52s
      openshift-dns              dns-default-m6sck                          2/2     Running   0              4m58s
      openshift-dns              node-resolver-tfjwm                        1/1     Running   0              5m49s
      openshift-ingress          router-default-575b4fc7-nh67m              1/1     Running   0              5m47s
      openshift-ovn-kubernetes   ovnkube-master-xhw2k                       4/4     Running   1 (5m ago)     5m49s
      openshift-ovn-kubernetes   ovnkube-node-8n5xv                         1/1     Running   1 (5m1s ago)   5m49s
      openshift-service-ca       service-ca-9db855698-78zf8                 1/1     Running   0              5m46s
      openshift-storage          lvms-operator-7f544467bc-r7sqq             1/1     Running   0              5m50s
      openshift-storage          vg-manager-w5j2x                           1/1     Running   0              80s

      Additional info:

      If I replace the container engine policy.json with one that does not require signature validation anywhere, then I can restart microshoft and the LVM Storage operator pods start fine
      
      $ cat /etc/containers/policy.json 
      {
          "default": [
              {
                  "type": "insecureAcceptAnything"
              }
          ],
          "transports":
              {
                  "docker-daemon":
                      {
                          "": [{"type":"insecureAcceptAnything"}]
                      }
              }
      }
      
      But we want customers to deploy and run with signature checks enabled, else they are vulnerable to supply chain attacks.
      
      I think we need a documented and tested proccess to mirror MicroShift images (and also images from selected operators, such as gitops and multus, that you can install from MicroShift RPMS) which also copies their signatures. I have no idea if the mirror registry supports that or if we need additional infrastructure.

              ggiguash@redhat.com Gregory Giguashvili
              rht-flozano Fernando Lozano
              John George John George
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated: