Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-24035

On an SNO the new CA certificate is not loaded after updating user-ca-bundle configmap

    XMLWordPrintable

Details

    • ?
    • Critical
    • Yes
    • MCO Sprint 246
    • 1
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, the Machine Config Operator allowed user-provided certificate authority bundle changes to be applied to the cluster without needing a machine config, to prevent disruption. Because of this, the `user-ca` bundle was not propagating to applications running on the cluster and required a reboot to see the changes take effect. With this update, the MCO now runs the `update-ca-trust` command and restarts the CRI-O service so that the new CA properly applies. (link:https://issues.redhat.com/browse/OCPBUGS-24035[*OCPBUGS-24035*])
      Show
      * Previously, the Machine Config Operator allowed user-provided certificate authority bundle changes to be applied to the cluster without needing a machine config, to prevent disruption. Because of this, the `user-ca` bundle was not propagating to applications running on the cluster and required a reboot to see the changes take effect. With this update, the MCO now runs the `update-ca-trust` command and restarts the CRI-O service so that the new CA properly applies. (link: https://issues.redhat.com/browse/OCPBUGS-24035 [* OCPBUGS-24035 *])
    • Bug Fix
    • Done

    Description

      Description of problem:

           
        On an SNO a new CA certificate is not loaded after updating user-ca-bundle
       configmap and as a result the cluster cannot pull images from a 
      registry with a certificate signed by the new CA.

      Version-Release number of selected component (if applicable):

          

      How reproducible:

          

      Steps to Reproduce:

          1. Update ca bundle.crt replace with a new certificate if applicable )      in `user-ca-bundle` configmap under openshift-config namespace : 
        * On the node ensure that /etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt was updated with the new certificate 
           2. Create a pod which uses an image from a registry that has its certificate signed by the new CA cert provided in ca-bundle.crt 
           3.
          

      Actual results:

          Pod fails to pull image
       *** Failed to pull image "registry.ztp-hub-00.mobius.lab.eng.rdu2.redhat.com:5000/centos/centos:8": rpc error: {  code  = Unknown desc = pinging container registry registry.ztp-hub-00.mobius.lab.eng.rdu2.redhat.com : 5000: Get "https://registry.ztp-hub-00.mobius.lab.eng.rdu2.redhat.com:5000/v2/": tls: failed to vierify certificate: x509: certificate signed by unknown authority 
        * On the node try to reach the registry via curl [https://registry.ztp-hub-00.mobius.lab.eng.rdu2.redhat.com:5000|https://registry.ztp-hub-00.mobius.lab.eng.rdu2.redhat.com:5000/] 
      ** certificate validation fails: curl [https://registry.ztp-hub-00.mobius.lab.eng.rdu2.redhat.com:5000|https://registry.ztp-hub-00.mobius.lab.eng.rdu2.redhat.com:5000/] 
       curl: (60) SSL certificate problem: self-signed certificate 
       More details here: [https://curl.se/docs/sslcerts.html] 
      
       To be able to create a pod I had to 
        ** Run `sudo update-ca-trust`. After that curl [https//registry.ztp-hub-00.mobius.lab.eng.rdu2.redhat.com:5000|https://registry.ztp-hub-00.mobius.lab.eng.rdu2.redhat.com:5000/]
       worked without issues but the pod creation still fails due to tls: 
      failed to verify certificate: x509: certificate signed by unknown 
      authority error 
        ** Run `sudo systemctl restart crio`. After that the pod creation succeeded and could pull the image

      Expected results:

          

      Additional info:

      Attaching must gather    

      Attachments

        Issue Links

          Activity

            People

              cdoern@redhat.com Charlie Doern
              mcornea@redhat.com Marius Cornea
              Rio Liu Rio Liu
              Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: