-
Bug
-
Resolution: Done-Errata
-
Critical
-
4.14
-
Critical
-
No
-
MCO Sprint 246
-
1
-
False
-
-
This is a clone of issue OCPBUGS-24035. The following is the description of the original issue:
—
Description of problem:
On an SNO a new CA certificate is not loaded after updating user-ca-bundle configmap and as a result the cluster cannot pull images from a registry with a certificate signed by the new CA.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. Update ca bundle.crt replace with a new certificate if applicable ) in `user-ca-bundle` configmap under openshift-config namespace :
* On the node ensure that /etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt was updated with the new certificate
2. Create a pod which uses an image from a registry that has its certificate signed by the new CA cert provided in ca-bundle.crt
3.
Actual results:
Pod fails to pull image
*** Failed to pull image "registry.ztp-hub-00.mobius.lab.eng.rdu2.redhat.com:5000/centos/centos:8": rpc error: { code = Unknown desc = pinging container registry registry.ztp-hub-00.mobius.lab.eng.rdu2.redhat.com : 5000: Get "https://registry.ztp-hub-00.mobius.lab.eng.rdu2.redhat.com:5000/v2/": tls: failed to vierify certificate: x509: certificate signed by unknown authority
* On the node try to reach the registry via curl [https://registry.ztp-hub-00.mobius.lab.eng.rdu2.redhat.com:5000|https://registry.ztp-hub-00.mobius.lab.eng.rdu2.redhat.com:5000/]
** certificate validation fails: curl [https://registry.ztp-hub-00.mobius.lab.eng.rdu2.redhat.com:5000|https://registry.ztp-hub-00.mobius.lab.eng.rdu2.redhat.com:5000/]
curl: (60) SSL certificate problem: self-signed certificate
More details here: [https://curl.se/docs/sslcerts.html]
To be able to create a pod I had to
** Run `sudo update-ca-trust`. After that curl [https//registry.ztp-hub-00.mobius.lab.eng.rdu2.redhat.com:5000|https://registry.ztp-hub-00.mobius.lab.eng.rdu2.redhat.com:5000/]
worked without issues but the pod creation still fails due to tls:
failed to verify certificate: x509: certificate signed by unknown
authority error
** Run `sudo systemctl restart crio`. After that the pod creation succeeded and could pull the image
Expected results:
Additional info:
Attaching must gather
- clones
-
-
- Closed
-
- is blocked by
-
-
- Closed
-
- links to
-
RHBA-2023:7831
OpenShift Container Platform 4.14.z bug fix update
[OCPBUGS-24596] On an SNO the new CA certificate is not loaded after updating user-ca-bundle configmap
verified on 4.14.0-0.ci-2023-12-12-031457
applied mc to update file /etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt
$ cat ca-trust.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: worker
name: update-user-ca-bundle
spec:
config:
ignition:
version: 3.4.0
storage:
files:
- contents:
source: data:text/plain;charset=utf;base64,LS0tLS..==
mode: 420
overwrite: true
path: /etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt
when update is completed, check daemon pod log, crio service is restarted, drain and reboot are skipped.
I1212 07:28:49.394886 2186 update.go:1977] Starting update from rendered-worker-d8d05fd0f5c3083f1bb2924010420974 to rendered-worker-36cdd0c95f258bcdd4ba2ecd6c0aca27: &{osUpdate:false kargs:false fips:false passwd:false files:true units:false kernelType:false extensions:false}
I1212 07:28:49.396425 2186 helpers.go:987] File diff: detected change to /etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt
I1212 07:28:49.396501 2186 update.go:497] Changes do not require drain, skipping.
I1212 07:28:49.396509 2186 update.go:1199] Updating files
I1212 07:28:49.396515 2186 file_writers.go:163] Writing file "/usr/local/bin/nm-clean-initrd-state.sh"
I1212 07:28:49.398422 2186 file_writers.go:163] Writing file "/etc/NetworkManager/conf.d/01-ipv6.conf"
I1212 07:28:49.400089 2186 file_writers.go:163] Writing file "/etc/NetworkManager/conf.d/20-keyfiles.conf"
I1212 07:28:49.401410 2186 file_writers.go:163] Writing file "/etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt"
I1212 07:28:49.402811 2186 file_writers.go:163] Writing file "/etc/audit/rules.d/mco-audit-quiet-containers.rules"
I1212 07:28:49.403929 2186 file_writers.go:163] Writing file "/etc/tmpfiles.d/cleanup-cni.conf"
I1212 07:28:49.405527 2186 file_writers.go:163] Writing file "/usr/local/bin/configure-ovs.sh"
I1212 07:28:49.407920 2186 file_writers.go:163] Writing file "/etc/containers/storage.conf"
I1212 07:28:49.409473 2186 file_writers.go:163] Writing file "/etc/mco/proxy.env"
I1212 07:28:49.410850 2186 file_writers.go:163] Writing file "/etc/systemd/system.conf.d/10-default-env-godebug.conf"
I1212 07:28:49.412340 2186 file_writers.go:163] Writing file "/etc/mco/internal-registry-pull-secret.json"
I1212 07:28:49.414433 2186 file_writers.go:163] Writing file "/etc/modules-load.d/iptables.conf"
I1212 07:28:49.415749 2186 file_writers.go:163] Writing file "/etc/node-sizing-enabled.env"
I1212 07:28:49.417419 2186 file_writers.go:163] Writing file "/usr/local/sbin/dynamic-system-reserved-calc.sh"
I1212 07:28:49.419009 2186 file_writers.go:163] Writing file "/etc/systemd/system.conf.d/kubelet-cgroups.conf"
I1212 07:28:49.420479 2186 file_writers.go:163] Writing file "/etc/systemd/system/kubelet.service.d/20-logging.conf"
I1212 07:28:49.422207 2186 file_writers.go:163] Writing file "/etc/NetworkManager/conf.d/sdn.conf"
I1212 07:28:49.423834 2186 file_writers.go:163] Writing file "/etc/NetworkManager/dispatcher.d/pre-up.d/10-ofport-request.sh"
I1212 07:28:49.425646 2186 file_writers.go:163] Writing file "/var/lib/kubelet/config.json"
I1212 07:28:49.427096 2186 file_writers.go:163] Writing file "/etc/kubernetes/ca.crt"
I1212 07:28:49.428622 2186 file_writers.go:163] Writing file "/etc/sysctl.d/arp.conf"
I1212 07:28:49.429838 2186 file_writers.go:163] Writing file "/etc/sysctl.d/inotify.conf"
I1212 07:28:49.431497 2186 file_writers.go:163] Writing file "/etc/sysctl.d/enable-userfaultfd.conf"
I1212 07:28:49.432860 2186 file_writers.go:163] Writing file "/etc/sysctl.d/vm-max-map.conf"
I1212 07:28:49.447766 2186 file_writers.go:163] Writing file "/usr/local/bin/kubenswrapper"
I1212 07:28:49.449071 2186 file_writers.go:223] Writing systemd unit "NetworkManager-clean-initrd-state.service"
I1212 07:28:49.450768 2186 file_writers.go:223] Writing systemd unit "aws-kubelet-nodename.service"
I1212 07:28:49.451826 2186 file_writers.go:223] Writing systemd unit "aws-kubelet-providerid.service"
I1212 07:28:49.453195 2186 file_writers.go:137] Writing systemd unit dropin "01-kubens.conf"
I1212 07:28:49.454634 2186 file_writers.go:123] Dropin for 10-mco-default-env.conf has no content, skipping write
I1212 07:28:49.454649 2186 file_writers.go:137] Writing systemd unit dropin "10-mco-profile-unix-socket.conf"
I1212 07:28:49.456168 2186 file_writers.go:137] Writing systemd unit dropin "05-mco-ordering.conf"
I1212 07:28:49.457247 2186 file_writers.go:137] Writing systemd unit dropin "10-mco-default-madv.conf"
I1212 07:28:49.842034 2186 update.go:1485] Preset systemd unit crio.service
I1212 07:28:49.842059 2186 file_writers.go:137] Writing systemd unit dropin "mco-disabled.conf"
I1212 07:28:49.848499 2186 update.go:1522] Could not reset unit preset for docker.socket, skipping. (Error msg: error running preset on unit: Failed to preset unit: Unit file docker.socket does not exist.
)
I1212 07:28:49.848519 2186 file_writers.go:223] Writing systemd unit "firstboot-osupdate.target"
I1212 07:28:49.850215 2186 file_writers.go:223] Writing systemd unit "kubelet-auto-node-size.service"
I1212 07:28:49.851618 2186 file_writers.go:223] Writing systemd unit "kubelet-dependencies.target"
I1212 07:28:50.232954 2186 update.go:1485] Preset systemd unit kubelet-dependencies.target
I1212 07:28:50.232988 2186 file_writers.go:137] Writing systemd unit dropin "20-node-env.conf"
I1212 07:28:50.234500 2186 file_writers.go:137] Writing systemd unit dropin "01-kubens.conf"
I1212 07:28:50.235939 2186 file_writers.go:123] Dropin for 10-mco-default-env.conf has no content, skipping write
I1212 07:28:50.235956 2186 file_writers.go:137] Writing systemd unit dropin "10-mco-default-madv.conf"
I1212 07:28:50.237389 2186 file_writers.go:223] Writing systemd unit "kubelet.service"
I1212 07:28:50.238802 2186 file_writers.go:223] Writing systemd unit "kubens.service"
I1212 07:28:50.240183 2186 file_writers.go:223] Writing systemd unit "machine-config-daemon-firstboot.service"
I1212 07:28:50.241663 2186 file_writers.go:223] Writing systemd unit "machine-config-daemon-pull.service"
I1212 07:28:50.243211 2186 file_writers.go:223] Writing systemd unit "node-valid-hostname.service"
I1212 07:28:50.244691 2186 file_writers.go:223] Writing systemd unit "nodeip-configuration.service"
I1212 07:28:50.246007 2186 file_writers.go:223] Writing systemd unit "ovs-configuration.service"
I1212 07:28:50.247573 2186 file_writers.go:137] Writing systemd unit dropin "10-ovs-vswitchd-restart.conf"
I1212 07:28:50.632350 2186 update.go:1485] Preset systemd unit ovs-vswitchd.service
I1212 07:28:50.632390 2186 file_writers.go:137] Writing systemd unit dropin "10-ovsdb-restart.conf"
I1212 07:28:50.633872 2186 file_writers.go:123] Dropin for 10-mco-default-env.conf has no content, skipping write
I1212 07:28:50.639084 2186 update.go:1522] Could not reset unit preset for pivot.service, skipping. (Error msg: error running preset on unit: Failed to preset unit: Unit file pivot.service does not exist.
)
I1212 07:28:50.639113 2186 file_writers.go:123] Dropin for 10-mco-default-env.conf has no content, skipping write
I1212 07:28:51.027963 2186 update.go:1485] Preset systemd unit rpm-ostreed.service
I1212 07:28:51.027983 2186 file_writers.go:137] Writing systemd unit dropin "mco-disabled.conf"
I1212 07:28:51.034581 2186 update.go:1522] Could not reset unit preset for zincati.service, skipping. (Error msg: error running preset on unit: Failed to preset unit: Unit file zincati.service does not exist.
)
I1212 07:28:51.455293 2186 update.go:1463] Enabled systemd units: [NetworkManager-clean-initrd-state.service aws-kubelet-nodename.service aws-kubelet-providerid.service firstboot-osupdate.target kubelet-auto-node-size.service kubelet.service machine-config-daemon-firstboot.service machine-config-daemon-pull.service node-valid-hostname.service openvswitch.service ovs-configuration.service ovsdb-server.service]
I1212 07:28:51.844226 2186 update.go:1474] Disabled systemd units [kubens.service nodeip-configuration.service]
I1212 07:28:51.844255 2186 update.go:1262] Deleting stale data
I1212 07:28:51.844311 2186 update.go:1660] updating the permission of the kubeconfig to: 0o600
I1212 07:28:51.844338 2186 update.go:1626] Checking if absent users need to be disconfigured
I1212 07:28:51.870884 2186 update.go:1651] Password has been configured
I1212 07:28:52.442340 2186 update.go:1962] Running: systemctl restart crio
I1212 07:28:53.196088 2186 update.go:1977] crio config restarted successfully! Desired config rendered-worker-36cdd0c95f258bcdd4ba2ecd6c0aca27 has been applied, skipping reboot
I1212 07:28:53.200800 2186 daemon.go:1308] Previous boot ostree-finalize-staged.service appears successful
I1212 07:28:53.200817 2186 daemon.go:1432] Current config: rendered-worker-d8d05fd0f5c3083f1bb2924010420974
I1212 07:28:53.200825 2186 daemon.go:1433] Desired config: rendered-worker-36cdd0c95f258bcdd4ba2ecd6c0aca27
I1212 07:28:53.200830 2186 daemon.go:1441] state: Done
I1212 07:28:53.202643 2186 daemon.go:1914] Completing update to target MachineConfig: rendered-worker-36cdd0c95f258bcdd4ba2ecd6c0aca27
I1212 07:29:03.220406 2186 update.go:1977] Update completed for config rendered-worker-36cdd0c95f258bcdd4ba2ecd6c0aca27 and node has been successfully uncordoned
I1212 07:29:03.234977 2186 daemon.go:1939] In desired state MachineConfig: rendered-worker-36cdd0c95f258bcdd4ba2ecd6c0aca27
OpenShift Jira Bot
added a comment - Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified.
OpenShift Jira Bot
added a comment - Hi cdoern@redhat.com ,
Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified.
Since the problem described in this issue should be resolved in a recent advisory, it has been closed.
For information on the advisory (Important: OpenShift Container Platform 4.14.7 bug fix and security update), and where to find the updated files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2023:7831