-
Bug
-
Resolution: Unresolved
-
Undefined
-
None
-
4.14.z
-
None
-
False
-
-
Description of problem:
During an Image-based upgrade scale test where 3486 SNOs were Image based upgraded from 4.14.31 to 4.15.20 and then rolled back to 4.14. 31, one cluster (vm01873) after rollback was missing the klusterlet. This was observed because that cluster was unable to be finalized. It is clear that the cluster could not finalize or apply a policy to its ibu object since none of the ACM policy components existed on the cluster after the rollback
Version-Release number of selected component (if applicable):
Hub 4.16.3 Deployed clusters 4.14.31 (upgraded to 4.15.20) ACM - 2.11.0-DOWNSTREAM-2024-07-10-21-49-48 LCA - 4.16.0
How reproducible:
Rare only one cluster produced this issue out of 3486 rolled back
Steps to Reproduce:
1.
2.
3.
Actual results:
Expected results:
Additional info:
Although the klusterlet did not exist while attempting to finalize the cluster, it later appeared again. (I assume the hub had eventually re-managed the cluster.) IBU during attempted finalize # oc --kubeconfig /root/hv-vm/kc/vm01873/kubeconfig get ibuNAME AGE DESIRED STAGE STATE DETAILS upgrade 15h Rollback Completed Rollback completed Pods after rollback (During attempted finalize): # oc --kubeconfig /root/hv-vm/kc/vm01873/kubeconfig get po -A NAMESPACE NAME READY STATUS RESTARTS AGE assisted-installer assisted-installer-controller-tbrlg 0/1 Completed 0 2d23h open-cluster-management-addon-observability endpoint-observability-operator-7645d47887-gcr8m 1/1 Running 2 2d23h open-cluster-management-addon-observability metrics-collector-deployment-b8459ffcd-dt5wz 0/1 CrashLoopBackOff 23 (2m32s ago) 95m openshift-adp openshift-adp-controller-manager-7c8cdb89bd-t27dx 1/1 Running 3 2d23h openshift-adp velero-57c8599c7c-4xj2p 1/1 Running 2 2d23h openshift-apiserver-operator openshift-apiserver-operator-74b9b7c7b9-gfk7n 1/1 Running 4 2d23h openshift-apiserver apiserver-7ff8d59dc9-zs5hx 2/2 Running 10 (96m ago) 2d23h openshift-authentication-operator authentication-operator-678c4f9b6f-npd7f 1/1 Running 5 (96m ago) 2d23h openshift-authentication oauth-openshift-59587dd9b4-82gml 1/1 Running 4 (96m ago) 2d23h openshift-cloud-controller-manager-operator cluster-cloud-controller-manager-operator-9f9bb55bf-b9r57 2/2 Running 4 2d23h openshift-cloud-credential-operator cloud-credential-operator-6756b577bf-m6bqg 2/2 Running 4 2d23h openshift-cluster-machine-approver machine-approver-75fbcd7957-xvstg 2/2 Running 4 2d23h openshift-cluster-node-tuning-operator cluster-node-tuning-operator-55b84c985-hps5k 1/1 Running 2 2d23h openshift-cluster-node-tuning-operator tuned-rfkxg 1/1 Running 2 2d23h openshift-cluster-version cluster-version-operator-69b5875f7-nb24x 1/1 Running 4 2d23h openshift-config-operator openshift-config-operator-6579b9775b-96pd2 1/1 Running 8 (95m ago) 2d23h openshift-controller-manager-operator openshift-controller-manager-operator-5fcf4bb64c-74dhv 1/1 Running 4 (96m ago) 2d23h openshift-controller-manager controller-manager-bb5dc549d-crg22 1/1 Running 2 2d openshift-dns-operator dns-operator-85766944-bbmlh 2/2 Running 4 2d23h openshift-dns dns-default-vcjpg 2/2 Running 4 2d23h openshift-dns node-resolver-r5czb 1/1 Running 2 2d23h openshift-etcd-operator etcd-operator-6f8fdcc44-wkmvr 1/1 Running 3 2d23h openshift-etcd etcd-vm01873 4/4 Running 8 2d23h openshift-etcd installer-2-vm01873 0/1 Completed 0 2d23h openshift-etcd installer-3-vm01873 0/1 Completed 0 2d23h openshift-ingress-canary ingress-canary-655m9 1/1 Running 2 2d23h openshift-ingress-operator ingress-operator-56df5f9f64-874b6 2/2 Running 11 (93m ago) 2d23h openshift-ingress router-default-56d759fb45-tkk8p 1/1 Running 6 (95m ago) 2d23h openshift-kube-apiserver-operator kube-apiserver-operator-779647dd5c-psrts 1/1 Running 3 2d23h openshift-kube-apiserver installer-10-vm01873 0/1 Completed 0 2d4h openshift-kube-apiserver installer-11-vm01873 0/1 Completed 0 2d1h openshift-kube-apiserver installer-12-vm01873 0/1 Completed 0 2d openshift-kube-apiserver installer-9-vm01873 0/1 Completed 0 2d4h openshift-kube-apiserver kube-apiserver-vm01873 5/5 Running 11 (96m ago) 2d openshift-kube-apiserver revision-pruner-10-vm01873 0/1 Completed 0 2d4h openshift-kube-apiserver revision-pruner-11-vm01873 0/1 Completed 0 2d1h openshift-kube-apiserver revision-pruner-12-vm01873 0/1 Completed 0 2d openshift-kube-apiserver revision-pruner-8-vm01873 0/1 Completed 0 2d4h openshift-kube-apiserver revision-pruner-9-vm01873 0/1 Completed 0 2d4h openshift-kube-controller-manager-operator kube-controller-manager-operator-74cd7b86bd-5nfmq 1/1 Running 5 (96m ago) 2d23h openshift-kube-controller-manager installer-3-vm01873 0/1 Error 0 2d23h openshift-kube-controller-manager installer-4-vm01873 0/1 Completed 0 2d23h openshift-kube-controller-manager installer-5-vm01873 0/1 Completed 0 2d23h openshift-kube-controller-manager installer-6-vm01873 0/1 Completed 0 2d23h openshift-kube-controller-manager kube-controller-manager-vm01873 4/4 Running 17 (97m ago) 2d23h openshift-kube-scheduler-operator openshift-kube-scheduler-operator-748d4bbd6c-72kwt 1/1 Running 4 (96m ago) 2d23h openshift-kube-scheduler installer-2-vm01873 0/1 Completed 0 2d23h openshift-kube-scheduler installer-3-retry-1-vm01873 0/1 Completed 0 2d23h openshift-kube-scheduler installer-3-vm01873 0/1 Error 0 2d23h openshift-kube-scheduler installer-6-vm01873 0/1 Completed 0 2d23h openshift-kube-scheduler openshift-kube-scheduler-vm01873 3/3 Running 6 2d23h openshift-kube-storage-version-migrator-operator kube-storage-version-migrator-operator-54644d5786-hlss5 1/1 Running 4 (96m ago) 2d23h openshift-kube-storage-version-migrator migrator-56cff8468d-7s548 1/1 Running 3 2d23h openshift-lifecycle-agent lca-prep-precache-ht578 0/1 Completed 0 7h38m openshift-lifecycle-agent lca-prep-stateroot-setup-w7gwm 0/1 Completed 0 7h43m openshift-lifecycle-agent lifecycle-agent-controller-manager-54c9b5fd89-zsl7r 2/2 Running 7 (96m ago) 2d23h openshift-local-storage diskmaker-manager-2vxdp 2/2 Running 4 2d23h openshift-local-storage local-storage-operator-674f6dcf97-jdxw9 1/1 Running 2 2d23h openshift-logging cluster-logging-operator-57746666b6-87fmb 1/1 Running 3 (93m ago) 2d23h openshift-logging collector-mp5ww 1/1 Running 2 2d23h openshift-machine-config-operator kube-rbac-proxy-crio-vm01873 1/1 Running 7 2d23h openshift-machine-config-operator machine-config-controller-56fc59fdcf-9z4fl 2/2 Running 4 2d23h openshift-machine-config-operator machine-config-daemon-dfgdq 2/2 Running 4 2d23h openshift-machine-config-operator machine-config-operator-575bdfffc6-wbqmc 2/2 Running 4 2d23h openshift-machine-config-operator machine-config-server-rnfc6 1/1 Running 2 2d23h openshift-marketplace 1b2c2453438ee4107ff3cff145ed8069da558a8018f6626c4e79a26d50l6jk2 0/1 Completed 0 2d23h openshift-marketplace 1bdc271018164eb689792710b23bd6084a65e86807aa857636d23c1b52nsq5b 0/1 Completed 0 2d23h openshift-marketplace 4e692e78da9b950f392ff2575748192ff63f400191354113107002ee95hslwh 0/1 Completed 0 2d23h openshift-marketplace c98953b06c18521e7fb4292b7767ae0febdc6fbda19e4991737e35868f5mrkk 0/1 Completed 0 2d23h openshift-marketplace cc42e337969c795e1a26850fca1b25e1a4bc7b46f1f1d6df9ae5bc4ee8sbnkh 0/1 Completed 0 2d23h openshift-marketplace f45b93fb23220872b7e775ce6452784c7abe59f24d6a81693c18729c04kctp6 0/1 Completed 0 2d23h openshift-marketplace marketplace-operator-5c87d74f6c-n2cgv 1/1 Running 6 (96m ago) 2d23h openshift-marketplace rh-du-operators-79dtl 1/1 Running 2 2d23h openshift-monitoring cluster-monitoring-operator-6f68bbf775-mnqvb 1/1 Running 4 (96m ago) 2d23h openshift-monitoring kube-state-metrics-5d5f7556bc-ndw7k 3/3 Running 6 2d23h openshift-monitoring node-exporter-64b2n 2/2 Running 4 2d23h openshift-monitoring openshift-state-metrics-7b754dcf96-kfvsd 3/3 Running 6 2d23h openshift-monitoring prometheus-adapter-5c58d77db6-d6jmz 1/1 Running 2 47h openshift-monitoring prometheus-k8s-0 6/6 Running 12 2d23h openshift-monitoring prometheus-operator-64c4867485-gbnls 2/2 Running 4 2d23h openshift-monitoring prometheus-operator-admission-webhook-7cd56c97cd-wg5n5 1/1 Running 2 2d23h openshift-monitoring thanos-querier-c5b48f55f-hspfb 6/6 Running 13 2d23h openshift-multus multus-additional-cni-plugins-5tks6 1/1 Running 2 2d23h openshift-multus multus-admission-controller-b6d946975-285zw 2/2 Running 0 93m openshift-multus multus-jwxkk 1/1 Running 3 2d23h openshift-multus network-metrics-daemon-shz4w 2/2 Running 4 2d23h openshift-network-node-identity network-node-identity-fndbw 2/2 Running 6 2d23h openshift-network-operator network-operator-55cdbbdf9-f8ld6 1/1 Running 3 2d23h openshift-oauth-apiserver apiserver-d5c77fb74-fbjm9 1/1 Running 4 (96m ago) 2d23h openshift-operator-lifecycle-manager catalog-operator-5bc76757b-d4tvl 1/1 Running 2 2d23h openshift-operator-lifecycle-manager collect-profiles-28696200-kxh6s 0/1 Completed 0 33m openshift-operator-lifecycle-manager collect-profiles-28696215-rdsnd 0/1 Completed 0 18m openshift-operator-lifecycle-manager collect-profiles-28696230-pwkqt 0/1 Completed 0 3m33s openshift-operator-lifecycle-manager olm-operator-5d6488c5f6-trgl5 1/1 Running 2 2d23h openshift-operator-lifecycle-manager package-server-manager-58d76848f9-nvrtp 1/1 Running 2 2d23h openshift-operator-lifecycle-manager packageserver-74c9bf7785-dzm6f 1/1 Running 6 (95m ago) 2d23h openshift-ovn-kubernetes ovnkube-control-plane-7bb497b9f-dzkm5 2/2 Running 4 2d23h openshift-ovn-kubernetes ovnkube-node-wzhm4 8/8 Running 17 (98m ago) 2d23h openshift-ptp linuxptp-daemon-slq25 2/2 Running 4 2d23h openshift-ptp ptp-operator-5ff4f97b98-xd54q 1/1 Running 2 2d23h openshift-route-controller-manager route-controller-manager-75c5b7f645-52xlz 1/1 Running 4 (96m ago) 2d openshift-service-ca-operator service-ca-operator-cfb65db99-9jxtz 1/1 Running 4 2d23h openshift-service-ca service-ca-5b587f8669-kb85t 1/1 Running 2 2d23h openshift-sriov-network-operator network-resources-injector-zkk6q 1/1 Running 2 2d23h openshift-sriov-network-operator operator-webhook-7lm49 1/1 Running 2 2d23h openshift-sriov-network-operator sriov-network-config-daemon-9bbch 1/1 Running 2 2d23h openshift-sriov-network-operator sriov-network-operator-7b5d8dd8bf-287j9 1/1 Running 2 2d23h
During finalize, we can see that the klusterlet pods did not exist, the next morning while further investigating, the pods had re-appeared, and we can see that they have a much younger age compared to other pods and compared to clusters that had successfully rolled back and finalized.
# oc --kubeconfig /root/hv-vm/kc/vm01873/kubeconfig get po -A NAMESPACE NAME READY STATUS RESTARTS AGE assisted-installer assisted-installer-controller-tbrlg 0/1 Completed 0 3d14h open-cluster-management-addon-observability endpoint-observability-operator-7645d47887-gcr8m 1/1 Running 2 3d13h open-cluster-management-addon-observability metrics-collector-deployment-774c7f9bdd-ln7qr 1/1 Running 0 13h open-cluster-management-agent-addon cluster-proxy-proxy-agent-867577f6c6-r2hlv 3/3 Running 0 13h open-cluster-management-agent-addon config-policy-controller-6bc86559c4-89scx 2/2 Running 2 (13h ago) 13h open-cluster-management-agent-addon governance-policy-framework-6f87985d8d-d8msl 2/2 Running 0 13h open-cluster-management-agent-addon klusterlet-addon-search-5fb55bc9f8-46ptp 1/1 Running 0 13h open-cluster-management-agent-addon klusterlet-addon-workmgr-86556bf4f8-j5ngj 1/1 Running 0 13h open-cluster-management-agent-addon managed-serviceaccount-addon-agent-58dd8675b4-99c7k 1/1 Running 0 13h open-cluster-management-agent klusterlet-6797c5c856-vmqq7 1/1 Running 0 13h open-cluster-management-agent klusterlet-agent-69d5dddbc6-tj5h9 1/1 Running 0 13h openshift-adp openshift-adp-controller-manager-7c8cdb89bd-t27dx 1/1 Running 3 3d13h openshift-adp velero-57c8599c7c-4xj2p 1/1 Running 2 3d13h openshift-apiserver-operator openshift-apiserver-operator-74b9b7c7b9-gfk7n 1/1 Running 4 3d14h openshift-apiserver apiserver-7ff8d59dc9-zs5hx 2/2 Running 10 (15h ago) 3d13h openshift-authentication-operator authentication-operator-678c4f9b6f-npd7f 1/1 Running 5 (15h ago) 3d14h openshift-authentication oauth-openshift-59587dd9b4-82gml 1/1 Running 4 (15h ago) 3d13h openshift-cloud-controller-manager-operator cluster-cloud-controller-manager-operator-9f9bb55bf-b9r57 2/2 Running 4 3d14h openshift-cloud-credential-operator cloud-credential-operator-6756b577bf-m6bqg 2/2 Running 4 3d14h openshift-cluster-machine-approver machine-approver-75fbcd7957-xvstg 2/2 Running 4 3d13h openshift-cluster-node-tuning-operator cluster-node-tuning-operator-55b84c985-hps5k 1/1 Running 2 3d14h openshift-cluster-node-tuning-operator tuned-rfkxg 1/1 Running 2 3d13h openshift-cluster-version cluster-version-operator-69b5875f7-nb24x 1/1 Running 4 3d13h openshift-config-operator openshift-config-operator-6579b9775b-96pd2 1/1 Running 8 (15h ago) 3d14h openshift-controller-manager-operator openshift-controller-manager-operator-5fcf4bb64c-74dhv 1/1 Running 4 (15h ago) 3d14h openshift-controller-manager controller-manager-bb5dc549d-crg22 1/1 Running 2 2d14h openshift-dns-operator dns-operator-85766944-bbmlh 2/2 Running 4 3d14h openshift-dns dns-default-vcjpg 2/2 Running 4 3d13h openshift-dns node-resolver-r5czb 1/1 Running 2 3d13h openshift-etcd-operator etcd-operator-6f8fdcc44-wkmvr 1/1 Running 3 3d14h openshift-etcd etcd-vm01873 4/4 Running 8 3d13h openshift-etcd installer-2-vm01873 0/1 Completed 0 3d13h openshift-etcd installer-3-vm01873 0/1 Completed 0 3d13h openshift-ingress-canary ingress-canary-655m9 1/1 Running 2 3d13h openshift-ingress-operator ingress-operator-56df5f9f64-874b6 2/2 Running 11 (15h ago) 3d14h openshift-ingress router-default-56d759fb45-tkk8p 1/1 Running 6 (15h ago) 3d13h openshift-kube-apiserver-operator kube-apiserver-operator-779647dd5c-psrts 1/1 Running 3 3d14h openshift-kube-apiserver installer-10-vm01873 0/1 Completed 0 2d19h openshift-kube-apiserver installer-11-vm01873 0/1 Completed 0 2d15h openshift-kube-apiserver installer-12-vm01873 0/1 Completed 0 2d14h openshift-kube-apiserver installer-9-vm01873 0/1 Completed 0 2d19h openshift-kube-apiserver kube-apiserver-vm01873 5/5 Running 11 (15h ago) 2d14h openshift-kube-apiserver revision-pruner-10-vm01873 0/1 Completed 0 2d19h openshift-kube-apiserver revision-pruner-11-vm01873 0/1 Completed 0 2d15h openshift-kube-apiserver revision-pruner-12-vm01873 0/1 Completed 0 2d14h openshift-kube-apiserver revision-pruner-8-vm01873 0/1 Completed 0 2d19h openshift-kube-apiserver revision-pruner-9-vm01873 0/1 Completed 0 2d19h openshift-kube-controller-manager-operator kube-controller-manager-operator-74cd7b86bd-5nfmq 1/1 Running 5 (15h ago) 3d14h openshift-kube-controller-manager installer-3-vm01873 0/1 Error 0 3d13h openshift-kube-controller-manager installer-4-vm01873 0/1 Completed 0 3d13h openshift-kube-controller-manager installer-5-vm01873 0/1 Completed 0 3d13h openshift-kube-controller-manager installer-6-vm01873 0/1 Completed 0 3d13h openshift-kube-controller-manager kube-controller-manager-vm01873 4/4 Running 17 (15h ago) 3d13h openshift-kube-scheduler-operator openshift-kube-scheduler-operator-748d4bbd6c-72kwt 1/1 Running 4 (15h ago) 3d14h openshift-kube-scheduler installer-2-vm01873 0/1 Completed 0 3d13h openshift-kube-scheduler installer-3-retry-1-vm01873 0/1 Completed 0 3d13h openshift-kube-scheduler installer-3-vm01873 0/1 Error 0 3d13h openshift-kube-scheduler installer-6-vm01873 0/1 Completed 0 3d13h openshift-kube-scheduler openshift-kube-scheduler-vm01873 3/3 Running 6 3d13h openshift-kube-storage-version-migrator-operator kube-storage-version-migrator-operator-54644d5786-hlss5 1/1 Running 4 (15h ago) 3d14h openshift-kube-storage-version-migrator migrator-56cff8468d-7s548 1/1 Running 3 3d13h openshift-lifecycle-agent lca-prep-precache-ht578 0/1 Completed 0 21h openshift-lifecycle-agent lca-prep-stateroot-setup-w7gwm 0/1 Completed 0 21h openshift-lifecycle-agent lifecycle-agent-controller-manager-54c9b5fd89-zsl7r 2/2 Running 7 (15h ago) 3d13h openshift-local-storage diskmaker-manager-2vxdp 2/2 Running 4 3d13h openshift-local-storage local-storage-operator-674f6dcf97-jdxw9 1/1 Running 2 3d13h openshift-logging cluster-logging-operator-57746666b6-87fmb 1/1 Running 3 (15h ago) 3d13h openshift-logging collector-mp5ww 1/1 Running 2 3d13h openshift-machine-config-operator kube-rbac-proxy-crio-vm01873 1/1 Running 7 3d14h openshift-machine-config-operator machine-config-controller-56fc59fdcf-9z4fl 2/2 Running 4 3d13h openshift-machine-config-operator machine-config-daemon-dfgdq 2/2 Running 4 3d13h openshift-machine-config-operator machine-config-operator-575bdfffc6-wbqmc 2/2 Running 4 3d14h openshift-machine-config-operator machine-config-server-rnfc6 1/1 Running 2 3d13h openshift-marketplace 1b2c2453438ee4107ff3cff145ed8069da558a8018f6626c4e79a26d50l6jk2 0/1 Completed 0 3d13h openshift-marketplace 1bdc271018164eb689792710b23bd6084a65e86807aa857636d23c1b52nsq5b 0/1 Completed 0 3d13h openshift-marketplace 4e692e78da9b950f392ff2575748192ff63f400191354113107002ee95hslwh 0/1 Completed 0 3d13h openshift-marketplace c98953b06c18521e7fb4292b7767ae0febdc6fbda19e4991737e35868f5mrkk 0/1 Completed 0 3d13h openshift-marketplace cc42e337969c795e1a26850fca1b25e1a4bc7b46f1f1d6df9ae5bc4ee8sbnkh 0/1 Completed 0 3d13h openshift-marketplace f45b93fb23220872b7e775ce6452784c7abe59f24d6a81693c18729c04kctp6 0/1 Completed 0 3d13h openshift-marketplace marketplace-operator-5c87d74f6c-n2cgv 1/1 Running 6 (15h ago) 3d14h openshift-marketplace rh-du-operators-79dtl 1/1 Running 2 3d13h openshift-monitoring cluster-monitoring-operator-6f68bbf775-mnqvb 1/1 Running 4 (15h ago) 3d14h openshift-monitoring kube-state-metrics-5d5f7556bc-ndw7k 3/3 Running 6 3d13h openshift-monitoring node-exporter-64b2n 2/2 Running 4 3d13h openshift-monitoring openshift-state-metrics-7b754dcf96-kfvsd 3/3 Running 6 3d13h openshift-monitoring prometheus-adapter-5c58d77db6-d6jmz 1/1 Running 2 2d14h openshift-monitoring prometheus-k8s-0 6/6 Running 12 3d13h openshift-monitoring prometheus-operator-64c4867485-gbnls 2/2 Running 4 3d13h openshift-monitoring prometheus-operator-admission-webhook-7cd56c97cd-wg5n5 1/1 Running 2 3d13h openshift-monitoring thanos-querier-c5b48f55f-hspfb 6/6 Running 13 3d13h openshift-multus multus-additional-cni-plugins-5tks6 1/1 Running 2 3d13h openshift-multus multus-admission-controller-b6d946975-285zw 2/2 Running 0 15h openshift-multus multus-jwxkk 1/1 Running 3 3d13h openshift-multus network-metrics-daemon-shz4w 2/2 Running 4 3d13h openshift-network-node-identity network-node-identity-fndbw 2/2 Running 6 3d13h openshift-network-operator network-operator-55cdbbdf9-f8ld6 1/1 Running 3 3d14h openshift-oauth-apiserver apiserver-d5c77fb74-fbjm9 1/1 Running 4 (15h ago) 3d13h openshift-operator-lifecycle-manager catalog-operator-5bc76757b-d4tvl 1/1 Running 2 3d14h openshift-operator-lifecycle-manager collect-profiles-28697055-bxt7x 0/1 Completed 0 33m openshift-operator-lifecycle-manager collect-profiles-28697070-nmddm 0/1 Completed 0 18m openshift-operator-lifecycle-manager collect-profiles-28697085-rhl6k 0/1 Completed 0 3m30s openshift-operator-lifecycle-manager olm-operator-5d6488c5f6-trgl5 1/1 Running 2 3d14h openshift-operator-lifecycle-manager package-server-manager-58d76848f9-nvrtp 1/1 Running 2 3d14h openshift-operator-lifecycle-manager packageserver-74c9bf7785-dzm6f 1/1 Running 6 (15h ago) 3d13h openshift-ovn-kubernetes ovnkube-control-plane-7bb497b9f-dzkm5 2/2 Running 4 3d13h openshift-ovn-kubernetes ovnkube-node-wzhm4 8/8 Running 17 (15h ago) 3d13h openshift-ptp linuxptp-daemon-slq25 2/2 Running 4 3d13h openshift-ptp ptp-operator-5ff4f97b98-xd54q 1/1 Running 2 3d13h openshift-route-controller-manager route-controller-manager-75c5b7f645-52xlz 1/1 Running 4 (15h ago) 2d14h openshift-service-ca-operator service-ca-operator-cfb65db99-9jxtz 1/1 Running 4 3d14h openshift-service-ca service-ca-5b587f8669-kb85t 1/1 Running 2 3d13h openshift-sriov-network-operator network-resources-injector-zkk6q 1/1 Running 2 3d13h openshift-sriov-network-operator operator-webhook-7lm49 1/1 Running 2 3d13h openshift-sriov-network-operator sriov-network-config-daemon-9bbch 1/1 Running 2 3d13h openshift-sriov-network-operator sriov-network-operator-7b5d8dd8bf-287j9 1/1 Running 2 3d13h
