Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-2520

Cleaning crio produces kubelet errors

    XMLWordPrintable

Details

    • False
    • Hide

      None

      Show
      None
    • If Release Note Needed, Set a Value
    • Set a Value
    • Customer Escalated

    Description

      Description of problem:

      If I have to make a crio cleaning process following this:

https://docs.openshift.com/container-platform/4.10/support/troubleshooting/troubleshooting-crio-issues.html

the kubelet will get some errors with some pods

       

       

      Version-Release number of selected component (if applicable):

      ocp 4.10

      How reproducible:

      Follow the procedure: https://docs.openshift.com/container-platform/4.10/support/troubleshooting/troubleshooting-crio-issues.html

      Steps to Reproduce:

      1.  cordon a worker
      $> oc adm cordon worker-0.el8k-ztp-1.hpecloud.org 
node/worker-0.el8k-ztp-1.hpecloud.org cordoned
      1. drain the worker

       

      > oc adm drain worker-0.el8k-ztp-1.hpecloud.org --ignore-daemonsets --delete-emptydir-data 
node/worker-0.el8k-ztp-1.hpecloud.org already cordoned
WARNING: ignoring DaemonSet-managed Pods: openshift-cluster-node-tuning-operator/tuned-jdlqq, openshift-dns/dns-default-47s42, openshift-dns/node-resolver-vm5v2, openshift-image-registry/node-ca-9fs4h, openshift-ingress-canary/ingress-canary-ck2jg, openshift-machine-config-operator/machine-config-daemon-9zdsw, openshift-monitoring/node-exporter-k7hrs, openshift-multus/multus-8pf46, openshift-multus/multus-additional-cni-plugins-pq7kb, openshift-multus/network-metrics-daemon-gk2sh, openshift-network-diagnostics/network-check-target-q4lhb, openshift-ovn-kubernetes/ovnkube-node-kq42w, openshift-ptp/linuxptp-daemon-87bqh, openshift-sriov-network-operator/sriov-network-config-daemon-s7m5s
evicting pod open-cluster-management-agent/klusterlet-work-agent-6fb8db56c6-ckfl4
evicting pod open-cluster-management-agent/klusterlet-registration-agent-84d6cb554b-85lc4
evicting pod openshift-marketplace/e87e0640ee2264cc35ef2c1ed97be0a46a4ef9100d11a48b638a6ae158tbtd5
evicting pod openshift-marketplace/d661dd28df06e41d7badaf307c5742fc9e10ffb0f717371a3be77ad8ackvpmp
evicting pod openshift-monitoring/prometheus-k8s-1
evicting pod openshift-operator-lifecycle-manager/collect-profiles-27756615-582pz
evicting pod open-cluster-management-agent/klusterlet-registration-agent-84d6cb554b-shsm6
evicting pod open-cluster-management-agent-addon/governance-policy-framework-656578c6d6-k9pcn
evicting pod open-cluster-management-agent/klusterlet-work-agent-6fb8db56c6-4kdj9
evicting pod openshift-monitoring/prometheus-adapter-8bc44b565-m2c6f
evicting pod openshift-ingress/router-default-5c47d949df-zjwk5
evicting pod openshift-marketplace/72ad2bad22e0747d66a0cc0ec31afee3f40f84e256d2c0c17d6369903ffq5kt
evicting pod openshift-marketplace/5ae0fc37abddd09b5423b3a6ed1245a2aab9106478b24a2d51ee7cd878v4vkf
evicting pod openshift-monitoring/thanos-querier-8574bbf668-hnn2t
evicting pod open-cluster-management-agent-addon/klusterlet-addon-workmgr-785f549-qlz9j
pod/5ae0fc37abddd09b5423b3a6ed1245a2aab9106478b24a2d51ee7cd878v4vkf evicted
pod/e87e0640ee2264cc35ef2c1ed97be0a46a4ef9100d11a48b638a6ae158tbtd5 evicted
pod/72ad2bad22e0747d66a0cc0ec31afee3f40f84e256d2c0c17d6369903ffq5kt evicted
pod/d661dd28df06e41d7badaf307c5742fc9e10ffb0f717371a3be77ad8ackvpmp evicted
pod/collect-profiles-27756615-582pz evicted
I1010 12:50:07.152392   33827 request.go:665] Waited for 1.000279283s due to client-side throttling, not priority and fairness, request: POST:https://api.el8k-ztp-1.hpecloud.org:6443/api/v1/namespaces/open-cluster-management-agent/pods/klusterlet-registration-agent-84d6cb554b-85lc4/eviction
pod/klusterlet-work-agent-6fb8db56c6-4kdj9 evicted
pod/klusterlet-work-agent-6fb8db56c6-ckfl4 evicted
pod/prometheus-adapter-8bc44b565-m2c6f evicted
pod/klusterlet-registration-agent-84d6cb554b-85lc4 evicted
pod/prometheus-k8s-1 evicted
pod/governance-policy-framework-656578c6d6-k9pcn evicted
pod/klusterlet-registration-agent-84d6cb554b-shsm6 evicted
pod/thanos-querier-8574bbf668-hnn2t evicted
pod/klusterlet-addon-workmgr-785f549-qlz9j evicted
pod/router-default-5c47d949df-zjwk5 evicted
node/worker-0.el8k-ztp-1.hpecloud.org evicted
      1.  stop kubelet, and clean crio
      [root@worker-0 ~]# systemctl stop kubelet
[root@worker-0 ~]# crictl rmp -fa
Stopped sandbox 73f1fe0dcf4e76bfe2d09a6ce0eb634412412ee85078699daae785bf2feb4ec3
Stopped sandbox 08cd6da07afdfac85dd5f286dd174e48013214ec3fe16536780d36f7bd20f478
Stopped sandbox 20f53bd011e84ea2beb9972c8ee018a7e3c9595459477386a437ed4a395fef1d
Stopped sandbox d618ecc7aec8a6e4faf4d9f7184d099e4b6a00a9b267f28a2a0049814673502b
Stopped sandbox fb26be3e86316841682cbc1d090653b03b327cb260a24840454957fb3a9e85ef
Stopped sandbox 9a10d52fda9fe07fd90949c9d2481e34220db4d78737028c72edfc8d47353d98
Stopped sandbox eb7cb280d3daceaf56083f5a3d27f34ebcecb332dda4f79ec6c705dca8e1ab38
Removed sandbox 20f53bd011e84ea2beb9972c8ee018a7e3c9595459477386a437ed4a395fef1d
Removed sandbox 73f1fe0dcf4e76bfe2d09a6ce0eb634412412ee85078699daae785bf2feb4ec3
Removed sandbox 9a10d52fda9fe07fd90949c9d2481e34220db4d78737028c72edfc8d47353d98
Removed sandbox eb7cb280d3daceaf56083f5a3d27f34ebcecb332dda4f79ec6c705dca8e1ab38
Removed sandbox d618ecc7aec8a6e4faf4d9f7184d099e4b6a00a9b267f28a2a0049814673502b
Removed sandbox 08cd6da07afdfac85dd5f286dd174e48013214ec3fe16536780d36f7bd20f478
Removed sandbox fb26be3e86316841682cbc1d090653b03b327cb260a24840454957fb3a9e85ef
Stopped sandbox be3a7142586a1db6dbad8c72f7ca15e9fd2e86f7409785dd8dbc4a5f2e602266
Stopped sandbox d4643eba7146e89d7c447125540ba35ec2246af5da27201693acf1d6ab563f75
Stopped sandbox a2488d7e78261e28b44df4909ea8708c560c205965a63b51ebf814d72be4239b
Removed sandbox d4643eba7146e89d7c447125540ba35ec2246af5da27201693acf1d6ab563f75
Removed sandbox a2488d7e78261e28b44df4909ea8708c560c205965a63b51ebf814d72be4239b
Removed sandbox be3a7142586a1db6dbad8c72f7ca15e9fd2e86f7409785dd8dbc4a5f2e602266
Stopped sandbox 113815c1e11e371826feb880ca978cd7fe126bdfd3a79731b43fcf6ec0d4b170
Removed sandbox 113815c1e11e371826feb880ca978cd7fe126bdfd3a79731b43fcf6ec0d4b170
Stopped sandbox aca2d072e72518befd030037b0ee90b632c80c4cc4de06e31a0a79bfc8b8e9d0
Removed sandbox aca2d072e72518befd030037b0ee90b632c80c4cc4de06e31a0a79bfc8b8e9d0
stopping the pod sandbox "ae23fddcad20f190975afffafb7a2b0a09d75307cd870a771128f82c556a567f" failed: rpc error: code = Unknown desc = failed to destroy network for pod sandbox k8s_network-check-target-q4lhb_openshift-network-diagnostics_9020fd5c-6f77-4d0f-ac05-ee9164da6e11_0(ae23fddcad20f190975afffafb7a2b0a09d75307cd870a771128f82c556a567f): error removing pod openshift-network-diagnostics_network-check-target-q4lhb from CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (delete): delegateDel: error invoking DelegateDel - "ovn-k8s-cni-overlay": error in getting result from DelNetwork: failed to send CNI request: Post "http://dummy/": dial unix /var/run/ovn-kubernetes/cni//ovn-cni-server.sock: connect: connection refused
stopping the pod sandbox "d3f6e044a8db57742318f7b08cadba542ceb5066a7ef9225382c4e33380d4b87" failed: rpc error: code = Unknown desc = failed to destroy network for pod sandbox k8s_dns-default-47s42_openshift-dns_476c52e7-cc08-48d5-84f9-09e02dc9cc84_0(d3f6e044a8db57742318f7b08cadba542ceb5066a7ef9225382c4e33380d4b87): error removing pod openshift-dns_dns-default-47s42 from CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (delete): delegateDel: error invoking DelegateDel - "ovn-k8s-cni-overlay": error in getting result from DelNetwork: failed to send CNI request: Post "http://dummy/": dial unix /var/run/ovn-kubernetes/cni//ovn-cni-server.sock: connect: connection refused
stopping the pod sandbox "71eba57be398d9e05bf116f8a4cc6da629aaa14ccc907c79ed599eda4a7868c8" failed: rpc error: code = Unknown desc = failed to destroy network for pod sandbox k8s_ingress-canary-ck2jg_openshift-ingress-canary_7a6524e1-7561-4cc7-89e5-76f259f32acb_0(71eba57be398d9e05bf116f8a4cc6da629aaa14ccc907c79ed599eda4a7868c8): error removing pod openshift-ingress-canary_ingress-canary-ck2jg from CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (delete): delegateDel: error invoking DelegateDel - "ovn-k8s-cni-overlay": error in getting result from DelNetwork: failed to send CNI request: Post "http://dummy/": dial unix /var/run/ovn-kubernetes/cni//ovn-cni-server.sock: connect: connection refused
stopping the pod sandbox "f344b4d695010ec8fdba56b3ff35fb23cf8bf73ee6454442a882616aea2b869d" failed: rpc error: code = Unknown desc = failed to destroy network for pod sandbox k8s_network-metrics-daemon-gk2sh_openshift-multus_9106be97-2586-4033-9ac8-e7507e6db948_0(f344b4d695010ec8fdba56b3ff35fb23cf8bf73ee6454442a882616aea2b869d): error removing pod openshift-multus_network-metrics-daemon-gk2sh from CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (delete): delegateDel: error invoking DelegateDel - "ovn-k8s-cni-overlay": error in getting result from DelNetwork: failed to send CNI request: Post "http://dummy/": dial unix /var/run/ovn-kubernetes/cni//ovn-cni-server.sock: connect: connection refused
      1. these are the pods not deleted:

       

      [root@worker-0 ~]# crictl pods
POD ID              CREATED             STATE               NAME                           NAMESPACE                       ATTEMPT             RUNTIME
d3f6e044a8db5       About an hour ago   Ready               dns-default-47s42              openshift-dns                   0                   (default)
71eba57be398d       About an hour ago   Ready               ingress-canary-ck2jg           openshift-ingress-canary        0                   (default)
f344b4d695010       About an hour ago   Ready               network-metrics-daemon-gk2sh   openshift-multus                0                   (default)
ae23fddcad20f       About an hour ago   Ready               network-check-target-q4lhb     openshift-network-diagnostics   0                   (default)

       

      1. wipe and start crio and kubelet

       

       

      root@worker-0 ~]# crio wipe -f
INFO[2022-10-10 11:00:56.070613828Z] Starting CRI-O, version: 1.23.2-8.rhaos4.10.git8ad5d25.el8, git: () 
INFO[2022-10-10 11:00:56.074752086Z] Internal wipe not set, meaning crio wipe will wipe. In the future, all wipes after reboot will happen when starting the crio server. 
INFO[2022-10-10 11:00:56.078155353Z] Wiping containers                            
INFO[2022-10-10 11:00:56.112267649Z] Deleted container ae23fddcad20f190975afffafb7a2b0a09d75307cd870a771128f82c556a567f 
INFO[2022-10-10 11:00:56.140222855Z] Deleted container f344b4d695010ec8fdba56b3ff35fb23cf8bf73ee6454442a882616aea2b869d 
INFO[2022-10-10 11:00:56.164197265Z] Deleted container 71eba57be398d9e05bf116f8a4cc6da629aaa14ccc907c79ed599eda4a7868c8 
INFO[2022-10-10 11:00:56.189218701Z] Deleted container d3f6e044a8db57742318f7b08cadba542ceb5066a7ef9225382c4e33380d4b87 
INFO[2022-10-10 11:00:56.209358281Z] Deleted container 1e87f96cd3a739dfe24ba9794128fc445b679d8ac22ae3968f1d44a736e262e4 
INFO[2022-10-10 11:00:56.233239891Z] Deleted container 3b8a01ca7146ef94d5df7c3e11705c45e4ae379cf82c65a1e2a3d3abbdf1c1da 
INFO[2022-10-10 11:00:56.253301668Z] Deleted container f7937d108128e6acb1929c537263b94640a6fae019df3f145e8026471197a25a 
INFO[2022-10-10 11:00:56.273304826Z] Deleted container bc1fdcd0d7957fad14d07e635b456abde5a6bc57a87551ff6b084386cbfe96f5 
INFO[2022-10-10 11:00:56.297301871Z] Deleted container fbb8f79a8f2d3dda48b7830a658d9ac56a47bc78fb0f2a43845245fe6c56ebeb 
INFO[2022-10-10 11:00:56.318363548Z] Deleted container 080b887d99ff3a674337a1788f8b028d220031bcf38c95f3f4414daae38f380c 
INFO[2022-10-10 11:00:56.318411991Z] Wiping images                                
INFO[2022-10-10 11:00:56.337293162Z] Deleted image 0cdc4dd092f48d93d3248a0e0d9004ba30efb3ababf7c6bd35d92c1c4e8269c9 
ERRO[2022-10-10 11:00:56.337881188Z] Unable to delete image 0cdc4dd092f48d93d3248a0e0d9004ba30efb3ababf7c6bd35d92c1c4e8269c9: identifier is not an image 
ERRO[2022-10-10 11:00:56.337948972Z] Unable to delete image 0cdc4dd092f48d93d3248a0e0d9004ba30efb3ababf7c6bd35d92c1c4e8269c9: identifier is not an image 
ERRO[2022-10-10 11:00:56.338005058Z] Unable to delete image 0cdc4dd092f48d93d3248a0e0d9004ba30efb3ababf7c6bd35d92c1c4e8269c9: identifier is not an image 
INFO[2022-10-10 11:00:56.347659751Z] Deleted image ca29654379ff0964142fd04fbf0bf4e6a6328195bfaf76968d4a8e2b1ce27577 
INFO[2022-10-10 11:00:56.357479790Z] Deleted image 55e294662f2c60c72271f4261f19b1be9c52515ffc76ba79f93bb44022fec181 
INFO[2022-10-10 11:00:56.368732292Z] Deleted image 174eb5411b86f6e54e111e45a6c864b2b3d3bb344aee395a477b045832975826 
INFO[2022-10-10 11:00:56.402073510Z] Deleted image 0c90509e9921497c98350bd8f46a886e8d9b662403f74dae9621567b9d16ce23 
ERRO[2022-10-10 11:00:56.402397379Z] Unable to delete image 55e294662f2c60c72271f4261f19b1be9c52515ffc76ba79f93bb44022fec181: identifier is not an image 
INFO[2022-10-10 11:00:56.411666732Z] Deleted image 58fbd478b7936c1c8a61b2626b29f4c871c4d61db97c5788fca56efedc09090a 
[root@worker-0 ~]# systemctl start crio 
[root@worker-0 ~]# systemctl start kubelet

       

      1. Errors from kubelet logs
      Oct 10 11:13:58 worker-0.el8k-ztp-1.hpecloud.org bash[134676]: I1010 11:13:58.838068  134676 kubelet_getters.go:176] "Pod status updated" pod="openshift-kni-infra/coredns-worker-0.el8k-ztp-1.hpecloud.org" status=Running
Oct 10 11:13:58 worker-0.el8k-ztp-1.hpecloud.org bash[134676]: I1010 11:13:58.838125  134676 kubelet_getters.go:176] "Pod status updated" pod="openshift-kni-infra/keepalived-worker-0.el8k-ztp-1.hpecloud.org" status=Running
Oct 10 11:13:59 worker-0.el8k-ztp-1.hpecloud.org bash[134676]: E1010 11:13:59.037734  134676 manager.go:1123] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a2c094acc516ba5cec01cf8460bb05.slice/crio-113815c1e11e371826feb880ca978cd7fe126bdfd3a79731b43fcf6ec0d4b170.scope: Error finding container 113815c1e11e371826feb880ca978cd7fe126bdfd3a79731b43fcf6ec0d4b170: Status 404 returned error &{%!s(*http.body=&{0xc001b90df8 <nil> <nil> false false {0 0} false false false <nil>}) {%!s(int32=0) %!s(uint32=0)} %!s(bool=false) <nil> %!s(func(error) error=0x859fa0) %!s(func() error=0x85a0a0)}
Oct 10 11:13:59 worker-0.el8k-ztp-1.hpecloud.org bash[134676]: E1010 11:13:59.038557  134676 manager.go:1123] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b86fc0c_29fb_4462_83d2_ad7e8abecc4d.slice/crio-08cd6da07afdfac85dd5f286dd174e48013214ec3fe16536780d36f7bd20f478.scope: Error finding container 08cd6da07afdfac85dd5f286dd174e48013214ec3fe16536780d36f7bd20f478: Status 404 returned error &{%!s(*http.body=&{0xc000fe53c8 <nil> <nil> false false {0 0} false false false <nil>}) {%!s(int32=0) %!s(uint32=0)} %!s(bool=false) <nil> %!s(func(error) error=0x859fa0) %!s(func() error=0x85a0a0)}
Oct 10 11:13:59 worker-0.el8k-ztp-1.hpecloud.org bash[134676]: E1010 11:13:59.039362  134676 manager.go:1123] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9020fd5c_6f77_4d0f_ac05_ee9164da6e11.slice/crio-bc1fdcd0d7957fad14d07e635b456abde5a6bc57a87551ff6b084386cbfe96f5.scope: Error finding container bc1fdcd0d7957fad14d07e635b456abde5a6bc57a87551ff6b084386cbfe96f5: Status 404 returned error &{%!s(*http.body=&{0xc002e020c0 <nil> <nil> false false {0 0} false false false <nil>}) {%!s(int32=0) %!s(uint32=0)} %!s(bool=false) <nil> %!s(func(error) error=0x859fa0) %!s(func() error=0x85a0a0)}
Oct 10 11:13:59 worker-0.el8k-ztp-1.hpecloud.org bash[134676]: E1010 11:13:59.040085  134676 manager.go:1123] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8964bd68_8345_475d_8be3_f0e705419a65.slice/crio-d618ecc7aec8a6e4faf4d9f7184d099e4b6a00a9b267f28a2a0049814673502b.scope: Error finding container d618ecc7aec8a6e4faf4d9f7184d099e4b6a00a9b267f28a2a0049814673502b: Status 404 returned error &{%!s(*http.body=&{0xc002e02108 <nil> <nil> false false {0 0} false false false <nil>}) {%!s(int32=0) %!s(uint32=0)} %!s(bool=false) <nil> %!s(func(error) error=0x859fa0) %!s(func() error=0x85a0a0)}
Oct 10 11:13:59 worker-0.el8k-ztp-1.hpecloud.org bash[134676]: E1010 11:13:59.040761  134676 manager.go:1123] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9020fd5c_6f77_4d0f_ac05_ee9164da6e11.slice/crio-ae23fddcad20f190975afffafb7a2b0a09d75307cd870a771128f82c556a567f.scope: Error finding container ae23fddcad20f190975afffafb7a2b0a09d75307cd870a771128f82c556a567f: Status 404 returned error &{%!s(*http.body=&{0xc001b90e88 <nil> <nil> false false {0 0} false false false <nil>}) {%!s(int32=0) %!s(uint32=0)} %!s(bool=false) <nil> %!s(func(error) error=0x859fa0) %!s(func() error=0x85a0a0)}
Oct 10 11:13:59 worker-0.el8k-ztp-1.hpecloud.org bash[134676]: E1010 11:13:59.041568  134676 manager.go:1123] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd09327_6df6_4f08_a264_a33292850671.slice/crio-be3a7142586a1db6dbad8c72f7ca15e9fd2e86f7409785dd8dbc4a5f2e602266.scope: Error finding container be3a7142586a1db6dbad8c72f7ca15e9fd2e86f7409785dd8dbc4a5f2e602266: Status 404 returned error &{%!s(*http.body=&{0xc001b90f00 <nil> <nil> false false {0 0} false false false <nil>}) {%!s(int32=0) %!s(uint32=0)} %!s(bool=false) <nil> %!s(func(error) error=0x859fa0) %!s(func() error=0x85a0a0)}
Oct 10 11:13:59 worker-0.el8k-ztp-1.hpecloud.org bash[134676]: E1010 11:13:59.042211  134676 manager.go:1123] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dcf8a95_8afc_4ace_a783_6cf08fb7fbb2.slice/crio-9a10d52fda9fe07fd90949c9d2481e34220db4d78737028c72edfc8d47353d98.scope: Error finding container 9a10d52fda9fe07fd90949c9d2481e34220db4d78737028c72edfc8d47353d98: Status 404 returned error &{%!s(*http.body=&{0xc000fe5470 <nil> <nil> false false {0 0} false false false <nil>}) {%!s(int32=0) %!s(uint32=0)} %!s(bool=false) <nil> %!s(func(error) error=0x859fa0) %!s(func() error=0x85a0a0)}
Oct 10 11:13:59 worker-0.el8k-ztp-1.hpecloud.org bash[134676]: E1010 11:13:59.042702  134676 manager.go:1123] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6247d83e_50b1_4722_a7da_6fe3373bce95.slice/crio-eb7cb280d3daceaf56083f5a3d27f34ebcecb332dda4f79ec6c705dca8e1ab38.scope: Error finding container eb7cb280d3daceaf56083f5a3d27f34ebcecb332dda4f79ec6c705dca8e1ab38: Status 404 returned error &{%!s(*http.body=&{0xc002e02198 <nil> <nil> false false {0 0} false false false <nil>}) {%!s(int32=0) %!s(uint32=0)} %!s(bool=false) <nil> %!s(func(error) error=0x859fa0) %!s(func() error=0x85a0a0)}
Oct 10 11:13:59 worker-0.el8k-ztp-1.hpecloud.org bash[134676]: E1010 11:13:59.043103  134676 manager.go:1123] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9106be97_2586_4033_9ac8_e7507e6db948.slice/crio-f7937d108128e6acb1929c537263b94640a6fae019df3f145e8026471197a25a.scope: Error finding container f7937d108128e6acb1929c537263b94640a6fae019df3f145e8026471197a25a: Status 404 returned error &{%!s(*http.body=&{0xc000fe54b8 <nil> <nil> false false {0 0} false false false <nil>}) {%!s(int32=0) %!s(uint32=0)} %!s(bool=false) <nil> %!s(func(error) error=0x859fa0) %!s(func() error=0x85a0a0)}
Oct 10 11:13:59 worker-0.el8k-ztp-1.hpecloud.org bash[134676]: E1010 11:13:59.043800  134676 manager.go:1123] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode632c179_ef5c_4021_a66a_816dfd58bda5.slice/crio-20f53bd011e84ea2beb9972c8ee018a7e3c9595459477386a437ed4a395fef1d.scope: Error finding container 20f53bd011e84ea2beb9972c8ee018a7e3c9595459477386a437ed4a395fef1d: Status 404 returned error &{%!s(*http.body=&{0xc000fe5500 <nil> <nil> false false {0 0} false false false <nil>}) {%!s(int32=0) %!s(uint32=0)} %!s(bool=false) <nil> %!s(func(error) error=0x859fa0) %!s(func() error=0x85a0a0)}
Oct 10 11:13:59 worker-0.el8k-ztp-1.hpecloud.org bash[134676]: E1010 11:13:59.044415  134676 manager.go:1123] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c980f6_d920_4d2b_b218_7bc02d2004dc.slice/crio-a2488d7e78261e28b44df4909ea8708c560c205965a63b51ebf814d72be4239b.scope: Error finding container a2488d7e78261e28b44df4909ea8708c560c205965a63b51ebf814d72be4239b: Status 404 returned error &{%!s(*http.body=&{0xc000fe5548 <nil> <nil> false false {0 0} false false false <nil>}) {%!s(int32=0) %!s(uint32=0)} %!s(bool=false) <nil> %!s(func(error) error=0x859fa0) %!s(func() error=0x85a0a0)}
Oct 10 11:13:59 worker-0.el8k-ztp-1.hpecloud.org bash[134676]: E1010 11:13:59.044759  134676 manager.go:1123] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a432ba0abc09387dbb8cd7c2122738.slice/crio-aca2d072e72518befd030037b0ee90b632c80c4cc4de06e31a0a79bfc8b8e9d0.scope: Error finding container aca2d072e72518befd030037b0ee90b632c80c4cc4de06e31a0a79bfc8b8e9d0: Status 404 returned error &{%!s(*http.body=&{0xc001b90ff0 <nil> <nil> false false {0 0} false false false <nil>}) {%!s(int32=0) %!s(uint32=0)} %!s(bool=false) <nil> %!s(func(error) error=0x859fa0) %!s(func() error=0x85a0a0)}
Oct 10 11:13:59 worker-0.el8k-ztp-1.hpecloud.org bash[134676]: E1010 11:13:59.045132  134676 manager.go:1123] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476c52e7_cc08_48d5_84f9_09e02dc9cc84.slice/crio-d3f6e044a8db57742318f7b08cadba542ceb5066a7ef9225382c4e33380d4b87.scope: Error finding container d3f6e044a8db57742318f7b08cadba542ceb5066a7ef9225382c4e33380d4b87: Status 404 returned error &{%!s(*http.body=&{0xc000fe55c0 <nil> <nil> false false {0 0} false false false <nil>}) {%!s(int32=0) %!s(uint32=0)} %!s(bool=false) <nil> %!s(func(error) error=0x859fa0) %!s(func() error=0x85a0a0)}
Oct 10 11:13:59 worker-0.el8k-ztp-1.hpecloud.org bash[134676]: E1010 11:13:59.045743  134676 manager.go:1123] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24727384_7c62_477e_9129_2ebceccbe627.slice/crio-d4643eba7146e89d7c447125540ba35ec2246af5da27201693acf1d6ab563f75.scope: Error finding container d4643eba7146e89d7c447125540ba35ec2246af5da27201693acf1d6ab563f75: Status 404 returned error &{%!s(*http.body=&{0xc000fe5608 <nil> <nil> false false {0 0} false false false <nil>}) {%!s(int32=0) %!s(uint32=0)} %!s(bool=false) <nil> %!s(func(error) error=0x859fa0) %!s(func() error=0x85a0a0)}
Oct 10 11:13:59 worker-0.el8k-ztp-1.hpecloud.org bash[134676]: E1010 11:13:59.046279  134676 manager.go:1123] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf395187c_3da4_4604_8b16_7029aac70988.slice/crio-fb26be3e86316841682cbc1d090653b03b327cb260a24840454957fb3a9e85ef.scope: Error finding container fb26be3e86316841682cbc1d090653b03b327cb260a24840454957fb3a9e85ef: Status 404 returned error &{%!s(*http.body=&{0xc0028f0db0 <nil> <nil> false false {0 0} false false false <nil>}) {%!s(int32=0) %!s(uint32=0)} %!s(bool=false) <nil> %!s(func(error) error=0x859fa0) %!s(func() error=0x85a0a0)}
Oct 10 11:13:59 worker-0.el8k-ztp-1.hpecloud.org bash[134676]: E1010 11:13:59.051894  134676 manager.go:1123] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a6524e1_7561_4cc7_89e5_76f259f32acb.slice/crio-080b887d99ff3a674337a1788f8b028d220031bcf38c95f3f4414daae38f380c.scope: Error finding container 080b887d99ff3a674337a1788f8b028d220031bcf38c95f3f4414daae38f380c: Status 404 returned error &{%!s(*http.body=&{0xc00277a018 <nil> <nil> false false {0 0} false false false <nil>}) {%!s(int32=0) %!s(uint32=0)} %!s(bool=false) <nil> %!s(func(error) error=0x859fa0) %!s(func() error=0x85a0a0)}
Oct 10 11:13:59 worker-0.el8k-ztp-1.hpecloud.org bash[134676]: E1010 11:13:59.052642  134676 manager.go:1123] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea32469_5962_4b46_a31c_ee8027da691e.slice/crio-73f1fe0dcf4e76bfe2d09a6ce0eb634412412ee85078699daae785bf2feb4ec3.scope: Error finding container 73f1fe0dcf4e76bfe2d09a6ce0eb634412412ee85078699daae785bf2feb4ec3: Status 404 returned error &{%!s(*http.body=&{0xc000fe4018 <nil> <nil> false false {0 0} false false false <nil>}) {%!s(int32=0) %!s(uint32=0)} %!s(bool=false) <nil> %!s(func(error) error=0x859fa0) %!s(func() error=0x85a0a0)}
Oct 10 11:13:59 worker-0.el8k-ztp-1.hpecloud.org bash[134676]: E1010 11:13:59.053435  134676 manager.go:1123] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476c52e7_cc08_48d5_84f9_09e02dc9cc84.slice/crio-1e87f96cd3a739dfe24ba9794128fc445b679d8ac22ae3968f1d44a736e262e4.scope: Error finding container 1e87f96cd3a739dfe24ba9794128fc445b679d8ac22ae3968f1d44a736e262e4: Status 404 returned error &{%!s(*http.body=&{0xc0024ce078 <nil> <nil> false false {0 0} false false false <nil>}) {%!s(int32=0) %!s(uint32=0)} %!s(bool=false) <nil> %!s(func(error) error=0x859fa0) %!s(func() error=0x85a0a0)}
Oct 10 11:13:59 worker-0.el8k-ztp-1.hpecloud.org bash[134676]: E1010 11:13:59.054086  134676 manager.go:1123] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a6524e1_7561_4cc7_89e5_76f259f32acb.slice/crio-71eba57be398d9e05bf116f8a4cc6da629aaa14ccc907c79ed599eda4a7868c8.scope: Error finding container 71eba57be398d9e05bf116f8a4cc6da629aaa14ccc907c79ed599eda4a7868c8: Status 404 returned error &{%!s(*http.body=&{0xc001ae4060 <nil> <nil> false false {0 0} false false false <nil>}) {%!s(int32=0) %!s(uint32=0)} %!s(bool=false) <nil> %!s(func(error) error=0x859fa0) %!s(func() error=0x85a0a0)}
Oct 10 11:13:59 worker-0.el8k-ztp-1.hpecloud.org bash[134676]: E1010 11:13:59.054602  134676 manager.go:1123] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9106be97_2586_4033_9ac8_e7507e6db948.slice/crio-f344b4d695010ec8fdba56b3ff35fb23cf8bf73ee6454442a882616aea2b869d.scope: Error finding container f344b4d695010ec8fdba56b3ff35fb23cf8bf73e

       

      If I am not wrong (I have tested some of them), these errors came from the pods that were never drained, nor deleted (crictl rmp -fa):

      openshift-cluster-node-tuning-operator             tuned-jdlqq                                                       1/1     Running     1             21h     10.19.10.110   worker-0.el8k-ztp-1.hpecloud.org   <none>           <none>
openshift-dns                                      dns-default-47s42                                                 2/2     Running     2             21h     10.131.0.6     worker-0.el8k-ztp-1.hpecloud.org   <none>           <none>
openshift-dns                                      node-resolver-vm5v2                                               1/1     Running     1             21h     10.19.10.110   worker-0.el8k-ztp-1.hpecloud.org   <none>           <none>
openshift-image-registry                           node-ca-9fs4h                                                     1/1     Running     1             21h     10.19.10.110   worker-0.el8k-ztp-1.hpecloud.org   <none>           <none>
openshift-ingress-canary                           ingress-canary-ck2jg                                              1/1     Running     1             21h     10.131.0.7     worker-0.el8k-ztp-1.hpecloud.org   <none>           <none>
openshift-kni-infra                                coredns-worker-0.el8k-ztp-1.hpecloud.org                          2/2     Running     2             21h     10.19.10.110   worker-0.el8k-ztp-1.hpecloud.org   <none>           <none>
openshift-kni-infra                                keepalived-worker-0.el8k-ztp-1.hpecloud.org                       2/2     Running     2             21h     10.19.10.110   worker-0.el8k-ztp-1.hpecloud.org   <none>           <none>
openshift-machine-config-operator                  machine-config-daemon-9zdsw                                       2/2     Running     2             21h     10.19.10.110   worker-0.el8k-ztp-1.hpecloud.org   <none>           <none>
openshift-monitoring                               node-exporter-k7hrs                                               2/2     Running     2             21h     10.19.10.110   worker-0.el8k-ztp-1.hpecloud.org   <none>           <none>
openshift-multus                                   multus-8pf46                                                      1/1     Running     1             21h     10.19.10.110   worker-0.el8k-ztp-1.hpecloud.org   <none>           <none>
openshift-multus                                   multus-additional-cni-plugins-pq7kb                               1/1     Running     1             21h     10.19.10.110   worker-0.el8k-ztp-1.hpecloud.org   <none>           <none>
openshift-multus                                   network-metrics-daemon-gk2sh                                      2/2     Running     2             21h     10.131.0.3     worker-0.el8k-ztp-1.hpecloud.org   <none>           <none>
openshift-network-diagnostics                      network-check-target-q4lhb                                        1/1     Running     1             21h     10.131.0.4     worker-0.el8k-ztp-1.hpecloud.org   <none>           <none>
openshift-ovn-kubernetes                           ovnkube-node-kq42w                                                5/5     Running     5             21h     10.19.10.110   worker-0.el8k-ztp-1.hpecloud.org   <none>           <none>
openshift-ptp                                      linuxptp-daemon-87bqh                                             2/2     Running     2             20h     10.19.10.110   worker-0.el8k-ztp-1.hpecloud.org   <none>           <none>
openshift-sriov-network-operator                   sriov-network-config-daemon-s7m5s                                 3/3     Running     3             20h     10.19.10.110   worker-0.el8k-ztp-1.hpecloud.org   <none>           <none>

       

       

      Actual results:

       

      Expected results:

       

      Additional info:

       

      Attachments

        Issue Links

          clones

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-2157 Documentation for cleaning crio produces kubelet errors

          • Normal - To be worked after urgent and high priorities are resolved. This term is chosen when the severity of an issue is relatively close to the level of effort to fix it. The existence of an easily implemented workaround can also lead to this priority level instead of a higher priority.
          • Closed

          Activity

            People

              pehunt@redhat.com Peter Hunt
              jgato@redhat.com Jose Gato Luis
              Sunil Choudhary Sunil Choudhary
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: