-
Sub-task
-
Resolution: Unresolved
-
Critical
-
None
-
None
-
None
-
None
-
False
-
-
False
Sonobuoy does not detect if the plugin pod exists and still report as 'running'[1].
It happened twice, randomly, with plugins (first time was kube-conformance).
We can see on the events that the pod was evicted[2].
[1] level3 plugin keeps running for sonobuoy, even if pod does not exists
$ sonobuoy status
PLUGIN STATUS RESULT COUNT PROGRESS
openshift-kube-conformance complete 1 345/345 (0 failures)
openshift-provider-cert-level1 complete 1 81/81 (0 failures)
openshift-provider-cert-level2 complete 1 17/17 (0 failures)
openshift-provider-cert-level3 running 1 0/0 (0 failures)
$ oc get pods -n sonobuoy
NAME READY STATUS RESTARTS AGE
sonobuoy 1/1 Running 0 41m
sonobuoy-openshift-kube-conformance-job-e869040ea8974b8c 0/3 Completed 0 41m
sonobuoy-openshift-provider-cert-level1-job-8602ea07f60b48e1 0/3 Completed 0 41m
sonobuoy-openshift-provider-cert-level2-job-60fd5c0169d242d3 0/3 Completed 0 41m
$ sonobuoy status --json |jq .
(...)
{
"plugin": "openshift-provider-cert-level3",
"node": "global",
"status": "running",
"result-status": "",
"result-counts": null,
"progress":
{ "name": "openshift-provider-cert-level3", "node": "global", "timestamp": "2022-04-13T01:52:17.520733562Z", "msg": "status=blocked-by=openshift-provider-cert-level2=(0/-17/0)=[0/100]", "total": 0, "completed": 0 }
}
],
"status": "running",
[2]events
$ oc get events -n sonobuoy |grep cert-level3 36m Normal Scheduled pod/sonobuoy-openshift-provider-cert-level3-job-a98f30dc10554a14 Successfully assigned sonobuoy/sonobuoy-openshift-provider-cert-level3-job-a98f30dc10554a14 to ip-10-0-138-63.ec2.internal 36m Normal AddedInterface pod/sonobuoy-openshift-provider-cert-level3-job-a98f30dc10554a14 Add eth0 [10.129.2.229/23] from openshift-sdn 36m Normal Pulling pod/sonobuoy-openshift-provider-cert-level3-job-a98f30dc10554a14 Pulling image "quay.io/mrbraga/openshift-tests-provider-cert:latest" 36m Normal Pulled pod/sonobuoy-openshift-provider-cert-level3-job-a98f30dc10554a14 Successfully pulled image "quay.io/mrbraga/openshift-tests-provider-cert:latest" in 451.346913ms 36m Normal Created pod/sonobuoy-openshift-provider-cert-level3-job-a98f30dc10554a14 Created container report-progress 36m Normal Started pod/sonobuoy-openshift-provider-cert-level3-job-a98f30dc10554a14 Started container report-progress 36m Normal Pulled pod/sonobuoy-openshift-provider-cert-level3-job-a98f30dc10554a14 Container image "quay.io/mrbraga/openshift-tests-provider-cert:latest" already present on machine 36m Normal Created pod/sonobuoy-openshift-provider-cert-level3-job-a98f30dc10554a14 Created container plugin 36m Normal Started pod/sonobuoy-openshift-provider-cert-level3-job-a98f30dc10554a14 Started container plugin 36m Normal Pulled pod/sonobuoy-openshift-provider-cert-level3-job-a98f30dc10554a14 Container image "sonobuoy/sonobuoy:v0.56.0" already present on machine 36m Normal Created pod/sonobuoy-openshift-provider-cert-level3-job-a98f30dc10554a14 Created container sonobuoy-worker 36m Normal Started pod/sonobuoy-openshift-provider-cert-level3-job-a98f30dc10554a14 Started container sonobuoy-worker 24m Normal TaintManagerEviction pod/sonobuoy-openshift-provider-cert-level3-job-a98f30dc10554a14 Marking for deletion Pod sonobuoy/sonobuoy-openshift-provider-cert-level3-job-a98f30dc10554a14 24m Normal TaintManagerEviction pod/sonobuoy-openshift-provider-cert-level3-job-a98f30dc10554a14 Cancelling deletion of Pod sonobuoy/sonobuoy-openshift-provider-cert-level3-job-a98f30dc10554a14 24m Normal Killing pod/sonobuoy-openshift-provider-cert-level3-job-a98f30dc10554a14 Stopping container report-progress 24m Normal Killing pod/sonobuoy-openshift-provider-cert-level3-job-a98f30dc10554a14 Stopping container sonobuoy-worker 24m Normal Killing pod/sonobuoy-openshift-provider-cert-level3-job-a98f30dc10554a14 Stopping container plugin
- links to