-
Bug
-
Resolution: Duplicate
-
Major
-
None
-
None
-
None
-
None
-
Critical
-
sst_container_tools
-
3
-
False
-
-
None
-
None
-
None
-
None
-
None
crun won't reparent and reap zombie process because ptm2 buffer is full
Our customer requested pod deletion, but the pod is stuck in terminating. Upon inspection of the node logs, we can see that the container will never shut down, triggering known crio bug `level=warning msg="Stopping container (...) with stop signal timed out. Killing"` https://issues.redhat.com/browse/OCPBUGS-28981. OCPBUGS-28981 just makes the issue more apparent, but is more than likely not related to it. We then look at the crio container, and can see that it's not being shut down. We can see that the thread group leader (`rsyslogd` in this specific case) with PID 1 received a kill -9 and is trying to shut down as hard as it can, but it's stuck in `zap_pid_ns_processes`. The apparent reason is that we see a bash process in `ZO` (zombie) state which belongs to `crun`. However, `crun` never reaps the zombie process as it's stuck sleeping, waiting to be able to write to tty `ptm2`. It cannot do so, because the tty is full. The tty belongs to `crio`.
This is the same as upstream issue https://github.com/cri-o/cri-o/issues/6699, see https://github.com/cri-o/cri-o/issues/6699#issuecomment-1452796427
We also have a cri-o bug for this https://issues.redhat.com/browse/OCPBUGS-31317 as it's not sure which component should handle this