Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-30102

crun won't reap zombie process because ptm2 buffer is full

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Major Major
    • None
    • None
    • crun
    • None
    • None
    • Critical
    • rhel-sst-container-tools
    • 3
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None

      crun won't reparent and reap zombie process because ptm2 buffer is full

      Our customer requested pod deletion, but the pod is stuck in terminating. Upon inspection of the node logs, we can see that the container will never shut down, triggering known crio bug `level=warning msg="Stopping container (...) with stop signal timed out. Killing"` https://issues.redhat.com/browse/OCPBUGS-28981. OCPBUGS-28981 just makes the issue more apparent, but is more than likely not related to it. We then look at the crio container, and can see that it's not being shut down. We can see that the thread group leader (`rsyslogd` in this specific case) with PID 1 received a kill -9 and is trying to shut down as hard as it can, but it's stuck in `zap_pid_ns_processes`. The apparent reason is that we see a bash process in `ZO` (zombie) state which belongs to `crun`. However, `crun` never reaps the zombie process as it's stuck sleeping, waiting to be able to write to tty `ptm2`. It cannot do so, because the tty is full. The tty belongs to `crio`.

      This is the same as upstream issue https://github.com/cri-o/cri-o/issues/6699, see https://github.com/cri-o/cri-o/issues/6699#issuecomment-1452796427

      We also have a cri-o bug for this https://issues.redhat.com/browse/OCPBUGS-31317 as it's not sure which component should handle this

              kolyshkin Kirill Kolyshkin
              akaris@redhat.com Andreas Karis
              Container Runtime Eng Bot Container Runtime Eng Bot
              Container Runtime Bugs Bot Container Runtime Bugs Bot
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

                Created:
                Updated:
                Resolved: