Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-8902

AMQ 7 on OpenShift leaks threads when acceptor is shut down

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Undefined Undefined
    • None
    • AMQ 7.11.6.GA
    • None

      If I do an installation of AMQ 7 using the OpenShift operator, exposing one AMQP acceptor, shutting down the acceptor (e.g., using the web console) causes the broker eventually to fail through lack of resources. It isn't clear from the broker log what the exhausted resource is, but routine monitoring shows it: there is a constant build-up of zombie threads.

      1000650+   27135       1  0 10:45 ?        00:00:00 [sleep] <defunct>
      ...  

       
      This starts as soon as the acceptor is shut down, and continues until it is either started again, or the pod fails. Presumably, the thread count eventually exceeds nproc.

      I believe that the zombie threads are associated with the default readiness probe, `readinessProbe.sh`. If I change the readiness probe in the broker CR to `exec /usr/bin/true`, I no longer see the zombie threads. However, I can't reproduce the thread leak just by running `readinessProbe.sh` at the prompt. So the connection between the readiness probe and these threads is unclear to me.

      If a broker's acceptor is shut down, the broker should stay up, in the 'not ready' state – at least according to OpenShift documentation. A failed liveness probe should result in the broker shutting down. So the default readiness behaviour of AMQ seems to be correct – apart from the thread leak.

       

            gtully@redhat.com Gary Tully
            rhn-support-kboone Kevin Boone
            Mikhail Krutov Mikhail Krutov
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: