Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-14679

"ConnectionForced: Too many heartbeats missed" is logged periodically by Nova API

XMLWordPrintable

    • 0
    • False
    • Hide

      None

      Show
      None
    • False
    • ?
    • None
    • Moderate

      To Reproduce Steps to reproduce the behavior:
      nova-api processes on all controller nodes periodically log the following messages for connections to some rabbitmq servers (including local one):

      2025-03-11 12:32:09.697 20 ERROR oslo.messaging._drivers.impl_rabbit [-] [4991c2cc-83dd-414d-bc57-04da7961185b] AMQP server on controller-1.internalapi.example.com:5672 is unreachable: Too many heartbeats missed. Trying again in 1 seconds.: amqp.exceptions.ConnectionForced: Too many heartbeats missed
      

      On rabbitmq side I can see the following:

      2025-03-11 12:32:09.697 [warning] <0.26325.194> closing AMQP connection <0.26325.194> (192.168.1.35:48636 -> 192.168.1.33:5672 - mod_wsgi:20:4991c2cc-83dd-414d-bc57-04da7961185b, vhost: '/', user: 'guest'):
      client unexpectedly closed TCP connection
      

      As a result, it looks like python-amqp reports exception because of missed heartbeats and then restarts connection while server is not aware about any problems.

      This doesn't look like network problem because other services are not affected.

      Expected behavior
      Too many heartbeats missed error is logged when something is wrong with network.

      Bug impact
      Looks cosmetic, but looking for a second look

      Known workaround
      None

              Unassigned Unassigned
              rhn-support-astupnik Alex Stupnikov
              rhos-workloads-compute
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: