Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-20331

RHOSO 18.0 RabbitMQ uses TLS 1.3 and key update events cause RabbitMQ network partition

XMLWordPrintable

    • 3
    • False
    • Hide

      None

      Show
      None
    • False
    • ?
    • None
    • Sprint 6, Sprint 7
    • 2
    • Critical

      To Reproduce Steps to reproduce the behavior:

      1. Deploy RHOSO 18.0
      2. RabbitMQ uses TLS 1.3, not TLS 1.2
        # oc get RabbitMQCluster rabbitmq-cell1  -o yaml |grep tlsv1.3
                {versions, ['tlsv1.2','tlsv1.3']}
                {versions, ['tlsv1.2','tlsv1.3']}
              {versions, ['tlsv1.2','tlsv1.3']}
      1. Wait for a while
      2. RabbitMQ network partition issue occurs after TLS key update events
        [error] <x.xxxxx.x>     event: TLS sender received unexpected event
        [error] <x.xxxxx.x>     reason: [{type,internal},{message,{key_update,{<x.xxxxx.x>,undefined}}}]
          :
        [error] <x.xxxxx.x> Partial partition detected:
        [error] <x.xxxxx.x>  * We saw DOWN from rabbit@rabbitmq-cell1-server-0.rabbitmq-cell1-nodes.openstack
        [error] <x.xxxxx.x>  * We can still see rabbit@rabbitmq-cell1-server-2.rabbitmq-cell1-nodes.openstack which can see rabbit@rabbitmq-cell1-server-0.rabbitmq-cell1-nodes.openstack 

      Expected behavior

      • RabbitMQ is stable
      • RabbitMQ uses TLS 1.2, not TLS 1.3

      Bug impact

      • RabbitMQ cluster experiences network partition randomly and it makes RHOSO control plane unusable.
      • The RabbitMQ is not recovered automatically until we delete/recreate RabbitMQ pods manually.
        Even if we recover it, the issue recurs a few days later.

      Known workaround

      • delete/recreate RabbitMQ pods manually after the issue occurrence

      Additional context

              rhn-engineering-dciabrin Damien Ciabrini
              rhn-support-yatanaka Yamato Tanaka
              rhos-dfg-pidone
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Created:
                Updated: