Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-10227

RabbitMQ operator fails to scale down replicas

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • None
    • rhos-18.0.1
    • None
    • Moderate

      We have a 250 node RHOSO cluster running GA on 3 node masters/worker OCP cluster.
      Recently we have scaled default replicas of rabbitmq-cell1-server from 1 to 3, but scaling down replicas from openstack controlplane CR does not have any effect. We have forcefully changed replicas back to 1 in the statefulset which reduced replicas to 1, bug again containers in the remaining replica pod does not come up.

      error logs from the pod:
      2024-09-18 04:51:45.598431+00:00 [info] <0.245.0> Waiting for Mnesia tables for 30000 ms, 0 retries left
      2024-09-18 04:52:15.599243+00:00 [error] <0.245.0> Feature flag `quorum_queue`: migration function crashed: {error,
      2024-09-18 04:52:15.599243+00:00 [error] <0.245.0>                                                           {timeout_waiting_for_tables,                                                                                                           
      2024-09-18 04:52:15.599243+00:00 [error] <0.245.0>                                                            ['rabbit@rabbitmq-cell1-server-1.rabbitmq-cell1-nodes.openstack',                                                                     
      2024-09-18 04:52:15.599243+00:00 [error] <0.245.0>                                                             'rabbit@rabbitmq-cell1-server-2.rabbitmq-cell1-nodes.openstack',                                                                     
      2024-09-18 04:52:15.599243+00:00 [error] <0.245.0>                                                             'rabbit@rabbitmq-cell1-server-0.rabbitmq-cell1-nodes.openstack'],                                                                    
      2024-09-18 04:52:15.599243+00:00 [error] <0.245.0>                                                            [rabbit_durable_queue]}}                                                                                                              
      2024-09-18 04:52:15.599243+00:00 [error] <0.245.0> [{rabbit_table,wait,3,[

      {file,"rabbit_table.erl"}

      ,{line,121}]},
      2024-09-18 04:52:15.599243+00:00 [error] <0.245.0>  

      {rabbit_core_ff,quorum_queue_migration,3, 2024-09-18 04:52:15.599243+00:00 [error] <0.245.0>                  [\{file,"rabbit_core_ff.erl"}

      ,{line,77}]},
      2024-09-18 04:52:15.599243+00:00 [error] <0.245.0>  

      {rabbit_feature_flags,run_migration_fun,3, 2024-09-18 04:52:15.599243+00:00 [error] <0.245.0>                        [\{file,"rabbit_feature_flags.erl"}

      ,{line,1602}]},
      2024-09-18 04:52:15.599243+00:00 [error] <0.245.0>  

      {rabbit_feature_flags,'-verify_which_feature_flags_are_actually_enabled/0-fun-0-',                                                                                                               2024-09-18 04:52:15.599243+00:00 [error] <0.245.0>                        3, 2024-09-18 04:52:15.599243+00:00 [error] <0.245.0>                        [\{file,"rabbit_feature_flags.erl"}

      ,{line,2269}]},
      2024-09-18 04:52:15.599243+00:00 [error] <0.245.0>  {maps,fold_1,3,[

      {file,"maps.erl"}

      ,{line,410}]},
      2024-09-18 04:52:15.599243+00:00 [error] <0.245.0>  

      {rabbit_feature_flags,verify_which_feature_flags_are_actually_enabled,0,                                                                                                                         2024-09-18 04:52:15.599243+00:00 [error] <0.245.0>                        [\{file,"rabbit_feature_flags.erl"}

      ,{line,2267}]},
      2024-09-18 04:52:15.599243+00:00 [error] <0.245.0>  

      {rabbit_feature_flags,sync_feature_flags_with_cluster,3, 2024-09-18 04:52:15.599243+00:00 [error] <0.245.0>                        [\{file,"rabbit_feature_flags.erl"}

      ,{line,2082}]},
      2024-09-18 04:52:15.599243+00:00 [error] <0.245.0>  

      {rabbit_mnesia,ensure_feature_flags_are_in_sync,2, 2024-09-18 04:52:15.599243+00:00 [error] <0.245.0>                 [\{file,"rabbit_mnesia.erl"}

      ,{line,644}]}]
      2024-09-18 04:52:15.599560+00:00 [info] <0.245.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
      2024-09-18 04:52:45.600333+00:00 [warning] <0.245.0> Error while waiting for Mnesia tables:

      {timeout_waiting_for_tables, 2024-09-18 04:52:45.600333+00:00 [warning] <0.245.0>                                         ['rabbit@rabbitmq-cell1-server-1.rabbitmq-cell1-nodes.openstack',                                                                                       2024-09-18 04:52:45.600333+00:00 [warning] <0.245.0>                                          'rabbit@rabbitmq-cell1-server-2.rabbitmq-cell1-nodes.openstack',                                                                                       2024-09-18 04:52:45.600333+00:00 [warning] <0.245.0>                                          'rabbit@rabbitmq-cell1-server-0.rabbitmq-cell1-nodes.openstack'],                                                                                      2024-09-18 04:52:45.600333+00:00 [warning] <0.245.0>                                         [rabbit_user,rabbit_user_permission,                                                                                                                    2024-09-18 04:52:45.600333+00:00 [warning] <0.245.0>                                          rabbit_topic_permission,rabbit_vhost,                                                                                                                  2024-09-18 04:52:45.600333+00:00 [warning] <0.245.0>                                          rabbit_durable_route, 2024-09-18 04:52:45.600333+00:00 [warning] <0.245.0>                                          rabbit_durable_exchange, 2024-09-18 04:52:45.600333+00:00 [warning] <0.245.0>                                          rabbit_runtime_parameters, 2024-09-18 04:52:45.600333+00:00 [warning] <0.245.0>                                          rabbit_durable_queue]}

       

            Unassigned Unassigned
            rpulapak@redhat.com Rajesh Pulapakula
            rhos-dfg-pidone
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: