Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-7127

rabbitmq operator fails to reconcile due to missing binary rabbitmq-queues

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Undefined Undefined
    • None
    • rhos-18.0 Beta
    • rabbitmq-server
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • ?
    • ?
    • ?
    • ?
    • None
    • PIDONE Board
    • Important

      When trying to scale-up rabbitmq cluster from 1-replica to 3 replicas, the rabbitmq operator internally calls rabbitmq-queues in the running rabbitmq pods to rebablance master queues in the rabbitmq cluster.

       

      This is currently failing because the rabbitmq container image that we consume doesn't have rabbitmq-queues in the PATH. The rabbimq operator signals a failed reconciliation and blocks further reconciliations:

       

      {{{}{"level":"error","ts":"2024-05-16T17:25:51Z","msg":"failed to run queue rebalance on pod","controller":"rabbitmqcluster","controllerGroup":"rabbitmq.com","controllerKind":"RabbitmqCluster","RabbitmqCluster":

      {"name":"rabbitmq","namespace":"openstack"}

      {}}},"namespace":"openstack","name":"rabbitmq","reconcileID":"01c025b8-3f12-4e4b-ae41-6255515b9de5","pod":"rabbitmq-server-0","command":"rabbitmq-queues rebalance all","stdout":"","stderr":"sh: line 1: rabbitmq-queues: command not found\n","error":"command terminated with exit code 127","stacktrace":"github.com/rabbitmq/cluster-operator/controllers.(*RabbitmqClusterReconciler).runQueueRebalanceCommand\n\t/remote-source/app/controllers/reconcile_cli.go:112\ngithub.com/rabbitmq/cluster-operator/controllers.(*RabbitmqClusterReconciler).runRabbitmqCLICommandsIfAnnotated\n\t/remote-source/app/controllers/reconcile_cli.go:60\ngithub.com/rabbitmq/cluster-operator/controllers.(*RabbitmqClusterReconciler).Reconcile\n\t/remote-source/app/controllers/rabbitmqcluster_controller.go:230\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}{}

              rhn-engineering-dciabrin Damien Ciabrini
              rhn-engineering-dciabrin Damien Ciabrini
              rhos-dfg-pidone
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: