Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-12137

[upstream-only] When updating to latest, rabbitmq fails with "failed to run queue rebalance on pod"

XMLWordPrintable

    • 5
    • False
    • Hide

      None

      Show
      None
    • False
    • ?
    • None
    • PIDONE 18.0.4, PIDONE 18.0.5
    • 2
    • Important

      When I did a minor update test, rabbitmq failed to get ready, because the operator triggered a `rabbitmq-queues`, but the command was not found. 
       
      There seems to be an issue with the upstream rabbitmq image. its missing the rabbitmq-queues command. 
       
      When updating to latest, rabbitmq fails with "failed to run queue rebalance on pod":

      {"level":"error","ts":"2024-12-03T07:54:17Z","msg":"failed to run queue rebalance on pod","controller":"rabbitmqcluster","controllerGroup":"rabbitmq.com","controllerKind":"RabbitmqCluster","RabbitmqCluster":{"name":"rabbitmq","namespace":"openstack"},"namespace":"openstack","name":"rabbitmq","reconcileID":"54a92b40-282e-4c4d-a910-9c6c05ac8556","pod":"rabbitmq-server-0","command":"rabbitmq-queues rebalance all","stdout":"","stderr":"sh: line 1: rabbitmq-queues: command not found\n","error":"command terminated with exit code 127","stacktrace":"github.com/rabbitmq/cluster-operator/v2/controllers.(*RabbitmqClusterReconciler).runQueueRebalanceCommand\n\t/workspace/controllers/reconcile_cli.go:112\ngithub.com/rabbitmq/cluster-operator/v2/controllers.(*RabbitmqClusterReconciler).runRabbitmqCLICommandsIfAnnotated\n\t/workspace/controllers/reconcile_cli.go:60\ngithub.com/rabbitmq/cluster-operator/v2/controllers.(*RabbitmqClusterReconciler).Reconcile\n\t/workspace/controllers/rabbitmqcluster_controller.go:246\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-12-03T07:54:17Z","msg":"Reconciler error","controller":"rabbitmqcluster","controllerGroup":"rabbitmq.com","controllerKind":"RabbitmqCluster","RabbitmqCluster":{"name":"rabbitmq","namespace":"openstack"},"namespace":"openstack","name":"rabbitmq","reconcileID":"54a92b40-282e-4c4d-a910-9c6c05ac8556","error":"failed to run queue rebalance on pod rabbitmq-server-0: command terminated with exit code 127","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-12-03T07:54:17Z","msg":"Start reconciling","controller":"rabbitmqcluster","controllerGroup":"rabbitmq.com","controllerKind":"RabbitmqCluster","RabbitmqCluster":{"name":"rabbitmq-cell1","namespace":"openstack"},"namespace":"openstack","name":"rabbitmq-cell1","reconcileID":"e5a4e89f-d2bf-4b58-b568-97a52aa1b2ec"}

      in upstream rabbit image `quay.io/podified-antelope-centos9/openstack-rabbitmq:current-podified`
       

      sh-5.1$ rpm -qa |grep rabbit
rabbitmq-server-3.9.21-1.el9s.x86_64
sh-5.1$ rabbitmq
rabbitmqctl           rabbitmq-defaults     rabbitmq-diagnostics  rabbitmq-env          rabbitmq-plugins      rabbitmq-server

      while in downstream `registry.redhat.io/rhoso/openstack-rabbitmq-rhel9@sha256:78e0fb4b21fb751fd0a2be1996f851e6d776c5d2a149f72d112bc2362411e9a9`

      [zuul@controller-0 ~]$ oc rsh -n openstack rabbitmq-cell1-server-0
Defaulted container "rabbitmq" out of: rabbitmq, setup-container (init)
sh-5.1$ rabbitmq
rabbitmqctl           rabbitmq-defaults     rabbitmq-diagnostics  rabbitmq-env          rabbitmq-plugins      rabbitmq-queues       rabbitmq-server       rabbitmq-streams      rabbitmq-upgrade
sh-5.1$ which rabbitmq-queues 
/usr/lib/rabbitmq/bin/rabbitmq-queues
sh-5.1$ rpm -qf /usr/lib/rabbitmq/bin/rabbitmq-queues
rabbitmq-server-3.9.10-6.el9ost.x86_64

              rhn-support-lmiccini Luca Miccini
              rhn-support-mschuppe Martin Schuppert
              rhos-dfg-pidone
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: