Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-3021

Drainer pod is not started when broker is scaled down using AMQ operator

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Done
    • AMQ 7.5.0.GA, AMQ 7.6.0.GA
    • None
    • operator
    • None

    Description

      After scale down the size of an ActiveMQArtemis resource the drainer pod is not started. AMQ operator shows the following messages:

      {"level":"info","ts":1571390223.4943252,"logger":"controller_activemqartemisscaledown","msg":"Found orphaned PVC(s) for ordinal 5. Creating drain pod amqp-broker-ss-5"}
      {"level":"info","ts":1571390223.4943724,"logger":"controller_activemqartemisscaledown","msg":"Ordinal zero pod amqp-broker-ss podCondition Ready True, proceeding to create drainer pod."}
      {"level":"error","ts":1571390223.502789,"logger":"controller_activemqartemisscaledown","msg":"Error while creating drain Pod amqp-broker-ss-5: ","error":"pods \"amqp-broker-ss-5\" is forbidden: error looking up service account kronos-dev/activemq-artemis-operator: serviceaccount \"activemq-artemis-operator\" not found","stacktrace":"github.com/rh-messaging/activemq-artemis-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/tmp/go/src/github.com/rh-messaging/activemq-artemis-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/rh-messaging/activemq-artemis-operator/pkg/draincontroller.(*Controller).processStatefulSet\n\t/tmp/go/src/github.com/rh-messaging/activemq-artemis-operator/pkg/draincontroller/controller.go:418\ngithub.com/rh-messaging/activemq-artemis-operator/pkg/draincontroller.(*Controller).syncHandler\n\t/tmp/go/src/github.com/rh-messaging/activemq-artemis-operator/pkg/draincontroller/controller.go:287\ngithub.com/rh-messaging/activemq-artemis-operator/pkg/draincontroller.(*Controller).processNextWorkItem.func1\n\t/tmp/go/src/github.com/rh-messaging/activemq-artemis-operator/pkg/draincontroller/controller.go:239\ngithub.com/rh-messaging/activemq-artemis-operator/pkg/draincontroller.(*Controller).processNextWorkItem\n\t/tmp/go/src/github.com/rh-messaging/activemq-artemis-operator/pkg/draincontroller/controller.go:247\ngithub.com/rh-messaging/activemq-artemis-operator/pkg/draincontroller.(*Controller).runWorker\n\t/tmp/go/src/github.com/rh-messaging/activemq-artemis-operator/pkg/draincontroller/controller.go:199\ngithub.com/rh-messaging/activemq-artemis-operator/pkg/draincontroller.(*Controller).runWorker-fm\n\t/tmp/go/src/github.com/rh-messaging/activemq-artemis-operator/pkg/draincontroller/controller.go:185\ngithub.com/rh-messaging/activemq-artemis-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/tmp/go/src/github.com/rh-messaging/activemq-artemis-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/rh-messaging/activemq-artemis-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/tmp/go/src/github.com/rh-messaging/activemq-artemis-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/rh-messaging/activemq-artemis-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/tmp/go/src/github.com/rh-messaging/activemq-artemis-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
      E1018 09:17:03.502887       1 controller.go:250] error syncing 'kronos-dev/amqp-broker-ss: pods "amqp-broker-ss-5" is forbidden: error looking up service account kronos-dev/activemq-artemis-operator: serviceaccount "activemq-artemis-operator" not found
      

      It seems that AMQ operator wants to use the SA activemq-artemis-operator, however that SA does not exists because our operator uses other one called amq-broker-operator.

      AMQ broker operator has the following deployment configuration:

      $ oc describe deployment amq-broker-operator
      Name:                   amq-broker-operator
      Namespace:              amq-broker-dev
      CreationTimestamp:      Fri, 18 Oct 2019 15:30:42 +0200
      Labels:                 <none>
      Annotations:            deployment.kubernetes.io/revision=1
      Selector:               name=amq-broker-operator
      Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
      StrategyType:           RollingUpdate
      MinReadySeconds:        0
      RollingUpdateStrategy:  25% max unavailable, 25% max surge
      Pod Template:
        Labels:           name=amq-broker-operator
        Service Account:  amq-broker-operator
        Containers:
         amq-broker-operator:
          Image:      registry.redhat.io/amq7/amq-broker-rhel7-operator:0.9
          Port:       <none>
          Host Port:  <none>
          Command:
            /home/amq-broker-operator/bin/entrypoint
          Args:
            --zap-level debug
          Environment:
            WATCH_NAMESPACE:   (v1:metadata.namespace)
            POD_NAME:          (v1:metadata.name)
            OPERATOR_NAME:    amq-broker-operator
          Mounts:             <none>
        Volumes:              <none>
      Conditions:
        Type           Status  Reason
        ----           ------  ------
        Available      True    MinimumReplicasAvailable
        Progressing    True    NewReplicaSetAvailable
      OldReplicaSets:  <none>
      NewReplicaSet:   amq-broker-operator-5c455b9b4d (1/1 replicas created)
      Events:          <none>
      

      Attachments

        Issue Links

          Activity

            People

              rhn-support-rkieley Roderick Kieley
              rmarting@redhat.com Roman Martin Gil
              Roman Vais Roman Vais
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: