Upgrade the operator - even when the broker images are fixed using SHA tags - causes all the brokers to restart. While it would be necessary to restart the brokers if they had actually changed, no change can be detected here, and the use of SHA tags forces the operator to maintain a specific image.

This problem certainly affects the upgrade from 7.11.3 to 7.11.4. It's been reported to affect the upgrade from 7.11.1 to 7.11.4 as well. I'm not sure what other versions are affected, because it's somewhat tedious to test.

Steps to reproduce.

1. Install clean OCP (I'm using AWS)
2. Deploy a subscription for the 7.11.3 operator:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: amq-broker-rhel8
  namespace: openshift-operators
spec:
  channel: 7.11.x
  installPlanApproval: Automatic
  name: amq-broker-rhel8
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  startingCSV: amq-broker-operator.v7.11.3-opr-1
  installPlanApproval: Manual  

Note that 'Manual' is crucial here, else the OLM will immediately upgrade to 7.11.4, which will break the test.

3. Manually approve the install plan, e.g., using the console

4. Deploy brokers using the existing operator:

apiVersion: broker.amq.io/v1beta1
kind: ActiveMQArtemis
metadata:
  name: ex-aao
  namespace: broker
spec:
  deploymentPlan:
    image: registry.redhat.io/amq7/amq-broker-rhel8@sha256:0edbc311d89da507081c529a1696697f95fdc1b22ff07bdbae21a9fd44c49358
    initImage: registry.redhat.io/amq7/amq-broker-init-rhel8@sha256:8b4c4400a02c6cacfd978147b158e2ccbe5da955578b4b970a5ee7edac865326
    journalType: nio
    managementRBACEnabled: true
    messageMigration: false
    persistenceEnabled: false
    requireLogin: false
    size: 2

I'm using SHA hashes after a discussion with Gary T. We thought that the operator might assume that images identified with tags only might be able to change. Upgrading the brokers might be a conservative action.

5. Wait until the ex-aao-N brokers are running

6. In the console, go to 'installed operators'. Note that the AMQ operator shows 'upgrade available'. Approve the upgrade.

7. Using, e.g., 'oc get pods' to check the broker pods. They will restart after a minute or two. This isn't obvious unless you're actually watching, because:

8. Using., e.g., 'oc describe pod ex-aao-ss-0', see that the pods have not actually changes. Specifically, they are using the same image hash as in the original subscription. So these pods have restarted, even though they haven't changed – at least, their images haven't changed.