Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-7111

[Operator, upgrade] 7.10 versions of operator tend to remove stateful set during upgrade

XMLWordPrintable

    • False
    • None
    • False
    • Documentation (Ref Guide, User Guide, etc.), Release Notes, User Experience
    • Hide

      Version within the channel:

      1. Install operator 7.10.0-opr3 (OLM and 7.10.x channel used)
      2. Deploy CR with basic broker deployment with persistence and message migration size (size doesn't matter this one has just been used during testing) and upgrades enabled
      3. Update index image to trigger the upgrade procedure
      4. Operator 7.10.1-opr1 will be installed and takes over the deployment
      5. expectation: Broker pod with highest ordinal number will shut down and spins back up. Then the same happens to the pod with lower ordinal number till all pods have been updated.
      6. result: all pods will terminate and start spinning up like in case of new deployment.
      Show
      Version within the channel: Install operator 7.10.0-opr3 (OLM and 7.10.x channel used) Deploy CR with basic broker deployment with persistence and message migration size (size doesn't matter this one has just been used during testing) and upgrades enabled Update index image to trigger the upgrade procedure Operator 7.10.1-opr1 will be installed and takes over the deployment expectation: Broker pod with highest ordinal number will shut down and spins back up. Then the same happens to the pod with lower ordinal number till all pods have been updated. result: all pods will terminate and start spinning up like in case of new deployment.

      In case of upgrading the operator version, usually there are no changes of the custom resources or anything else that would require update of immutable field in stateful set. Only the container image url is changed when upgrades of the broker deployment are enabled. That change should result only in rolling update, where individual pods are terminated and then brought back up one by one. That's the behavior observed with upgrades of 7.8 and 7.9 aligned operators. With the 7.10 versions, upgrades of the stateful set seem to be failing and therefore operator will remove and recreate it. That causes all pods to shut down at the same time and causes short outage in service availability. This should not be happening.

        1. amq-broker-operator-710-1.log
          624 kB
        2. amq-broker-operator-710-2.log
          463 kB
        3. amq-broker-operator-710-without-persistence.log
          440 kB
        4. cr_file_without_persistence.yaml
          0.7 kB
        5. cr_file.yaml
          0.7 kB
        6. debug-amq-broker-operator7.10.0-opr3.log
          281 kB
        7. debug-amq-broker-operator7.10.1-opr1.log
          423 kB

              gtully@redhat.com Gary Tully
              rvais Roman Vais (Inactive)
              Roman Vais Roman Vais (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: