Uploaded image for project: 'Red Hat Process Automation Manager'
  1. Red Hat Process Automation Manager
  2. RHPAM-3296

BA Operator 7.9 stuck in infinite reconcile loop

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • None
    • 7.9.0.GA
    • Cloud
    • None
    • OCP 4.6.1+

    • False
    • False
    • Undefined
    • Hide

      Install BA Operator from Operator Hub, then check BA Operator pod logs.

      Show
      Install BA Operator from Operator Hub, then check BA Operator pod logs.
    • ---
    • ---
    • 2020 Week 46-48 (from Nov 9)

      Due to an Operator Lifecycle Manager change in OCP 4.6.1, there's now a bug in BA operator 7.9 which causes an infinite reconcile loop on a few objects.
      This is due to a new label that OLM is setting on objects a CSV owns... which is causing the operator to get stuck in a reconcile loop for the console UI deployment.

      Operator repeatedly reconciles console objects owned by CSV...

      {"level":"info","ts":"2020-11-06T19:32:09.064455626Z","logger":"kieapp.controller","msg":"No Custom Resource found named businessautomation-operator.7.9.1-dev-66g4svx9hl. Checking for dependent objects to delete."}
      {"level":"info","ts":1604691129.065619,"logger":"comparator","msg":"Objects are not equal","deployed":{"operators.coreos.com/businessautomation-operator.test":""},"requested":null}
      {"level":"info","ts":1604691129.0657241,"logger":"comparator","msg":"Resources are not equal","deployed":{"apiVersion":"rbac.authorization.k8s.io/v1","kind":"Role","namespace":"test","name":"console-cr-form"},"requested":{"namespace":"test","name":"console-cr-form"}}
      {"level":"info","ts":1604691129.0657523,"logger":"comparator","msg":"Objects are not equal","deployed":{"operators.coreos.com/businessautomation-operator.test":""},"requested":null}
      {"level":"info","ts":1604691129.0658062,"logger":"comparator","msg":"Resources are not equal","deployed":{"apiVersion":"rbac.authorization.k8s.io/v1","kind":"RoleBinding","namespace":"test","name":"console-cr-form"},"requested":{"namespace":"test","name":"console-cr-form"}}
      {"level":"info","ts":1604691129.06583,"logger":"comparator","msg":"Objects are not equal","deployed":{"operators.coreos.com/businessautomation-operator.test":""},"requested":null}
      {"level":"info","ts":1604691129.0658462,"logger":"comparator","msg":"Resources are not equal","deployed":{"apiVersion":"v1","kind":"ServiceAccount","namespace":"test","name":"console-cr-form"},"requested":{"namespace":"test","name":"console-cr-form"}}
      {"level":"info","ts":1604691129.0658891,"logger":"comparator","msg":"Objects are not equal","deployed":{"operators.coreos.com/businessautomation-operator.test":""},"requested":null}
      

      CRC authorization server eventually stops responding -

      {"error":"server_error","error_description":"The authorization server encountered an unexpected condition that prevented it from fulfilling the request.","state":"1680a864"}
      

      QE was able to fully reproduce this issue in OCP running in OpenStack. On OCP running in AWS we only saw loop in BA Operator pod's log.

      Opened as a blocker as this issue can cause instability of OpenShift environment.

              tohughes Tommy Hughes
              jakubschwan Jakub Schwan
              Jakub Schwan Jakub Schwan
              Jakub Schwan Jakub Schwan
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: