Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-42393

No alerting in openshift-controller-manager during pathological behavior / rate limiting

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Important
    • No
    • None
    • None
    • Rejected
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      https://issues.redhat.com/browse/OCPBUGS-42106 demonstrated a scenario where the openshift-controller-manager was spinning constantly because of a failure to reconcile. It was triggering client-side rate limiting logic, as indicated by the pod logs, but otherwise not being reported.
      
      

      Version-Release number of selected component (if applicable):

          4.18

      How reproducible:

      n/a. https://issues.redhat.com/browse/OCPBUGS-42106 suggests a scenario where duplicate image registry hostnames can be specified in the openshiftcontrollermanager configuration to cause the problem. However, this has been fixed already in 4.18.
      As such, it may only be possible to exercise this in a development environment.

      Steps to Reproduce:

          1.
          2.
          3.
          

      Actual results:

      Client side throttling can persist indefinitely due to a irreconcilable control loop with no direct notification to SRE.

      Expected results:

      Persistent client side throttling, or the detection of a condition that would cause such throttling, should cause an alert to fire.

      Additional info:

          

              rh-ee-sabiswas Sayan Biswas
              jupierce Justin Pierce
              David Eads
              None
              Jitendar Singh Jitendar Singh
              None
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: