Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-56921

Mass In-cluster Disruption / Test Failures

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • 4
    • None
    • None
    • None
    • None
    • None
    • Metal Platform 273, Metal Platform 274, Metal Platform 275
    • 3
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Followup from Slack investigation. We are seeing large batches of in cluster disruption that also results in a high number of test failures.

      These test failures in turn lead to Component Readiness regressions and block Feature Gate promotions.

      Working with rh-ee-tdomnesc he has isolated the worst offending node as `cir-0831` and put it in maintenance. We still don't have the root cause of the issue to explain the impact on the other nodes including both ibmcloud and ironic servers.

            97   "name": "cir-0831", "provider": "ibmcloud",
            9   "name": "cir-1126", "provider": "ibmcloud",
            9   "name": "cir-0829", "provider": "ibmcloud",
            8   "name": "cir-1117", "provider": "ibmcloud",
            8   "name": "cir-0997", "provider": "ibmcloud",
            8   "name": "cir-0839", "provider": "ibmcloud",
            5   "name": "cir-0636", "provider": "ironic",
            5   "name": "cir-0503", "provider": "ironic",
            4   "name": "cir-1093", "provider": "ibmcloud",
            3   "name": "cir-1130", "provider": "ibmcloud",
            3   "name": "cir-0853", "provider": "ibmcloud",
            3   "name": "cir-0847", "provider": "ibmcloud",
            3   "name": "cir-0833", "provider": "ibmcloud",
            3   "name": "cir-0807", "provider": "ibmcloud",
            2   "name": "cir-0851", "provider": "ibmcloud",
            1   "name": "cir-0815", "provider": "ibmcloud",
      

              rh-ee-tdomnesc Tudor Domnescu
              rh-ee-fbabcock Forrest Babcock
              None
              None
              Devan Goodwin Devan Goodwin
              None
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated: