Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-26421

ROSA HCP Outage in us-east-2 - High incoming rate causing service degradation

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False
    • Not Selected
    • OSDFM Sprint 1, OSDFM Sprint 2
    • None

      Starting at Nov 18 7:13 AM EST, the HCP incoming rate in the us-east-2 region began a steep and sustained increase, peaking at a critical load of 46 HCP per hour. See deatil in grafana dashboard

      Due to the us-east-2 region has only one Service Cluster, the cluster saturation quickly increased from the 60% to 90% and reached capacity limit around 10:30 AM EST, then the us-east-2 region went offline.

      New dormant cluster has been created 9:40 AM EST and became ready at 1:27 PM EST

      Slack thread: https://redhat-internal.slack.com/archives/C035W96HKN3/p1763485257458419

        1. Screenshot 2025-11-18 at 10.58.03 AM-1.png
          Screenshot 2025-11-18 at 10.58.03 AM-1.png
          136 kB
        2. Screenshot 2025-11-18 at 11.47.16 AM.png
          Screenshot 2025-11-18 at 11.47.16 AM.png
          131 kB

          There are no Sub-Tasks for this issue.

              rh-ee-chlu Chee Lu
              rh-ee-chlu Chee Lu
              Anna Francis Anna Francis
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: