Uploaded image for project: 'Red Hat OpenShift AI Engineering'
  1. Red Hat OpenShift AI Engineering
  2. RHOAIENG-2040

RHODS Dashboard Route Error Burn Rate

    XMLWordPrintable

Details

    • Dashboard - General 2.7
    • Critical

    Description

      All the rhods-dashboard pods are continuously crash-looping with the following error

      <--- Last few GCs --->
      [17:0x55c96512e860]   169203 ms: Mark-sweep (reduce) 1023.2 (1041.9) -> 1021.8 (1041.6) MB, 1802.7 / 0.1 ms  (+ 446.8 ms in 159 steps since start of marking, biggest step 85.8 ms, walltime since start of marking 2514 ms) (average mu = 0.341, current mu = [17:0x55c96512e860]   173006 ms: Mark-sweep (reduce) 1023.9 (1042.6) -> 1022.6 (1042.4) MB, 2511.5 / 0.1 ms  (+ 298.6 ms in 69 steps since start of marking, biggest step 83.4 ms, walltime since start of marking 2895 ms) (average mu = 0.301, current mu = 0
      <--- JS stacktrace --->
      FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1
      

      Further logs attached to JIRA https://issues.redhat.com/browse/OHSS-30965

      Have alerted the engineering via https://redhat-internal.slack.com/archives/C019V4BS8G6/p1705717665919479

      Attachments

        1. 1_subscription_creation.png
          1_subscription_creation.png
          24 kB
        2. 2_subscription_grid.png
          2_subscription_grid.png
          362 kB
        3. 3_pods.png
          3_pods.png
          100 kB
        4. delete-subs.sh
          0.5 kB
        5. delete-subs.txt
          0.3 kB
        6. make-subs.sh
          9 kB

        Activity

          People

            gkrumbac Gage Krumbach
            rhn-support-cabeywar Chamal Abeywardhana
            Federico Alonso
            RHOAI Dashboard
            Votes:
            0 Vote for this issue
            Watchers:
            14 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty