Uploaded image for project: 'Red Hat OpenShift Data Science'
  1. Red Hat OpenShift Data Science
  2. RHODS-4908

Downtime during openshift upgrade

XMLWordPrintable

    • False
    • None
    • False
    • No
    • No
    • No
    • Pending
    • None
    • Important

      Description of problem:

      During the upgrade we found two downtimes. One of them takes 52 seconds to be recovered and the second one 31 seconds.

      We have observed that blackbox-exporter is down during the openshift upgrade and also we lost Jupyterhub service. We spawn a notebook and we did run a infinite loop and the notebook fails during the upgrade. 

      Prerequisites (if any, like setup, operators/versions):

      Upgrade path is available in the channel and major versions, y stream

      Steps to Reproduce

       1. Change channel to candidate if the major version is not available in the current channel. 

      echo '{"version":{"channel_group":"candidate"}}' | ocm patch cluster <clusted id>

      2. Upgrade Openshift

      Go to cluster setting -> Click on Update tab-> Select major version you want to upgrade on the cluster -> Click Next -> Schedule Update ->Confirmation 

      3. Verify into Prometheus the metric

      rhods_aggregate_availability

      Look the downtime in the components

      Actual results:

      There are two downtimes

      Expected results:

      There is not downtime

      Reproducibility (Always/Intermittent/Only Once):

      Always

      Build Details:

      Workaround:

      Additional info:

        1. blackbox_prometheus_disrupted.png
          156 kB
          Pablo Felix
        2. image-2022-08-26-17-55-54-111.png
          54 kB
          Pablo Felix
        3. JH_downtime.png
          73 kB
          Pablo Felix
        4. prometheus_down.png
          89 kB
          Pablo Felix
        5. Screenshot from 2022-08-08 13-53-13.png
          50 kB
          Pablo Felix
        6. Screenshot from 2022-08-08 13-58-33.png
          45 kB
          Pablo Felix
        7. Screenshot from 2022-08-08 13-59-30.png
          48 kB
          Pablo Felix
        8. Screenshot from 2022-08-08 14-17-57.png
          270 kB
          Pablo Felix
        9. Screenshot from 2022-08-08 14-18-12.png
          45 kB
          Pablo Felix

              Unassigned Unassigned
              pablo-rhods Pablo Felix (Inactive)
              Pablo Felix Pablo Felix (Inactive)
              Tarun Kumar
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: