Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-10671

Risk cache warming takes too long on channel changes


    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • None
    • 4.13, 4.12, 4.11, 4.10
    • None
    • No
    • Rejected
    • False
    • Hide


    • NA

      This is a clone of issue OCPBUGS-10514. The following is the description of the original issue:

      This is a clone of issue OCPBUGS-10221. The following is the description of the original issue:

      This is a clone of issue OCPBUGS-5469. The following is the description of the original issue:

      Description of problem:

      When changing channels it's possible that multiple new conditional update risks will need to be evaluated. For instance, a cluster running 4.10.34 in a 4.10 channel today only has to evaluate `OpenStackNodeCreationFails` but when the channel is changed to a 4.11 channel multiple new risks require evaluation and the evaluation of new risks is throttled at one every 10 minutes. This means if there are three new risks it may take up to 30 minutes after the channel has changed for the full set of conditional updates to be computed. This leads to a perception that no update paths are recommended because most will not wait 30 minutes, they expect immediate feedback.

      Version-Release number of selected component (if applicable):

      4.10.z, 4.11.z, 4.12, 4.13

      How reproducible:


      Steps to Reproduce:

      1. Install 4.10.34
      2. Switch from stable-4.10 to stable-4.11

      Actual results:

      Observe no recommended updates for 10-20 minutes because all available paths to 4.11 have a risk associated with them

      Expected results:

      Risks are computed in a timely manner for an interactive UX, lets say < 10s

      Additional info:

      This was intentional in the design, we didn't want risks to continuously re-evaluate or overwhelm the monitoring stack, however we didn't anticipate that we'd have long standing pile of risks and realize how confusing the user experience would be.
      We intend to work around this in the deployed fleet by converting older risks from `type: promql` to `type: Always` avoiding the evaluation period but preserving the notification. While this may lead customers to believe they're exposed to a risk they may not be, as long as the set of outstanding risks to the latest version is limited to no more than one it's likely no one will notice. All 4.10 and 4.11 clusters currently have a clear path toward relatively recent 4.10.z or 4.11.z with no more than one risk to be evaluated.

            lmohanty@redhat.com Lalatendu Mohanty
            openshift-crt-jira-prow OpenShift Prow Bot
            Yang Yang Yang Yang
            0 Vote for this issue
            7 Start watching this issue