Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-569

CVO History Pruner is non-functional, letting history length above MaxHistory

XMLWordPrintable

    • Moderate
    • 1
    • OTA 224
    • 1
    • False
    • Hide

      None

      Show
      None

      Description of problem:
      Latest implementation of history pruner (pr805 [1]) had increased max upgrade history in cvo to 100, and implemented a weight based pruning priority strategy for in case history length grows any larger. This pruning however is not happening, letting history grow uncontrollably, and potentially reach resource limits of etcd or kubernetes.

      Observed the following while running continuous upgrade-rollback cycles:

      $ oc get clusterversion version -o json | jq '.status.history|length'
      203

      Version-Release number of selected component (if applicable):
      4.12.0-0.nightly-2022-08-23-223922
      4.12.0-0.nightly-2022-08-23-153511

      How reproducible:
      1/1

      Steps to Reproduce:
      Same as described in bz2097067 [2], with addition of waiting a few minutes after the first rollback to allow it to reach 'Completed' state.

      Actual results:
      History grows uncontrollably

      Expected results:
      History should be pruned to keep max size of 100

      Additional info:

      [1] https://github.com/openshift/cluster-version-operator/pull/805
      [2] https://bugzilla.redhat.com/show_bug.cgi?id=2097067#c4

            evakhoni@redhat.com Evgeni Vakhonin
            evakhoni@redhat.com Evgeni Vakhonin
            Evgeni Vakhonin Evgeni Vakhonin
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: