Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-17346

Avoid recreating prometheus Statefulset during 4.13 > 4.14 upgrades

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done-Errata
    • Major
    • 4.14.0
    • 4.14
    • Monitoring
    • None
    • Important
    • No
    • MON Sprint 240
    • 1
    • Approved
    • False
    • Hide

      None

      Show
      None
    • NA
    • Release Note Not Required

    Description

      Description of problem:

      Due to https://github.com/openshift/cluster-monitoring-operator/pull/1986, the prometheus-operator was instructed to inject the app.kubernetes.io/part-of: openshift-monitoring label (via its --labels option) to resources it creates.
      
      The label is also 

      Version-Release number of selected component (if applicable):

      4.14

      How reproducible:

      upgrade to a 4.14 version with the commit https://github.com/openshift/cluster-monitoring-operator/pull/1986

      Steps to Reproduce:

      1.
      2.
      3.
      

      Actual results:

       

      Expected results:

      We should avoid recreating the statefulset as this leads to downtime (for Prometheus, both Pods are recreated)

      Additional info:

       
      • See why prometheus-operator doesn't use cascade=orphan for deletion (to keep the Pods around and avoid downtime)
      • Maybe other statefulsets are recreated as well (alertmanager etc.), maybe removing the --labels option will fix it for all of them (they are all created by the operator)
      • See if we touched the matchLabels of other statefulsets outside the control of the prom operator
      • See if we can add an origin test to make sure Statefulsets (maybe other resources as well are not recreated), can we really live with that? (what if we really want to change an immutable field), maybe in origin we can specify upgrade versions??

      Attachments

        Activity

          People

            rh-ee-amrini Ayoub Mrini
            rh-ee-amrini Ayoub Mrini
            Junqi Zhao Junqi Zhao
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: