Uploaded image for project: 'OpenShift Monitoring'
  1. OpenShift Monitoring
  2. MON-2685

[2041570] Lost all collected metrics data when both Prometheus pods restarted

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Can't Do
    • Icon: Undefined Undefined
    • None
    • None
    • Prometheus
    • False
    • False
    • NEW
    • NEW
    • 0

      Description of problem:
      After draining nodes both pods (prometheus-k8s-0 and prometheus-k8s-1) restarted and all data that was previously collected by metrics removed from a cluster

      Version-Release number of selected component (if applicable):
      CNV-4.10

      How reproducible:
      100%

      Steps to Reproduce:
      1. drain first node where pod prometheus-k8s-0 is running
      2. return node back and drain second node where prometheus-k8s-1 is running
      3. check metrics on a cluster - old data removed

      Actual results:
      Metrics cleared when both pods prometheus-k8s-0 and prometheus-k8s-1 restarted

      Expected results:
      Old data should be kept

      Additional info:
      I guess some persistent storage should be using for keeping all metrics:
      https://docs.openshift.com/container-platform/4.9/monitoring/configuring-the-monitoring-stack.html#configuring-persistent-storage

            sradco Shirly Radco
            dshchedr@redhat.com Denys Shchedrivyi
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: