Uploaded image for project: 'OpenShift Monitoring'
  1. OpenShift Monitoring
  2. MON-1796

fsSpaceFillingUpWarningThreshold not aligned to Kubernetes Garbage Collection Threshold

    XMLWordPrintable

Details

    • False
    • False
    • NEW
    • NEW
    • Undefined
    • 0

    Description

      Based on https://github.com/prometheus-operator/kube-prometheus/issues/294, fsSpaceFillingUpCriticalThreshold was adjusted as it was in conflict with the Kubernetes Garbage Collection default values (nicely documented in https://www.openshift.com/blog/image-garbage-collection-in-openshift and https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/).

      But with the default value for imageGCLowThresholdPercent, we are still able to trigger NodeFilesystemSpaceFillingUp on warning level as this is starting when 40% space is left.

      Hence even with Garbage Collection running, the alert may continue to fire, even though necessary clean-up is done.

      This is why fsSpaceFillingUpWarningThreshold should be adjusted to a reasonable value and if going to 20% is not possible, it would be required to talk to kubelet Engineering group to understand whether Garbage Collection default values need to be adjusted.

      Attachments

        Activity

          People

            Unassigned Unassigned
            rhn-support-sreber Simon Reber
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: