Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-5381

Implement Alerts and Metrics Dashboard for Vector Output Buffer

XMLWordPrintable

    • 3
    • False
    • None
    • False
    • NEW
    • NEW
    • Hide
      This feature introduces an alert to trigger when log collector's are buffering logs to a cluster node's file system and the buffers are consuming more then 15% of the available space. This is a possible indicator of the log collectors experiencing back pressure from their configured outputs and that administrators should take action to keep the collectors from potentially destabilizing the cluster node
      Show
      This feature introduces an alert to trigger when log collector's are buffering logs to a cluster node's file system and the buffers are consuming more then 15% of the available space. This is a possible indicator of the log collectors experiencing back pressure from their configured outputs and that administrators should take action to keep the collectors from potentially destabilizing the cluster node
    • Feature
    • Log Collection - Sprint 252, Log Collection - Sprint 253

      Summary

      Add Alerts that should be fired in case collectors potentially consuming too much node disk space.
      Add Metrics Dashboard that will be show consuming space Vector Output Buffer on each node.

      Acceptance Criteria

      1. Implement Alerting:

      • Set up alerting rules to trigger alerts when the space consumed by the Vector Output Buffer exceeds 15% of the total disk space on each node.
      • Test the alerting system to ensure that alerts are fired appropriately when the criteria are met.

      2. Create Grafana Dashboards:

      • Panel 1: Display the absolute size of the Vector buffer via a graph by instance.
      • Panel 2: Display the percentage of buffer size relative to the total disk space on the node.

          Configure the panels in each dashboard to visualize the required metrics accurately.

      3. Update Documentation

      Notes

      • We must understand if it is possible for these alerts to be enabled for non-infra namespaces and how to do that. I believe there is no way for non-infra namespaces to be opted into cluster metrics

            [LOG-5381] Implement Alerts and Metrics Dashboard for Vector Output Buffer

            Errata Tool added a comment -

            Since the problem described in this issue should be resolved in a recent advisory, it has been closed.

            For information on the advisory (Logging for Red Hat OpenShift - 6.0.0), and where to find the updated files, follow the link below.

            If the solution does not work for you, open a new bug report.
            https://access.redhat.com/errata/RHBA-2024:6693

            Errata Tool added a comment - Since the problem described in this issue should be resolved in a recent advisory, it has been closed. For information on the advisory (Logging for Red Hat OpenShift - 6.0.0), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2024:6693

            GitLab CEE Bot added a comment -

            CPaaS Service Account mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch openshift-logging-6.0-rhel-9_upstream_01bfca2d8cacccf01239fb09e7112e9d:

            Updated 3 upstream sources

            GitLab CEE Bot added a comment - CPaaS Service Account mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch openshift-logging-6.0-rhel-9_ upstream _01bfca2d8cacccf01239fb09e7112e9d : Updated 3 upstream sources

            GitLab CEE Bot added a comment -

            CPaaS Service Account mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch openshift-logging-6.0-rhel-9_upstream_ad90d8fb3b89a0c7e4d601840eccddaa:

            Updated 2 upstream sources

            GitLab CEE Bot added a comment - CPaaS Service Account mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch openshift-logging-6.0-rhel-9_ upstream _ad90d8fb3b89a0c7e4d601840eccddaa : Updated 2 upstream sources

            GitLab CEE Bot added a comment -

            CPaaS Service Account mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch openshift-logging-6.0-rhel-9_upstream_1ddb486521e9f822fe28906aa13c853c:

            Updated US source to: 981ec0c LOG-5621: Add pipeline validations

            GitLab CEE Bot added a comment - CPaaS Service Account mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch openshift-logging-6.0-rhel-9_ upstream _1ddb486521e9f822fe28906aa13c853c : Updated US source to: 981ec0c LOG-5621 : Add pipeline validations

            GitLab CEE Bot added a comment -

            CPaaS Service Account mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch openshift-logging-6.0-rhel-9_upstream_d6a0258de348d197c69f857a2f67bf12:

            Updated US source to: ae3452e LOG-5605: Add input secret validation

            GitLab CEE Bot added a comment - CPaaS Service Account mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch openshift-logging-6.0-rhel-9_ upstream _d6a0258de348d197c69f857a2f67bf12 : Updated US source to: ae3452e LOG-5605 : Add input secret validation

            GitLab CEE Bot added a comment -

            CPaaS Service Account mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch openshift-logging-6.0-rhel-9_upstream_6e600b06a4ee53e863c48be2ecacda67:

            Updated US source to: 9d1938d LOG-5601: Stub migration and validation into controller

            GitLab CEE Bot added a comment - CPaaS Service Account mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch openshift-logging-6.0-rhel-9_ upstream _6e600b06a4ee53e863c48be2ecacda67 : Updated US source to: 9d1938d LOG-5601 : Stub migration and validation into controller

            GitLab CEE Bot added a comment -

            CPaaS Service Account mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch openshift-logging-5.9-rhel-9_upstream_4fc12aa7c355b54f83dc063f0b42b363:

            Updated US source to: 37feb80 LOG-5381: add alerts metric and dashboard panels

            GitLab CEE Bot added a comment - CPaaS Service Account mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch openshift-logging-5.9-rhel-9_ upstream _4fc12aa7c355b54f83dc063f0b42b363 : Updated US source to: 37feb80 LOG-5381 : add alerts metric and dashboard panels

            GitLab CEE Bot added a comment -

            CPaaS Service Account mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch openshift-logging-6.0-rhel-9_upstream_a5cef1cc78682ab8fe6f23270c9412cb:

            Updated US source to: 02fb449 Fix NPE if inputs spec not declared

            GitLab CEE Bot added a comment - CPaaS Service Account mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch openshift-logging-6.0-rhel-9_ upstream _a5cef1cc78682ab8fe6f23270c9412cb : Updated US source to: 02fb449 Fix NPE if inputs spec not declared

            GitLab CEE Bot added a comment -

            CPaaS Service Account mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch openshift-logging-6.0-rhel-9_upstream_a96f0657fd44c2e14a4fcd3618dc6470:

            Updated US source to: 0f2e265 Fix NPE is output type spec not declared

            GitLab CEE Bot added a comment - CPaaS Service Account mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch openshift-logging-6.0-rhel-9_ upstream _a96f0657fd44c2e14a4fcd3618dc6470 : Updated US source to: 0f2e265 Fix NPE is output type spec not declared

            GitLab CEE Bot added a comment -

            CPaaS Service Account mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch openshift-logging-6.0-rhel-9_upstream_d7ac97e0b6e0dc061202f5953efe63cc:

            Updated US source to: 1977791 LOG-5131: Remove old telemetry implementation from development branch

            GitLab CEE Bot added a comment - CPaaS Service Account mentioned this issue in a merge request of openshift-logging / Log Collection Midstream on branch openshift-logging-6.0-rhel-9_ upstream _d7ac97e0b6e0dc061202f5953efe63cc : Updated US source to: 1977791 LOG-5131 : Remove old telemetry implementation from development branch

              vparfono Vitalii Parfonov
              vparfono Vitalii Parfonov
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: