Uploaded image for project: 'Network Observability'
  1. Network Observability
  2. NETOBSERV-975

Flows dropped due to Loki stream limit during large traffic spikes

    XMLWordPrintable

Details

    • False
    • None
    • False
    • Important

    Description

      In recent NetObserv 1.2 performance testing, we have witnessed the following behavior:

      • During spikes of load, certain eBPF pods will be OOMKilled and go into CrashLoopBackOff state. Notably these have been observed to be the same pods co-located on nodes with LokiStack resources, which have high memory usage
        • This behavior was observed with both small and medium sized LokiStacks as well as the default eBPF memory limit of 800Mi as well as an increased limit of 1000Mi
      • Flows continue to be processed but some are dropped during these spike periods.

      The operator recovers after the load spikes end with eBPF pods recovering and flows returning to being written.

      Opening this bug to track the behavior and gather more data.

      Discussions relating to this bug:

      Attachments

        Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. NETOBSERV-717 Loki per_stream_rate_limit

          • Normal - To be worked after urgent and high priorities are resolved. This term is chosen when the severity of an issue is relatively close to the level of effort to fix it. The existence of an easily implemented workaround can also lead to this priority level instead of a higher priority.
          • Closed
          split from

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. NETOBSERV-902 QE: Run performance tests for 1.2 release

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. NETOBSERV-1068 Run performance tests for 1.3 release

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          links to

          GitHub netobserv/netobserv-ebpf-agent#119: NETOBSERV-975: avoid preallocating huge chunk of memory by default

          GitHub openshift/openshift-docs#63934: Network Observability 1.4 Release Notes

          Web Link RHSA-2023:116729 Network Observability 1.4.0 for OpenShift

          Activity

            People

              jpinsonn@redhat.com Julien Pinsonneau
              nweinber1 Nathan Weinberg
              Nathan Weinberg Nathan Weinberg
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: