Uploaded image for project: 'Network Observability'
  1. Network Observability
  2. NETOBSERV-674

Alert when loki ingestion limits are hit

XMLWordPrintable

    • BU Product Work
    • False
    • None
    • False
    • OCPSTRAT-458 - OpenShift Network Observability Operator enhancements
    • NetObserv - Sprint 230, NetObserv - Sprint 231, NetObserv - Sprint 232
    • QE Confirmed

      When logs volume exceeds loki ingestion limits, we get below errors in FLP, it would be good to signal user in UI to increase loki ingestion rate (ingestionRate param in LokiStack).

       

      This also results FLP being behind in flows processing, it reads oldest unread flows from Kafka which users may not realize without signal until they adjust time ranges.

       

      time=2022-10-21T20:04:56Z level=info component=client error=server returned HTTP status 429 Too Many Requests (429): Ingestion rate limit exceeded for user netobserv (limit: 15728640 bytes/sec) while attempting to ingest '19819' lines totaling '10485702' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased fields.level=warn fields.msg=error sending batch, will retry host=lokistack-distributor-http.openshift-operators-redhat.svc:3100 module=export/loki status=49

       

      The NOO should automatically create an Alert in OCP console to show these reached limits

       

        1. screenshot-1.png
          130 kB
          Mehul Modi

              ocazade@redhat.com Olivier Cazade
              rhn-support-memodi Mehul Modi
              Mehul Modi Mehul Modi
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: