POD_IPS: 10.129.1.121, PROM_BIND_IP: 0.0.0.0
Setting each total_size_limit for 4 buffers to 4809148876 bytes
Setting queued_chunks_limit_size for each buffer to 573
Setting chunk_limit_size for each buffer to 8388608
/var/lib/fluentd/pos/journal_pos.json exists, checking if yajl parser able to parse this json file without any error.
ruby 2.7.8p225 (2023-03-30 revision 1f4d455848) [x86_64-linux]
RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR=0.900000 (default value: 2.000000)
checking if /var/lib/fluentd/pos/journal_pos.json a valid json by calling yajl parser
2023-11-06 12:15:16 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp
2023-11-06 12:15:16 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp
/usr/local/share/gems/gems/fluent-plugin-elasticsearch-5.2.2/lib/fluent/plugin/elasticsearch_compat.rb:8: warning: already initialized constant TRANSPORT_CLASS
/usr/local/share/gems/gems/fluent-plugin-elasticsearch-5.2.2/lib/fluent/plugin/elasticsearch_compat.rb:3: warning: previous definition of TRANSPORT_CLASS was here
/usr/local/share/gems/gems/fluent-plugin-elasticsearch-5.2.2/lib/fluent/plugin/elasticsearch_compat.rb:25: warning: already initialized constant SELECTOR_CLASS
/usr/local/share/gems/gems/fluent-plugin-elasticsearch-5.2.2/lib/fluent/plugin/elasticsearch_compat.rb:20: warning: previous definition of SELECTOR_CLASS was here
2023-11-06 12:15:20 +0000 [warn]: For security reason, setting private_key_passphrase is recommended when cert_path is specified
2023-11-06 12:15:20 +0000 [warn]: [default_loki_audit] failed to write post to https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit/loki/api/v1/push (429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5013' lines totaling '7268054' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased
)
2023-11-06 12:15:20 +0000 [warn]: [default_loki_audit] failed to flush the buffer. retry_times=0 next_retry_time=2023-11-06 12:15:21 +0000 chunk="60978f0b4841bef69a0b4c4e1861634f" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5013' lines totaling '7268054' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"
  2023-11-06 12:15:20 +0000 [warn]: /usr/local/share/gems/gems/fluent-plugin-grafana-loki-1.2.18/lib/fluent/plugin/out_loki.rb:168:in `write'
  2023-11-06 12:15:20 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1179:in `try_flush'
2023-11-06 12:15:20 +0000 [warn]: [default_loki_audit] failed to write post to https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit/loki/api/v1/push (429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5476' lines totaling '8216236' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased
)
  2023-11-06 12:15:20 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1500:in `flush_thread_run'
  2023-11-06 12:15:20 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start'
  2023-11-06 12:15:20 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2023-11-06 12:15:20 +0000 [warn]: [default_loki_audit] failed to flush the buffer. retry_times=0 next_retry_time=2023-11-06 12:15:21 +0000 chunk="60978f0ac44b83e5653df5bba8cdb2ec" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5476' lines totaling '8216236' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"
  2023-11-06 12:15:20 +0000 [warn]: /usr/local/share/gems/gems/fluent-plugin-grafana-loki-1.2.18/lib/fluent/plugin/out_loki.rb:168:in `write'
  2023-11-06 12:15:20 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1179:in `try_flush'
  2023-11-06 12:15:20 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1500:in `flush_thread_run'
  2023-11-06 12:15:20 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start'
  2023-11-06 12:15:20 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2023-11-06 12:15:22 +0000 [warn]: [default_loki_audit] failed to write post to https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit/loki/api/v1/push (429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5013' lines totaling '7268054' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased
)
2023-11-06 12:15:22 +0000 [warn]: [default_loki_audit] failed to flush the buffer. retry_times=1 next_retry_time=2023-11-06 12:15:25 +0000 chunk="60978f0b4841bef69a0b4c4e1861634f" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5013' lines totaling '7268054' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"
  2023-11-06 12:15:22 +0000 [warn]: suppressed same stacktrace
2023-11-06 12:15:22 +0000 [warn]: [default_loki_audit] failed to write post to https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit/loki/api/v1/push (429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5476' lines totaling '8216236' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased
)
2023-11-06 12:15:22 +0000 [warn]: [default_loki_audit] failed to flush the buffer. retry_times=1 next_retry_time=2023-11-06 12:15:25 +0000 chunk="60978f0ac44b83e5653df5bba8cdb2ec" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5476' lines totaling '8216236' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"
  2023-11-06 12:15:22 +0000 [warn]: suppressed same stacktrace
2023-11-06 12:15:26 +0000 [warn]: [default_loki_audit] failed to write post to https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit/loki/api/v1/push (429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5013' lines totaling '7268054' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased
)
2023-11-06 12:15:26 +0000 [warn]: [default_loki_audit] failed to flush the buffer. retry_times=2 next_retry_time=2023-11-06 12:15:30 +0000 chunk="60978f0b4841bef69a0b4c4e1861634f" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5013' lines totaling '7268054' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"
  2023-11-06 12:15:26 +0000 [warn]: suppressed same stacktrace
2023-11-06 12:15:26 +0000 [warn]: [default_loki_audit] failed to write post to https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit/loki/api/v1/push (429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5476' lines totaling '8216236' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased
)
2023-11-06 12:15:26 +0000 [warn]: [default_loki_audit] failed to flush the buffer. retry_times=2 next_retry_time=2023-11-06 12:15:30 +0000 chunk="60978f0ac44b83e5653df5bba8cdb2ec" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5476' lines totaling '8216236' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"
  2023-11-06 12:15:26 +0000 [warn]: suppressed same stacktrace
2023-11-06 12:15:31 +0000 [warn]: [default_loki_audit] failed to write post to https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit/loki/api/v1/push (429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5013' lines totaling '7268054' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased
)
2023-11-06 12:15:31 +0000 [warn]: [default_loki_audit] failed to flush the buffer. retry_times=3 next_retry_time=2023-11-06 12:15:40 +0000 chunk="60978f0b4841bef69a0b4c4e1861634f" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5013' lines totaling '7268054' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"
  2023-11-06 12:15:31 +0000 [warn]: suppressed same stacktrace
2023-11-06 12:15:31 +0000 [warn]: [default_loki_audit] failed to write post to https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit/loki/api/v1/push (429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5476' lines totaling '8216236' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased
)
2023-11-06 12:15:31 +0000 [warn]: [default_loki_audit] failed to flush the buffer. retry_times=3 next_retry_time=2023-11-06 12:15:40 +0000 chunk="60978f0ac44b83e5653df5bba8cdb2ec" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5476' lines totaling '8216236' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"
  2023-11-06 12:15:31 +0000 [warn]: suppressed same stacktrace
2023-11-06 12:15:41 +0000 [warn]: [default_loki_audit] failed to write post to https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit/loki/api/v1/push (429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5013' lines totaling '7268054' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased
)
2023-11-06 12:15:41 +0000 [warn]: [default_loki_audit] failed to flush the buffer. retry_times=4 next_retry_time=2023-11-06 12:15:56 +0000 chunk="60978f0b4841bef69a0b4c4e1861634f" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5013' lines totaling '7268054' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"
  2023-11-06 12:15:41 +0000 [warn]: suppressed same stacktrace
2023-11-06 12:15:41 +0000 [warn]: [default_loki_audit] failed to write post to https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit/loki/api/v1/push (429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5476' lines totaling '8216236' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased
)
2023-11-06 12:15:41 +0000 [warn]: [default_loki_audit] failed to flush the buffer. retry_times=4 next_retry_time=2023-11-06 12:15:57 +0000 chunk="60978f0ac44b83e5653df5bba8cdb2ec" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5476' lines totaling '8216236' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"
  2023-11-06 12:15:41 +0000 [warn]: suppressed same stacktrace
2023-11-06 12:15:58 +0000 [warn]: [default_loki_audit] failed to write post to https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit/loki/api/v1/push (429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5013' lines totaling '7268054' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased
)
2023-11-06 12:15:58 +0000 [warn]: [default_loki_audit] failed to flush the buffer. retry_times=5 next_retry_time=2023-11-06 12:16:31 +0000 chunk="60978f0b4841bef69a0b4c4e1861634f" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5013' lines totaling '7268054' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"
  2023-11-06 12:15:58 +0000 [warn]: suppressed same stacktrace
2023-11-06 12:15:58 +0000 [warn]: [default_loki_audit] failed to write post to https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit/loki/api/v1/push (429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5476' lines totaling '8216236' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased
)
2023-11-06 12:15:58 +0000 [warn]: [default_loki_audit] failed to flush the buffer. retry_times=5 next_retry_time=2023-11-06 12:16:33 +0000 chunk="60978f0ac44b83e5653df5bba8cdb2ec" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5476' lines totaling '8216236' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"
  2023-11-06 12:15:58 +0000 [warn]: suppressed same stacktrace
2023-11-06 12:16:33 +0000 [warn]: [default_loki_audit] failed to write post to https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit/loki/api/v1/push (429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5013' lines totaling '7268054' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased
)
2023-11-06 12:16:33 +0000 [warn]: [default_loki_audit] failed to flush the buffer. retry_times=6 next_retry_time=2023-11-06 12:17:32 +0000 chunk="60978f0b4841bef69a0b4c4e1861634f" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5013' lines totaling '7268054' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"
  2023-11-06 12:16:33 +0000 [warn]: suppressed same stacktrace
2023-11-06 12:16:34 +0000 [warn]: [default_loki_audit] failed to write post to https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit/loki/api/v1/push (429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5476' lines totaling '8216236' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased
)
2023-11-06 12:16:34 +0000 [warn]: [default_loki_audit] failed to flush the buffer. retry_times=6 next_retry_time=2023-11-06 12:17:37 +0000 chunk="60978f0ac44b83e5653df5bba8cdb2ec" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5476' lines totaling '8216236' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"
  2023-11-06 12:16:34 +0000 [warn]: suppressed same stacktrace
2023-11-06 12:17:37 +0000 [warn]: [default_loki_audit] failed to write post to https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit/loki/api/v1/push (429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5013' lines totaling '7268054' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased
)
2023-11-06 12:17:37 +0000 [warn]: [default_loki_audit] failed to flush the buffer. retry_times=7 next_retry_time=2023-11-06 12:18:41 +0000 chunk="60978f0b4841bef69a0b4c4e1861634f" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5013' lines totaling '7268054' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"
  2023-11-06 12:17:37 +0000 [warn]: suppressed same stacktrace
2023-11-06 12:17:38 +0000 [warn]: [default_loki_audit] failed to write post to https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit/loki/api/v1/push (429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5476' lines totaling '8216236' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased
)
2023-11-06 12:17:38 +0000 [warn]: [default_loki_audit] failed to flush the buffer. retry_times=7 next_retry_time=2023-11-06 12:18:35 +0000 chunk="60978f0ac44b83e5653df5bba8cdb2ec" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5476' lines totaling '8216236' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"
  2023-11-06 12:17:38 +0000 [warn]: suppressed same stacktrace
2023-11-06 12:18:35 +0000 [warn]: [default_loki_audit] failed to write post to https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit/loki/api/v1/push (429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5476' lines totaling '8216236' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased
)
2023-11-06 12:18:35 +0000 [warn]: [default_loki_audit] failed to flush the buffer. retry_times=8 next_retry_time=2023-11-06 12:19:31 +0000 chunk="60978f0ac44b83e5653df5bba8cdb2ec" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5476' lines totaling '8216236' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"
  2023-11-06 12:18:35 +0000 [warn]: suppressed same stacktrace
2023-11-06 12:19:31 +0000 [warn]: [default_loki_audit] failed to write post to https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit/loki/api/v1/push (429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5013' lines totaling '7268054' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased
)
2023-11-06 12:19:31 +0000 [warn]: [default_loki_audit] failed to flush the buffer. retry_times=9 next_retry_time=2023-11-06 12:20:28 +0000 chunk="60978f0b4841bef69a0b4c4e1861634f" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5013' lines totaling '7268054' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"
  2023-11-06 12:19:31 +0000 [warn]: suppressed same stacktrace
2023-11-06 12:19:31 +0000 [warn]: [default_loki_audit] failed to write post to https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit/loki/api/v1/push (429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5476' lines totaling '8216236' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased
)
2023-11-06 12:19:31 +0000 [warn]: [default_loki_audit] failed to flush the buffer. retry_times=9 next_retry_time=2023-11-06 12:20:37 +0000 chunk="60978f0ac44b83e5653df5bba8cdb2ec" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5476' lines totaling '8216236' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"
  2023-11-06 12:19:31 +0000 [warn]: suppressed same stacktrace
2023-11-06 12:20:38 +0000 [warn]: [default_loki_audit] failed to write post to https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit/loki/api/v1/push (429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5013' lines totaling '7268054' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased
)
2023-11-06 12:20:38 +0000 [warn]: [default_loki_audit] failed to flush the buffer. retry_times=10 next_retry_time=2023-11-06 12:21:36 +0000 chunk="60978f0b4841bef69a0b4c4e1861634f" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5013' lines totaling '7268054' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"
  2023-11-06 12:20:38 +0000 [warn]: suppressed same stacktrace
2023-11-06 12:20:38 +0000 [warn]: [default_loki_audit] failed to write post to https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit/loki/api/v1/push (429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5476' lines totaling '8216236' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased
)
2023-11-06 12:20:38 +0000 [warn]: [default_loki_audit] failed to flush the buffer. retry_times=10 next_retry_time=2023-11-06 12:21:32 +0000 chunk="60978f0ac44b83e5653df5bba8cdb2ec" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5476' lines totaling '8216236' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"
  2023-11-06 12:20:38 +0000 [warn]: suppressed same stacktrace
2023-11-06 12:21:32 +0000 [warn]: [default_loki_audit] failed to write post to https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit/loki/api/v1/push (429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5476' lines totaling '8216236' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased
)
2023-11-06 12:21:32 +0000 [warn]: [default_loki_audit] failed to flush the buffer. retry_times=11 next_retry_time=2023-11-06 12:22:25 +0000 chunk="60978f0ac44b83e5653df5bba8cdb2ec" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5476' lines totaling '8216236' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"
  2023-11-06 12:21:32 +0000 [warn]: suppressed same stacktrace
2023-11-06 12:22:26 +0000 [warn]: [default_loki_audit] failed to write post to https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit/loki/api/v1/push (429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5013' lines totaling '7268054' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased
)
2023-11-06 12:22:26 +0000 [warn]: [default_loki_audit] failed to flush the buffer. retry_times=12 next_retry_time=2023-11-06 12:23:21 +0000 chunk="60978f0b4841bef69a0b4c4e1861634f" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5013' lines totaling '7268054' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"
  2023-11-06 12:22:26 +0000 [warn]: suppressed same stacktrace
2023-11-06 12:22:26 +0000 [warn]: [default_loki_audit] failed to write post to https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit/loki/api/v1/push (429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5476' lines totaling '8216236' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased
)
2023-11-06 12:22:26 +0000 [warn]: [default_loki_audit] failed to flush the buffer. retry_times=12 next_retry_time=2023-11-06 12:23:22 +0000 chunk="60978f0ac44b83e5653df5bba8cdb2ec" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5476' lines totaling '8216236' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"
  2023-11-06 12:22:26 +0000 [warn]: suppressed same stacktrace
2023-11-06 12:23:22 +0000 [warn]: [default_loki_audit] failed to write post to https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit/loki/api/v1/push (429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5013' lines totaling '7268054' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased
)
2023-11-06 12:23:22 +0000 [warn]: [default_loki_audit] failed to flush the buffer. retry_times=13 next_retry_time=2023-11-06 12:24:21 +0000 chunk="60978f0b4841bef69a0b4c4e1861634f" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5013' lines totaling '7268054' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"
  2023-11-06 12:23:22 +0000 [warn]: suppressed same stacktrace
2023-11-06 12:23:22 +0000 [warn]: [default_loki_audit] failed to write post to https://lokistack-dev-gateway-http.openshift-logging.svc:8080/api/logs/v1/audit/loki/api/v1/push (429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5476' lines totaling '8216236' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased
)
2023-11-06 12:23:22 +0000 [warn]: [default_loki_audit] failed to flush the buffer. retry_times=13 next_retry_time=2023-11-06 12:24:29 +0000 chunk="60978f0ac44b83e5653df5bba8cdb2ec" error_class=Fluent::Plugin::LokiOutput::LogPostError error="429 Too Many Requests Ingestion rate limit exceeded for user audit (limit: 4194304 bytes/sec) while attempting to ingest '5476' lines totaling '8216236' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased\n"
  2023-11-06 12:23:22 +0000 [warn]: suppressed same stacktrace