Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-5981

logforward - forward audit log to syslog udp with vector error

XMLWordPrintable

    • False
    • None
    • False
    • NEW
    • NEW
    • Bug Fix

      Description of problem:

      Try to use logforwarder to forward audit log out of the cluster using vector with output as a syslog udp. but their have an alerting about `CollectorHighErrorRate`.

      Version-Release number of selected component (if applicable): 5.9.4, 5.9.5

      How reproducible:

      Steps to Reproduce:

      1. Install openshift logging
      2. Configure logforwarder with udp syslog as output and use audit as input following the below spec.
      spec:
  filters:
    - kubeAPIAudit:
        omitStages:
          - RequestReceived
        rules:
          - level: Metadata
      name: filter-audit-metadata
      type: kubeAPIAudit
  outputs:
    - name: syslog-baac
      type: syslog
      url: 'udp://172.26.164.36:514'
  pipelines:
    - filterRefs: []
      inputRefs:
        - audit
      name: audit-logs
      outputRefs:
        - syslog-baac

      Actual results:

      Log error at `collector` pod on the master node.

      2024-08-19T13:12:53.324517Z ERROR sink{component_kind="sink" component_id=output_syslog_baac component_type=socket}: vector::internal_events::socket: Internal log [Error sending data.] has been suppressed 9 times.
2024-08-19T13:12:53.324536Z ERROR sink{component_kind="sink" component_id=output_syslog_baac component_type=socket}: vector::internal_events::socket: Error sending data. error=Connection refused (os error 111) error_code="socket_send" error_type="writer_failed" stage="sending" mode=udp internal_log_rate_limit=true
2024-08-19T13:12:53.324564Z ERROR sink{component_kind="sink" component_id=output_syslog_baac component_type=socket}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] has been suppressed 9 times.
2024-08-19T13:12:53.324566Z ERROR sink{component_kind="sink" component_id=output_syslog_baac component_type=socket}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Error sending data." internal_log_rate_limit=true
2024-08-19T13:12:53.637819Z ERROR sink{component_kind="sink" component_id=output_syslog_baac component_type=socket}: vector::internal_events::socket: Internal log [Error sending data.] is being suppressed to avoid flooding.
2024-08-19T13:12:53.638079Z ERROR sink{component_kind="sink" component_id=output_syslog_baac component_type=socket}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] is being suppressed to avoid flooding.
2024-08-19T13:13:03.824693Z ERROR sink{component_kind="sink" component_id=output_syslog_baac component_type=socket}: vector::internal_events::socket: Internal log [Error sending data.] has been suppressed 7 times.
2024-08-19T13:13:03.824712Z ERROR sink{component_kind="sink" component_id=output_syslog_baac component_type=socket}: vector::internal_events::socket: Error sending data. error=Connection refused (os error 111) error_code="socket_send" error_type="writer_failed" stage="sending" mode=udp internal_log_rate_limit=true
2024-08-19T13:13:03.824729Z ERROR sink{component_kind="sink" component_id=output_syslog_baac component_type=socket}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] has been suppressed 7 times.
2024-08-19T13:13:03.824731Z ERROR sink{component_kind="sink" component_id=output_syslog_baac component_type=socket}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Error sending data." internal_log_rate_limit=true
2024-08-19T13:13:04.859142Z ERROR sink{component_kind="sink" component_id=output_syslog_baac component_type=socket}: vector::internal_events::socket: Internal log [Error sending data.] is being suppressed to avoid flooding.
2024-08-19T13:13:04.859174Z ERROR sink{component_kind="sink" component_id=output_syslog_baac component_type=socket}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] is being suppressed to avoid flooding.
2024-08-19T13:13:14.104081Z ERROR sink{component_kind="sink" component_id=output_syslog_baac component_type=socket}: vector::internal_events::socket: Internal log [Error sending data.] has been suppressed 7 times.
2024-08-19T13:13:14.104100Z ERROR sink{component_kind="sink" component_id=output_syslog_baac component_type=socket}: vector::internal_events::socket: Error sending data. error=Connection refused (os error 111) error_code="socket_send" error_type="writer_failed" stage="sending" mode=udp internal_log_rate_limit=true
2024-08-19T13:13:14.104128Z ERROR sink{component_kind="sink" component_id=output_syslog_baac component_type=socket}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] has been suppressed 7 times.
2024-08-19T13:13:14.104130Z ERROR sink{component_kind="sink" component_id=output_syslog_baac component_type=socket}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Error sending data." internal_log_rate_limit=true
2024-08-19T13:13:15.432956Z ERROR sink{component_kind="sink" component_id=output_syslog_baac component_type=socket}: vector::internal_events::socket: Internal log [Error sending data.] is being suppressed to avoid flooding.
2024-08-19T13:13:15.432994Z ERROR sink{component_kind="sink" component_id=output_syslog_baac component_type=socket}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] is being suppressed to avoid flooding.
2024-08-19T13:13:25.240255Z ERROR sink{component_kind="sink" component_id=output_syslog_baac component_type=socket}: vector::internal_events::socket: Internal log [Error sending data.] has been suppressed 5 times.

      Expected results:

      no error for forwarding audit log with syslog udp.

      Additional info:

      We workaround with using `api server filter`. The ERROR message was fix but still have a WARN message following.

      `2024-08-19T13:11:28.229937Z WARN file_source::buffer: Internal log [Found line that exceeds max_line_bytes; discarding.] is being suppressed to avoid flooding.`

      Filter api server at logforwarder.

       

      spec:
  filters:
    - kubeAPIAudit:
        omitStages:
          - RequestReceived
        rules:
          - level: Metadata
      name: filter-audit-metadata
      type: kubeAPIAudit
  outputs:
    - name: syslog-baac
      type: syslog
      url: 'udp://172.26.164.36:514'
  pipelines:
    - filterRefs:
        - filter-audit-metadata
      inputRefs:
        - audit
      name: audit-logs
      outputRefs:
        - syslog-baac

       

       

       

        1. Screenshot 2567-08-19 at 20.15.59.png
          Screenshot 2567-08-19 at 20.15.59.png
          256 kB

              Unassigned Unassigned
              rh-ee-swongpai Surote Wongpaiboon
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: