Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-4672

Timestamp warning in Vector logs sending logs to Splunk

XMLWordPrintable

    • False
    • None
    • False
    • NEW
    • VERIFIED
    • Hide
      Before this update, not well formatted timestamp fields in audit log record leads to WARN messages in CLO logs.
      With this update, added remap transformation, that should ensure that the timestamp field will be well formatted.
      Show
      Before this update, not well formatted timestamp fields in audit log record leads to WARN messages in CLO logs. With this update, added remap transformation, that should ensure that the timestamp field will be well formatted.
    • Bug Fix
    • Low
    • Hide

      1) Deploy RHOL 5.7.6 version with Vector as collector.

      2) Deploy a CLF instance with Splunk as output and forwarding audit logs:

       

      spec:
  outputs:
  - name: splunk
    splunk: {}
    type: splunk
    url: http://test:8088
  pipelines:
  - inputRefs:
    - application
    - infrastructure
    - audit
    name: forwarder
    outputRefs:
    - splunk

      3) Check Vector logs:

       

      WARN sink{component_kind="sink" component_id=splunk component_type=splunk_hec_logs component_name=splunk}: vector::internal_events::splunk_hec::sink: Timestamp was an unexpected type. Deferring to Splunk to set the timestamp. invalid_type="string" internal_log_rate_limit=true
WARN sink{component_kind="sink" component_id=splunk component_type=splunk_hec_logs component_name=splunk}: vector::internal_events::splunk_hec::sink: Internal log [Timestamp was an unexpected type. Deferring to Splunk to set the timestamp.] is being rate limited.

       

      4) Not sure if it is related, checking Vector configuration when enabling audit logs, we can observe

      [transforms.host_audit_logs]
type = "remap"
inputs = ["raw_host_audit_logs"]
source = '''
.openshift.cluster_id = "${OPENSHIFT_CLUSTER_ID:-}"
.tag = ".linux-audit.log"
match1 = parse_regex(.message, r'type=(?P<type>[^ ]+)') ?? {}
envelop = {}
envelop |= {"type": match1.type}
match2, err = parse_regex(.message, r'msg=audit((?P<ts_record>[^ ]+)):')
if err == null {
sp, err = split(match2.ts_record,":")
if err == null && length(sp) == 2 {
ts = parse_timestamp(sp[0],"%s.%3f") ?? ""
 
 

       

       

      Show
      1) Deploy RHOL 5.7.6 version with Vector as collector. 2) Deploy a CLF instance with Splunk as output and forwarding audit logs:   spec:   outputs:   - name: splunk     splunk: {}     type: splunk     url: http: //test:8088   pipelines:   - inputRefs:     - application     - infrastructure     - audit     name: forwarder     outputRefs:     - splunk 3) Check Vector logs:   WARN sink{component_kind= "sink" component_id=splunk component_type=splunk_hec_logs component_name=splunk}: vector::internal_events::splunk_hec::sink: Timestamp was an unexpected type. Deferring to Splunk to set the timestamp. invalid_type= "string" internal_log_rate_limit= true WARN sink{component_kind= "sink" component_id=splunk component_type=splunk_hec_logs component_name=splunk}: vector::internal_events::splunk_hec::sink: Internal log [Timestamp was an unexpected type. Deferring to Splunk to set the timestamp.] is being rate limited.   4) Not sure if it is related, checking Vector configuration when enabling audit logs, we can observe [transforms.host_audit_logs] type = "remap" inputs = [ "raw_host_audit_logs" ] source = ''' .openshift.cluster_id = "${OPENSHIFT_CLUSTER_ID:-}" .tag = ".linux-audit.log" match1 = parse_regex(.message, r 'type=(?P<type>[^ ]+)' ) ?? {} envelop = {} envelop |= { "type" : match1.type} match2, err = parse_regex(.message, r 'msg=audit((?P<ts_record>[^ ]+)):' ) if err == null { sp, err = split(match2.ts_record, ":" ) if err == null && length(sp) == 2 { ts = parse_timestamp(sp[0], "%s.%3f" ) ?? ""        
    • Log Collection - Sprint 243, Log Collection - Sprint 244, Log Collection - Sprint 245, Log Collection - Sprint 246, Log Collection - Sprint 247, Log Collection - Sprint 252
    • Low

      Description of problem:

      After deployed a CLF instance with Splunk as output,  we can observe some warning logs in collector pods about the Timestamp this is different from (LOG-4413). I did some tests and I only see this message when audit logs are forwarded to Splunk.

       

      WARN sink{component_kind="sink" component_id=splunk component_type=splunk_hec_logs component_name=splunk}: vector::internal_events::splunk_hec::sink: Timestamp was an unexpected type. Deferring to Splunk to set the timestamp. invalid_type="string" internal_log_rate_limit=true
WARN sink{component_kind="sink" component_id=splunk component_type=splunk_hec_logs component_name=splunk}: vector::internal_events::splunk_hec::sink: Internal log [Timestamp was an unexpected type. Deferring to Splunk to set the timestamp.] is being rate limited.

       

      Version-Release number of selected component (if applicable):

      CLO 5.7.6

       

      Actual results:

      Warning  logs in Vector when sending logs to external third-party Splunk

      Expected results:

      No warning  logs in Vector when sending logs to external third-party Splunk

      Additional info:

      Doing some test, it seems related with Vector timestamp configuration of audit logs

            vparfono Vitalii Parfonov
            acandelp Adrian Candel
            Kabir Bharti Kabir Bharti
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: