Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-7311

Timestamp was not found warning in collector pods while forwarding logs to Splunk

XMLWordPrintable

    • Incidents & Support
    • False
    • Hide

      None

      Show
      None
    • False
    • NEW
    • VERIFIED
    • Hide
      Before this update, the log collector was configured to find the event timestamp at the root of the log record. When forwarding logs to Splunk with a payloadKey specified, this caused the @timestamp field to be missed inside the payload, leading to "Timestamp was not found" warnings in collector pods. With this update, the collector now uses an internal timestamp field that is not affected by the payloadKey setting. This resolves the warnings and ensures the correct timestamp is used when forwarding logs to Splunk.
      Show
      Before this update, the log collector was configured to find the event timestamp at the root of the log record. When forwarding logs to Splunk with a payloadKey specified, this caused the @timestamp field to be missed inside the payload, leading to "Timestamp was not found" warnings in collector pods. With this update, the collector now uses an internal timestamp field that is not affected by the payloadKey setting. This resolves the warnings and ensures the correct timestamp is used when forwarding logs to Splunk.
    • Bug Fix
    • Log Collection - Sprint 272, Log Collection - Sprint 273, Logging - Sprint 274, Logging - Sprint 275, Logging - Sprint 276

      • Timestamp warning in collector pods while forwarding logs to Splunk.
      • Collector pods are streaming the following warning message:
        $ oc logs collector-6z8wg
        
        2025-06-15T20:34:13.514496Z  WARN sink{component_kind="sink" component_id=output_splunk_aosqe component_type=splunk_hec_logs}: vector::internal_events::splunk_hec::sink: Timestamp was not found. Deferring to Splunk to set the timestamp. internal_log_rate_limit=true
        
        2025-06-15T20:34:14.552045Z  WARN sink{component_kind="sink" component_id=output_splunk_aosqe component_type=splunk_hec_logs}: vector::internal_events::splunk_hec::sink: Internal log [Timestamp was not found. Deferring to Splunk to set the timestamp.] is being suppressed to avoid flooding
      • OpenShift Logging Operator version:
        $ oc get csv -n openshift-logging
        
        NAME                                    DISPLAY                          VERSION   REPLACES                                PHASE
        cluster-logging.v6.3.0                  Red Hat OpenShift Logging        6.3.0                                             Succeeded

      Steps to Reproduce:

      1. Install Red Hat OpenShift Logging operator v6.3.0
      2. Create a ServiceAccount, Bind the Cluster Role, and Add additional roles to the collector service account:

      $ oc -n openshift-logging create serviceaccount collector 
      $ oc -n openshift-logging adm policy add-cluster-role-to-user logging-collector-logs-writer -z collector
      $ oc -n openshift-logging adm policy add-cluster-role-to-user collect-application-logs -z collector 

      3. Create a ClusterLogForwarder configuration with the `payloadKey` parameter:

      apiVersion: observability.openshift.io/v1
      kind: ClusterLogForwarder
      metadata:
        name: collector
        namespace: openshift-logging
      spec:
        managementState: Managed
        outputs:
          - name: splunk-logstore
            splunk:
              authentication:
                token:
                  key: hecToken
                  secretName: splunk-secret
              payloadKey: .kubernetes.labels.run                            <<=======
              url: 'https://splunk-default-service.splunk-aosqe.svc:8088'
            tls:
              ca:
                key: ca-bundle.crt
                secretName: splunk-secret
            type: splunk
        pipelines:
          - inputRefs:
              - application
            name: forward-log-splunk
            outputRefs:
              - splunk-logstore
        serviceAccount:
          name: collector 

      4. Check the collector pods status.

      Current Results:

      • Collector pods streaming following warning message:
        $ oc logs collector-6z8wg
        
        2025-06-15T20:34:13.514496Z  WARN sink{component_kind="sink" component_id=output_splunk_aosqe component_type=splunk_hec_logs}: vector::internal_events::splunk_hec::sink: Timestamp was not found. Deferring to Splunk to set the timestamp. internal_log_rate_limit=true 
        
        2025-06-15T20:34:14.552045Z  WARN sink{component_kind="sink" component_id=output_splunk_aosqe component_type=splunk_hec_logs}: vector::internal_events::splunk_hec::sink: Internal log [Timestamp was not found. Deferring to Splunk to set the timestamp.] is being suppressed to avoid flooding 

        Expected results:

      • Timestamp warning logs should not be present in the collector pods while forwarding logs to an external Splunk destination.

       

              vparfono Vitalii Parfonov
              rhn-support-pripatil Prithviraj Patil
              Kabir Bharti Kabir Bharti
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: