-
Bug
-
Resolution: Done
-
Normal
-
Logging 6.3.0
-
Incidents & Support
-
False
-
-
False
-
NEW
-
VERIFIED
-
-
Bug Fix
-
-
-
Log Collection - Sprint 272, Log Collection - Sprint 273, Logging - Sprint 274, Logging - Sprint 275, Logging - Sprint 276
- Timestamp warning in collector pods while forwarding logs to Splunk.
- Collector pods are streaming the following warning message:
$ oc logs collector-6z8wg 2025-06-15T20:34:13.514496Z WARN sink{component_kind="sink" component_id=output_splunk_aosqe component_type=splunk_hec_logs}: vector::internal_events::splunk_hec::sink: Timestamp was not found. Deferring to Splunk to set the timestamp. internal_log_rate_limit=true 2025-06-15T20:34:14.552045Z WARN sink{component_kind="sink" component_id=output_splunk_aosqe component_type=splunk_hec_logs}: vector::internal_events::splunk_hec::sink: Internal log [Timestamp was not found. Deferring to Splunk to set the timestamp.] is being suppressed to avoid flooding
- OpenShift Logging Operator version:
$ oc get csv -n openshift-logging NAME DISPLAY VERSION REPLACES PHASE cluster-logging.v6.3.0 Red Hat OpenShift Logging 6.3.0 Succeeded
Steps to Reproduce:
1. Install Red Hat OpenShift Logging operator v6.3.0
2. Create a ServiceAccount, Bind the Cluster Role, and Add additional roles to the collector service account:
$ oc -n openshift-logging create serviceaccount collector $ oc -n openshift-logging adm policy add-cluster-role-to-user logging-collector-logs-writer -z collector $ oc -n openshift-logging adm policy add-cluster-role-to-user collect-application-logs -z collector
3. Create a ClusterLogForwarder configuration with the `payloadKey` parameter:
apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
name: collector
namespace: openshift-logging
spec:
managementState: Managed
outputs:
- name: splunk-logstore
splunk:
authentication:
token:
key: hecToken
secretName: splunk-secret
payloadKey: .kubernetes.labels.run <<=======
url: 'https://splunk-default-service.splunk-aosqe.svc:8088'
tls:
ca:
key: ca-bundle.crt
secretName: splunk-secret
type: splunk
pipelines:
- inputRefs:
- application
name: forward-log-splunk
outputRefs:
- splunk-logstore
serviceAccount:
name: collector
4. Check the collector pods status.
Current Results:
- Collector pods streaming following warning message:
$ oc logs collector-6z8wg 2025-06-15T20:34:13.514496Z WARN sink{component_kind="sink" component_id=output_splunk_aosqe component_type=splunk_hec_logs}: vector::internal_events::splunk_hec::sink: Timestamp was not found. Deferring to Splunk to set the timestamp. internal_log_rate_limit=true 2025-06-15T20:34:14.552045Z WARN sink{component_kind="sink" component_id=output_splunk_aosqe component_type=splunk_hec_logs}: vector::internal_events::splunk_hec::sink: Internal log [Timestamp was not found. Deferring to Splunk to set the timestamp.] is being suppressed to avoid flooding
Expected results:
- Timestamp warning logs should not be present in the collector pods while forwarding logs to an external Splunk destination.