-
Bug
-
Resolution: Unresolved
-
Blocker
-
Logging 6.5.0
-
None
Description of problem:
Not all systems follow distributed tracing standards (like OpenTelemetry or W3C). Other systems( for example, Zipkin,AWS X-Ray, and SkyWalking) use different rules for ID lengths and formats. The LokiStack and OpenTelemetry Collector are rejecting log entries if the trace_id, span_id, and trace_flags do not comply with W3C.
Because the trace_id, span_id, trace_flags are from Log Entry, the value can be any word. we should either support all formats or strips the labels once the values do not follow distributed tracing standards.
when trace_flags is customized format in log entry
{"timestamp": "02/Mar/2026:02:54:50 +0000","id":1, "body":"traceFormatType1 ", "trace_id":"acs-req-00001", "span_id":"acs-req-00001-0001", "trace_flags":"xx", "format": "json" }
oc logs logging-loki-distributor-d488f5b88-f22vm
level=error ts=2026-03-02T02:50:27.314451473Z caller=manager.go:49 component=distributor path=write msg="write operation failed" details="couldn't parse push request: ReadUint32: strconv.ParseUint: parsing \"xx\": invalid syntax, error found in #10 byte of ...|lags\":\"xx\",\"observed|..., bigger context ...|s\\\":\\\"xx\\\", \\\"format\\\": \\\"json\\\" }\"},\"flags\":\"xx\",\"observedTimeUnixNano\":\"1772419826379519232\",\"sev|..." org_id=application
when trace_id is customized format in log entry
{"timestamp": "02/Mar/2026:02:54:50 +0000","id":1, "body":"traceFormatType1 ", "trace_id":"acs-req-00001", "span_id":"acs-req-00001-0001", "trace_flags":"01", "format": "json" }
oc log logging-loki-distributor-d488f5b88-rkppk
level=error ts=2026-03-02T03:43:00.648085476Z caller=manager.go:49 component=distributor path=write msg="write operation failed" details="couldn't parse push request: ID.UnmarshalJSONIter: length mismatch, error found in #10 byte of ...|0001-0001\",\"timeUnix|..., bigger context ...|everityText\":\"trace\",\"spanId\":\"acs-req-00001-0001\",\"timeUnixNano\":\"1772422979362446607\",\"traceId\":\"a|..." org_id=application
when span_id is customized format in log entry
{"timestamp": "02/Mar/2026:02:54:50 +0000","id":1, "body":"traceFormatType1 ", "trace_id":"a1c53496150e304c26bdf3187cad8669", "span_id":"acs-req-00001-0001", "trace_flags":"01", "format": "json" }
oc log logging-loki-distributor-d488f5b88-rkppk
level=error ts=2026-03-02T03:54:40.515367147Z caller=manager.go:49 component=distributor path=write msg="write operation failed" details="couldn't parse push request: ID.UnmarshalJSONIter: length mismatch, error found in #10 byte of ...|0001-0001\",\"timeUnix|..., bigger context ...|everityText\":\"trace\",\"spanId\":\"acs-req-00001-0001\",\"timeUnixNano\":\"1772423679312831345\",\"traceId\":\"a|..." org_id=application
Version-Release number of selected component (if applicable):
6.5
How reproducible:
Always
Steps to Reproduce:
- generate logs using non-standard format
{"timestamp": "02/Mar/2026:02:54:50 +0000","id":1, "body":"traceFormatType1 ", "trace_id":"acs-req-00001", "span_id":"acs-req-00001-0001", "trace_flags":"xx", "format": "json" }
{"timestamp": "02/Mar/2026:02:54:51 +0000","id":1, "body":"traceFormatType1 ", "trace_id":"acs-req-00001", "span_id":"acs-req-00001-0001", "trace_flags":"01", "format": "json" }
{"timestamp": "02/Mar/2026:02:54:52 +0000","id":1, "body":"traceFormatType1 ", "trace_id":"a1c53496150e304c26bdf3187cad8669", "span_id":"acs-req-00001-0001", "trace_flags":"01", "format": "json" }
- send logs to lokistack or Otel Collector
- ...
Actual results:
logs are rejected by LokiStack.
Expected results:
The logs can be received by LokiStack.
The Collector strips invalid trace IDs, span IDs, or flags. Conversely, Loki accepts both standard and non-standard trace formats
