Uploaded image for project: 'MicroShift'
  1. MicroShift
  2. USHIFT-5897

Otel-collector's logging schema changed leading to failing observability test on centos9

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Undefined Undefined
    • None
    • None
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • 1
    • None
    • None
    • uShift Sprint 273
    • None
    • None
    • None

      In 4.19 and 4.20 nightly bootc jobs there are observability failures for centos9.
      Comparing yesterdays run with todays, it looks like the logging schema changed, perhaps the version was bumped. If so, I expect it to fail soon-ish for RHEL9 as well.

      Old log:

      Jul 07 02:52:51 cos9-src-optional-bootc-host1 opentelemetry-collector[6045]: 2025-07-07T02:52:51.373Z        info        LogsExporter        {"kind": "exporter", "data_type": "logs", "name": "debug", "resource logs": 2, "log records": 15}
      

      New log:

      Jul 08 03:12:24 cos9-src-optional-bootc-host1 opentelemetry-collector[8880]: 2025-07-08T03:12:24.874Z        info        Logs        {"resource": {}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "resource logs": 1, "log records": 1}

      Current el9 log:

      Jul 07 18:43:26 el96-src-optional-bootc-host1 opentelemetry-collector[8022]: 2025-07-07T18:43:26.687Z        info        LogsExporter        {"kind": "exporter", "data_type": "logs", "name": "debug", "resource logs": 3, "log records": 3}

      We have the journal patterns hardcoded: https://github.com/openshift/microshift/blob/0d1d7d3410befa99dcce2dd16c35afea79a3892b/test/suites/optional/observability.robot#L60-L61
      But we also check Loki & Prometheus. So perhaps we could drop the debug exported and stop checking the journal?

              pmatusza@redhat.com Patryk Matuszak
              pmatusza@redhat.com Patryk Matuszak
              None
              None
              None
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: