Uploaded image for project: 'Subscription Watch'
  1. Subscription Watch
  2. SWATCH-2909

Propagate the Opentelemetry headers when consuming/producing messages with Kafka Streams

XMLWordPrintable

    • Icon: Task Task
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • None
    • CI/Infrastructure
    • None

      At the moment, we're using Kafka Streams when aggregating billable usages.
      The workflow is:
      1.- The tally service produces messages containing a number of tally snapshots into the topic "platform.rhsm-subscriptions.billable-usage":

      {"severity":"INFO","logger":"org.candlepin.subscriptions.tally.SnapshotSummaryProducer","thread":"rhsm-subscriptions-task-processor-0-C-1","message":"2024-09-09T23:10:24.794+0000 [thread=rhsm-subscriptions-task-processor-0-C-1] [level=INFO] [category=org.candlepin.subscriptions.tally.SnapshotSummaryProducer]  - Produced 1 TallySummary messages\n","properties":{"trace_id":"4beae9a48365393b8ec4decc4c2a934f","trace_flags":"01","span_id":"50f747eb8e947021"}}
      

      2.- Then, these messages are consumed by Kafka Streams in the billable usage service:

      {"severity":"INFO","message":"2024-09-09 23:00:21,202 INFO  [org.can.sub.bil.usa.BillableUsageAggregate] (platform.rhsm-subscriptions.swatch-billable-usage-aggregator-2122ca6f-eca9-4a24-97a5-7604e463c429-StreamThread-1) Adding billableUsage: org.candlepin.subscriptions.billable.usage.BillableUsage@2e99447f[uuid=378c75c2-4ffb-474c-beda-52020322b77d,orgId=11618340,tallyId=839000f8-83bf-4cfe-a5f8-2b2e83c95a92,billingProvider=aws,billingAccountId=204833053756,snapshotDate=2024-09-06T12:00:55.554174Z,productId=rhel-for-x86-els-payg-addon,sla=Premium,usage=Production,status=failed,errorCode=<null>,billedOn=<null>,metricId=vCPUs,value=2822.0,billingFactor=<null>,vendorProductCode=<null>,hardwareMeasurementType=PHYSICAL,currentTotal=<null>] to aggregate with aggregateId: eada50b6-b4da-45c3-b298-40b42458679c, totalValue:164506.0, remittanceUuids:... and windowTimestamp: 2024-09-09T23:00Z\n"}
      

      Note that the "trace_id" header from the first message is missed / not propagrated.

      When using Quarkus Smallrye Messaging Kafka consumers, this works out of the box, but using Kafka Streams, this seems not to be supported.

      Why is this very important? Propagate the "trace_id" header is a must to see how the workflow went from end to end (from the tally service to the final swatch producer aws/azure service).

       

      Done

      • We would like to have a unit test that validates that the headers are being set appropriately

              Unassigned Unassigned
              jcarvaja@redhat.com Jose Carvajal Hilario
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: