Depending on the time of day, payload tracker can take over an hour to process messages. As a concrete example, see request 820fb25ccfb248d8b51434696158e8a1:

curl -sSu "${username}:${password}" "https://console.redhat.com/api/ingress/v1/track/820fb25ccfb248d8b51434696158e8a1?verbosity=2" | jq '.data[] | select(.service=="inventory-mq-service") | [.status, .date, .created_at]'

[
  "success",
  "2023-08-01T12:18:47.870425Z",
  "2023-08-01T13:34:17.986358Z"
]
[
  "processing_success",
  "2023-08-01T12:18:47.864413Z",
  "2023-08-01T13:34:17.975082Z"
]
[
  "processing",
  "2023-08-01T12:18:47.762709Z",
  "2023-08-01T12:18:47.782885Z"
]
[
  "received",
  "2023-08-01T12:18:47.762615Z",
  "2023-08-01T12:18:47.77707Z"
]

For each message, date indicates when inventory-mq-service enqueues the message with kafka, and created_at indicates when payload-tracker dequeues that message from kafka. The "received" and "processing" messages were queued for less than a second, but the "procesing_success" and "success" messages were queued for ~74 minutes.

A delay of this magnitude makes payload tracker unreliable for either manual debugging for for automated testing. This issue negatively impacts automated Insights Remediations tests, as they can't rely on payload tracker to understand whether a payload has been processed or not.

The payload tracker kafka lag graph indicates that payload tracker's performance is compromised ~50% of the time.