-
Bug
-
Resolution: Duplicate
-
Critical
-
None
-
None
-
None
-
False
-
-
False
-
subs-swatch-lightning
-
-
PO Summary: The Tally service is flooding the system with redundant data that the Utilization service doesn't need, causing high traffic and memory issues that could lead to a service crash.
The swatch-utilization service is receiving a very large number of utilization summaries containing metrics that are not supported for the given product. This is not causing functional issues (the metrics are correctly ignored), but it generates unnecessary Kafka traffic, processing overhead, and was a major contributor to the memory leak described in the related Splunk HEC ticket.
In a single production pod's log file (86K lines, covering roughly 1.5 hours), there were 29,319 unsupported metricId warnings.
Example log lines:
WARN Received utilization summary with unsupported metricId 'INSTANCE_HOURS' in product 'RHEL for x86' WARN Received utilization summary with unsupported metricId 'CORES' in product 'RHEL for x86' WARN Received utilization summary with unsupported metricId 'INSTANCE_HOURS' in product 'Satellite Server' WARN Received utilization summary with unsupported metricId 'CORES' in product 'Satellite Server' WARN Received utilization summary with unsupported metricId 'INSTANCE_HOURS' in product 'OpenShift Container Platform'
The pattern is clear: the tally service (swatch-tally) is sending utilization summaries that include ALL metrics for a product (SOCKETS, CORES, INSTANCE_HOURS), but swatch-utilization only supports a subset of those metrics per product (e.g., only SOCKETS for "RHEL for x86"). The question is whether this is by design (tally sends everything, utilization filters) or whether tally should be smarter about which metrics it includes in the summary.
Acceptance Criteria
- Clarify why we're receiving too unsupported metrics and create a remedation plan to address this issue (in JIRA tickets)
- relates to
-
SWATCH-2634 Enforce snapshots only get created for relevant product_tag/metric_id/nightlyvshourly combinations
-
- Backlog
-