Uploaded image for project: 'Subscription Watch'
  1. Subscription Watch
  2. SWATCH-3974 Evaluate post-stage test failures to prepare for prod release
  3. SWATCH-4051

Investigate "test_rosa_get_account_and_product_tally_report" Failures

XMLWordPrintable

    • Icon: Sub-task Sub-task
    • Resolution: Done
    • Icon: Blocker Blocker
    • None
    • None
    • swatch-tally
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • subs-swatch-thunder
    • Swatch Thunder Sprint 2

      AssertionError: Recorded tally: 184 does not match expected tally: 192

      self = <iqe_rhsm_subscriptions.tests.longrun.test_verify_rosa.TestRosaLongrun object at 0x7ff22b20ef90>
      application = <iqe.base.application.Application object at 0x7ff22b2ada30>
      longrun_rosa_cluster_setup = {'billing_provider': 'aws', 'display_name': 'automation', 'instance_id': 'fake_cluster', 'metric_data': {'cores': 2976, 'instance-hours': 744}, ...}
      
          def test_rosa_get_account_and_product_tally_report(
              self, application, longrun_rosa_cluster_setup
          ):
              """Verify the tally report for the ROSA Longrun cluster.
              metadata:
                  assignee: tmcknigh
                  negative: false
                  importance: critical
                  level: system
                  requirements: longrun
                  test_steps:
                      1. Call Tally Report /v1/tally/products/{product_id}/{metric_id} endpoint
                      2. For each metric, evaluate data matches the expected value.
                  expected_results:
                      1. Tally data[0] (yesterday's usage) should match the expected value
              """
              for metric, expected_value in self.expected_metrics_daily.items():
                  tally = application.rhsm_subscriptions.get_tally_report(
                      product_id=longrun_rosa_cluster_setup["product_id"],
                      metric_id=metric,
                      beginning=self.beginning_of_day,
                      ending=self.end_of_day,
                      granularity="Daily",
                  )
                  logging.info(f"{metric} Tally Report:\n{tally}")
          
      >           assert tally.data[0].value == expected_value, (
                      f"Recorded tally: {tally.data[0].value} does not"
                      f" match expected tally: {expected_value}"
                  )
      E           AssertionError: Recorded tally: 184 does not match expected tally: 192
      E           assert 184 == 192
      E            +  where 184 = {'date': datetime.datetime(2025, 10, 5, 0, 0, tzinfo=tzlocal()),\n 'has_data': True,\n 'value': 184}.value
      
      /iqe_venv/lib/python3.12/site-packages/iqe_rhsm_subscriptions/tests/longrun/test_verify_rosa.py:201: AssertionError 

      We think this is related to a missing Event record (see https://redhat-internal.slack.com/archives/C09BCH0GRT7/p1760098787929349?thread_ts=1760049585.701799&cid=C09BCH0GRT7) for the 2025-10-05T20:00:00Z timestamp.

      There is NOT a missing datapoint in Prometheus.
      There's evidence in Splunk that swatch-tally did receive a message for that timestamp/instance.

      https://rhcorporate.splunkcloud.com/en-US/app/search/search?q=search%20index%3Drh_rhsm%20namespace%3Drhsm-stage%203b8c710c-4c5f-4e51-afae-7209e24a0aed%20host%3D%22swatch-tally-service-*%22%7C%20spath%20logger%20%7C%20search%20logger%3D%22org.candlepin.subscriptions.event.EventController%22&display.page.search.mode=smart&dispatch.sample_ratio=1&workload_pool=standard_perf&earliest=1759622400&latest=1759795200&sid=1760107072.55783_01DB40DB-8A11-4AEA-9C78-B454D70430F8

              mstead@redhat.com Michael Stead
              rh-ee-erthomps Eric Thompson
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: