Uploaded image for project: 'Subscription Watch'
  1. Subscription Watch
  2. SWATCH-3771

Fix flaky local test test_hbi_physical_x86_host_events

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Undefined Undefined
    • None
    • None
    • None
    • False
    • Hide

      None

      Show
      None
    • False

      Running locally test_hbi_created_updated_events.py highlights a flaky behavior of the test test_hbi_physical_x86_host_events.

      Investigate and fix the the flakiness.

      The flakiness could be related to how SwatchMetricsHbiTestHelper consumes events from local Kafka broker (maybe timeout issue).

      To try reproducing it:

      • Compose the cluster for local testing inside rhsm-subscriptions 
        podman compose up -d
      • Run inside rhsm-subscriptions
        ./mvnw install -Prun-migration 
      • Start swatch-metric-hbi inside rhsm-subscriptions
        make swatch-metrics-hbi 
      • Setup IQE to run locally (virtual env+ setup for local testing)
      • Run the test
        iqe tests plugin rhsm_subscriptions -k test_hbi_physical_x86_host_events 

      Failed run logs:

      ___________________________________________________________________________________________________________________________________ test_hbi_physical_x86_host_events[created-INSTANCE_CREATED] ___________________________________________________________________________________________________________________________________helpers = <iqe_rhsm_subscriptions.tests.component.swatch_metrics_hbi.helpers.SwatchMetricsHbiTestHelper object at 0x7f99b9f817f0>, hbi_event_type = 'created', swatch_event_type = 'INSTANCE_CREATED'    @pytest.mark.ephemeral
          @pytest.mark.ephemeral_only
          @pytest.mark.usefixtures("enable_swatch_event_sending")
          @pytest.mark.parametrize(
              "hbi_event_type,swatch_event_type",
              [("created", "INSTANCE_CREATED"), ("updated", "INSTANCE_UPDATED")],
          )
          def test_hbi_physical_x86_host_events(helpers, hbi_event_type, swatch_event_type):
              """Verify service accepts HBI Create/Update events for a physical x86 host
                 and produce the expected Swatch Event messages.
              metadata:
                  assignee: mstead
                  requirements: swatch_metrics_hbi
                  importance: high
                  test_steps:
                      1. Toggle feature flag to allow service to emit swatch events.
                      2. Send a created/updated message to the HBI event topic to simulate that a host was created in HBI.
                  expected_results:
                      1. The swatch-metrics-hbi service will ingest the event and should emit a Swatch Event message containing the
                      measurements for the host represented by the HBI event. NOTE: We expect the same result regardless of whether
                      the HBI event was host created OR host updated.
              """
              timestamp = today()
              expected_timestamp = datetime_to_iso8601_format(timestamp)
          
              hbi_event = helpers.hbi_events.physical_rhel_x86_host_event(
                  expected_timestamp, event_type=hbi_event_type, cores=2, sockets=2
              )
          
              swatch_event_dt, swatch_event_timestamp = helpers.as_swatch_event_dt(timestamp)
      >       swatch_event = helpers.kafka.trigger_and_find_swatch_event(
                  swatch_event_timestamp, hbi_event, swatch_event_type
              )iqe_rhsm_subscriptions/tests/component/swatch_metrics_hbi/test_hbi_created_updated_events.py:61: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
      iqe_rhsm_subscriptions/tests/component/swatch_metrics_hbi/helpers.py:223: in trigger_and_find_swatch_event
          event = self.find_swatch_event(swatch_event_type, inventory_id, expected_timestamp)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      iqe_rhsm_subscriptions/tests/component/swatch_metrics_hbi/helpers.py:207: in find_swatch_event
          event = self.swatch_event_consumer.find_msgs(
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _self = IQEKafkaConsumer(_consumer=<iqe_mq._kafka_consumer.InsightsDeserializingConsumer object at 0x7f99ba0d9e90>), topic = 'platform.rhsm-subscriptions.service-instance-ingress', data_to_match = {'event_type': 'INSTANCE_CREATED', 'instance_id': '0d6b0c92-d3a2-40c6-961b-904a487e2910', 'timestamp': '2025-07-17T12:00:00Z'}
      num_matches = 1, matching_func = <bound method SwatchMetricsHbiKafkaHelper._filter_by_mapping of <iqe_rhsm_subscriptions.tests.component.swatch_metrics_hbi.helpers.SwatchMetricsHbiKafkaHelper object at 0x7f99b9f81be0>>, allow_empty = False, timeout_in_seconds = 30    def find_msgs(
              self,
              topic: str | list[str],
              data_to_match: Any,
              num_matches: int = 1,
              matching_func: Callable[[Any, Any], bool] = operator.eq,
              allow_empty: bool = True,
              timeout_in_seconds: int = 30,
          ) -> list[Message | MessageWrapper] | Message | MessageWrapper:
              if topic is not None:
                  self.subscribe([topic] if isinstance(topic, str) else topic)
              logger.info(f"Searching messages in the topic: {topic}")
              result = list(
                  self._find_messages(
                      data_to_match=data_to_match,
                      num_matches=num_matches,
                      matching_func=matching_func,
                      timeout_in_seconds=timeout_in_seconds,
                  )
              )
          
              if not result:
                  if allow_empty:
                      return []
      >           raise NoMessageConsumedError()
      E           iqe_mq.exceptions.NoMessageConsumedError../.iqe-venv/lib64/python3.13/site-packages/iqe_mq/_kafka_consumer.py:172: NoMessageConsumedError
      ------------------------------------------------------------------------------------------------------------------------------------------------------ Captured stderr setup ------------------------------------------------------------------------------------------------------------------------------------------------------
      2025-07-17 14:44:20,416 - Using random seed value for random generation: 6997
      ------------------------------------------------------------------------------------------------------------------------------------------------------- Captured log setup --------------------------------------------------------------------------------------------------------------------------------------------------------
      2025-07-17 14:44:12 INFO Enabling flag swatch.swatch-metrics-hbi.emit-events, and waiting for changes to propagate to services.
      2025-07-17 14:44:12 INFO Setting Session for Unleash server as admin.
      2025-07-17 14:44:14 INFO Registering unleash client with unleash @ http://localhost:4242/api
      2025-07-17 14:44:14 INFO Registration request information: {'appName': 'default', 'instanceId': 'unleash-client-python', 'connectionId': '80f141e1-ff57-44d2-aa0d-cfe2b431d19d', 'sdkVersion': 'unleash-client-python:6.2.1', 'strategies': [], 'started': '2025-07-17T12:44:14.876779+00:00', 'interval': 60, 'platformName': 'CPython', 'platformVersion': '3.13.5', 'yggdrasilVersion': '0.17.4', 'specVersion': '5.1.9'}
      2025-07-17 14:44:14 INFO Unleash Client successfully registered!
      2025-07-17 14:44:14 INFO Getting feature flag.
      2025-07-17 14:44:14 INFO Scheduler started
      2025-07-17 14:44:14 INFO Added job "fetch_and_load_features" to job store "default"
      2025-07-17 14:44:14 INFO Added job "aggregate_and_send_metrics" to job store "default"
      2025-07-17 14:44:14 INFO Setting Unleash-client log level to: WARNING
      2025-07-17 14:44:14 INFO Using LOCAL conf path /home/tlencion/iqe/iqe-rhsm-subscriptions-plugin/iqe_rhsm_subscriptions/conf/
      2025-07-17 14:44:15 INFO Searching messages in the topic: platform.rhsm-subscriptions.service-instance-ingress
      2025-07-17 14:44:19 INFO Consumer subscribed to: {'platform.rhsm-subscriptions.service-instance-ingress': {0: -1001}}
      2025-07-17 14:44:20 INFO Topic scan time limit elapsed
      2025-07-17 14:44:20 INFO Using random seed value for random generation: 6997
      -------------------------------------------------------------------------------------------------------------------------------------------------------- Captured log call --------------------------------------------------------------------------------------------------------------------------------------------------------
      2025-07-17 14:44:20 INFO Message key=None for insights_id=ee3e6ee3-e385-4960-91c8-87568ff14dac delivered to platform.inventory.events (P 0 O 22)
      2025-07-17 14:44:20 INFO Searching messages in the topic: platform.rhsm-subscriptions.service-instance-ingress
      2025-07-17 14:44:20 INFO Consumer subscribed to: {'platform.rhsm-subscriptions.service-instance-ingress': {0: -1001}}
      2025-07-17 14:44:50 INFO Topic scan time limit elapsed 

      Successful run logs:

      iqe_rhsm_subscriptions/tests/component/swatch_metrics_hbi/test_hbi_created_updated_events.py::test_hbi_physical_x86_host_events[created-INSTANCE_CREATED] 
      --------------------------------------------------------------------------------------------------------------------------------------------------------- live log setup ----------------------------------------------------------------------------------------------------------------------------------------------------------
      2025-07-17 14:50:09 INFO Enabling flag swatch.swatch-metrics-hbi.emit-events, and waiting for changes to propagate to services.
      2025-07-17 14:50:09 INFO Setting Session for Unleash server as admin.
      2025-07-17 14:50:11 INFO Registering unleash client with unleash @ http://localhost:4242/api
      2025-07-17 14:50:11 INFO Registration request information: {'appName': 'default', 'instanceId': 'unleash-client-python', 'connectionId': 'e2464576-e308-4a08-936a-3e4377f8dd1c', 'sdkVersion': 'unleash-client-python:6.2.1', 'strategies': [], 'started': '2025-07-17T12:50:11.230094+00:00', 'interval': 60, 'platformName': 'CPython', 'platformVersion': '3.13.5', 'yggdrasilVersion': '0.17.4', 'specVersion': '5.1.9'}
      2025-07-17 14:50:11 INFO Unleash Client successfully registered!
      2025-07-17 14:50:11 INFO Getting feature flag.
      2025-07-17 14:50:11 INFO Scheduler started
      2025-07-17 14:50:11 INFO Added job "fetch_and_load_features" to job store "default"
      2025-07-17 14:50:11 INFO Added job "aggregate_and_send_metrics" to job store "default"
      2025-07-17 14:50:11 INFO Setting Unleash-client log level to: WARNING
      2025-07-17 14:50:11 INFO Using LOCAL conf path /home/tlencion/iqe/iqe-rhsm-subscriptions-plugin/iqe_rhsm_subscriptions/conf/
      2025-07-17 14:50:11 INFO Searching messages in the topic: platform.rhsm-subscriptions.service-instance-ingress
      2025-07-17 14:50:15 INFO Consumer subscribed to: {'platform.rhsm-subscriptions.service-instance-ingress': {0: -1001}}
      2025-07-17 14:50:16 INFO Topic scan time limit elapsed
      2025-07-17 14:50:16 INFO Using random seed value for random generation: 2043
      ---------------------------------------------------------------------------------------------------------------------------------------------------------- live log call ----------------------------------------------------------------------------------------------------------------------------------------------------------
      2025-07-17 14:50:16 INFO Message key=None for insights_id=ba840b94-fdcb-4ec4-a4a0-f501911effdc delivered to platform.inventory.events (P 0 O 62)
      2025-07-17 14:50:16 INFO Searching messages in the topic: platform.rhsm-subscriptions.service-instance-ingress
      2025-07-17 14:50:16 INFO Consumer subscribed to: {'platform.rhsm-subscriptions.service-instance-ingress': {0: -1001}}
      2025-07-17 14:50:16 INFO All messages found matching: {'event_type': 'INSTANCE_CREATED', 'instance_id': 'ffbf0b40-bdc4-4795-994a-b90a8a7cc683', 'timestamp': '2025-07-17T12:00:00Z'}
      PASSED                                                                                                                                                                                                                                                                                                                      [  6%]
      iqe_rhsm_subscriptions/tests/component/swatch_metrics_hbi/test_hbi_created_updated_events.py::test_hbi_physical_x86_host_events[updated-INSTANCE_UPDATED]  

       

              mstead@redhat.com Michael Stead
              rh-ee-tlencion Tommaso Lencioni
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: