Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-25134

fix: resolve e2e test failures caused by concurrent operations

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • None
    • Global Hub
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • False
    • GH Train-33
    • None

      Summary

      This PR fixes two critical bugs that were causing e2e test failures due to concurrent operation conflicts:

      1. Policy Event Handler - Duplicate Events Issue

      Problem: PostgreSQL error "ON CONFLICT DO UPDATE command cannot affect row a second time" occurred when duplicate events with the same composite key (event_name, count, created_at) were present in a single batch insert

      Solution: Added deduplication logic in policy event handlers using a map to filter duplicate events before batch insertion, keeping only the last occurrence of duplicates

      2. KafkaUser Update - Concurrent Modification Issue

      Problem: Massive KafkaUser update conflicts with error "the object has been modified; please apply your changes to the latest version and try again". In 2 seconds (12:10:00-12:10:02), operator attempted to update hub1-kafka-user over 20 times, all failing due to resource version conflicts

      Solution: Added retry logic with retry.RetryOnConflict to the EnsureUser method to handle optimistic concurrency conflicts when multiple controllers try to update the same KafkaUser resource simultaneously

      Changes

      • Deduplicate events in local_root_policy_event_handler.go and local_replicated_policy_event_handler.go before batch insertion
      • Add retry mechanism in KafkaUser update operations to handle concurrent modifications gracefully

      PR Link

      https://github.com/stolostron/multicluster-global-hub/pull/2044

              rh-ee-myan Meng Yan
              rh-ee-myan Meng Yan
              Yaheng Liu Yaheng Liu
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: