Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-29965

Fix flaky integration tests (race condition, CRD optimization, nil-safety guards)

XMLWordPrintable

    • Quality / Stability / Reliability
    • 2
    • False
    • Hide

      None

      Show
      None
    • False
    • GH Train-36
    • None

      Problem Statement

      Multiple integration test suites have flaky failures due to race conditions, missing CRD optimization, and nil-safety issues in test teardown.

      Version

      Global Hub 1.7.0

      Is it reproducible?

      Yes - intermittent failures in CI, especially under load.

      Steps to Reproduce

      Run integration tests repeatedly:

      make integration\-test
      
      Actual Results

      1. agent/status tests fail due to race condition on map access (concurrent goroutine writes to map[string]*cloudevents.Event while tests read)
      2. operator/controllers transporter test fails with 409 Conflict (concurrent Kafka CR modification)
      3. envtest suites fail with CRD install timeouts when loading all 35+ CRDs unnecessarily
      4. AfterSuite panics on nil testPostgres when BeforeSuite fails

      Expected Results

      All integration tests pass reliably without flaky failures.

      Additional Info

      Fix PR: https://github.com/stolostron/multicluster-global-hub/pull/2286

      Changes:
      - Replace map with sync.Map for concurrent event access in agent/status tests
      - Wrap transporter EnsureKafka in Eventually retry block for 409 conflicts
      - Optimize each test suite to load only required CRDs instead of full directory (35+ CRDs)
      - Add nil guards for testPostgres in AfterSuite across 4 operator/manager suites
      - Remove StartEnvTestWithRetry utility in favor of CRD optimization approach

      Generated with [Claude Code|https://claude.ai/claude-code]

              rh-ee-myan Meng Yan
              rh-ee-myan Meng Yan
              Yaheng Liu Yaheng Liu
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: