Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-23920

Some times the managedcluster lost in source hub when restart pod

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • Global Hub 1.6.0
    • Global Hub 1.6.0
    • Global Hub
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • False
    • GH Train-33
    • Moderate
    • None

      Description of problem:

      In performance env, I restart all agent and manager pod, the new created migration always hang in validating. and all my managed cluster lost.

      it looks source hub agent consum old kafka events, There is no mcm in cluster, but the agent still run registing and cleaning process, it makes the cluster deleted.

       

      log in manager:

      2025-09-09T08:12:16.812Z    INFO    migration/migration_pending.go:83    selected migration: migrate-300-cluster-round-1 (phase: Validating)
      2025-09-09T08:12:16.812Z    INFO    migration/migration_controller.go:141    processing migration instance: migrate-300-cluster-round-1
      2025-09-09T08:12:16.812Z    INFO    migration/migration_controller.go:291    set migration: migrate-300-cluster-round-1 timeouts to 10m0s
      2025-09-09T08:12:16.812Z    INFO    migration/migration_validating.go:74    migration: migrate-300-cluster-round-1 validating
      2025-09-09T08:12:16.812Z    INFO    migration/migration_validating.go:80    migration validating
      2025-09-09T08:12:16.812Z    INFO    migration/migration_validating.go:105    migration validating from hub
      2025-09-09T08:12:16.812Z    INFO    migration/migration_validating.go:116    migration validating to hub
      2025-09-09T08:12:16.812Z    INFO    migration/migration_validating.go:129    migration validating clusters
      2025-09-09T08:12:21.813Z    INFO    migration/migration_controller.go:128    reconcile managed cluster migration multicluster-global-hub/migrate-300-cluster-round-1
      2025-09-09T08:12:21.813Z    INFO    migration/migration_pending.go:83    selected migration: migrate-300-cluster-round-1 (phase: Validating)
      2025-09-09T08:12:21.813Z    INFO    migration/migration_controller.go:141    processing migration instance: migrate-300-cluster-round-1
      2025-09-09T08:12:21.813Z    INFO    migration/migration_controller.go:291    set migration: migrate-300-cluster-round-1 timeouts to 10m0s
      2025-09-09T08:12:21.813Z    INFO    migration/migration_validating.go:74    migration: migrate-300-cluster-round-1 validating
      2025-09-09T08:12:21.813Z    INFO    migration/migration_validating.go:80    migration validating
      2025-09-09T08:12:21.813Z    INFO    migration/migration_validating.go:105    migration validating from hub
      2025-09-09T08:12:21.813Z    INFO    migration/migration_validating.go:116    migration validating to hub
      2025-09-09T08:12:21.813Z    INFO    migration/migration_validating.go:129    migration validating clusters
      2025-09-09T08:12:24.842Z    INFO    migration/migration_controller.go:128    reconcile managed cluster migration multicluster-global-hub/migrate-300-cluster-round-1
      2025-09-09T08:12:24.842Z    INFO    migration/migration_controller.go:141    processing migration instance: migrate-300-cluster-round-1
      2025-09-09T08:12:24.842Z    INFO    migration/migration_controller.go:291    set migration: migrate-300-cluster-round-1 timeouts to 10m0s
      2025-09-09T08:12:24.842Z    INFO    migration/migration_eventstatus.go:62    clean up migration status for migrationId: 60e799cb-c04a-4a82-a503-4fddd2f23c8f
      2025-09-09T08:12:24.849Z    INFO    migration/migration_controller.go:223    clean up migration status for migrationId: 60e799cb-c04a-4a82-a503-4fddd2f23c8f
      2025-09-09T08:12:24.849Z    INFO    migration/migration_controller.go:128    reconcile managed cluster migration multicluster-global-hub/migrate-300-cluster-round-1

      source hub agent log:(it still run the previous migration registing which already deleted)

      2025-09-09T08:08:33.562Z    INFO    migration/migration_from_syncer.go:672    deleted managed cluster vm00110
      2025-09-09T08:08:33.573Z    INFO    migration/migration_from_syncer.go:672    deleted managed cluster vm00111
      2025-09-09T08:08:33.573Z    INFO    migration/migration_from_syncer.go:149    migration Cleaning completed: migrationId=9aeb253b-2b50-4164-8419-cc623717bbfe
      2025-09-09T08:08:33.573Z    INFO    migration/migration_to_syncer.go:71    received migration event from global-hub
      2025-09-09T08:08:33.574Z    INFO    migration/migration_to_syncer.go:159    migration Initializing started: migrationId=6a8007b6-8dd9-4b39-b96b-ccee88b27bfc, clusters=[]
      2025-09-09T08:08:33.585Z    INFO    migration/migration_to_syncer.go:419    creating migration clusterrole
      2025-09-09T08:08:33.622Z    INFO    migration/migration_to_syncer.go:535    creating subjectaccessreviews clusterrolebinding
      2025-09-09T08:08:33.711Z    INFO    migration/migration_to_syncer.go:477    creating agent registration clusterrolebindingclusterrolebindingglobal-hub-migration-migrate-100-cluster-round-1-bk-registration
      2025-09-09T08:08:33.760Z    INFO    migration/migration_to_syncer.go:167    migration Initializing completed: migrationId=6a8007b6-8dd9-4b39-b96b-ccee88b27bfc
      2025-09-09T08:08:33.760Z    INFO    migration/migration_to_syncer.go:71    received migration event from mh1
      2025-09-09T08:08:33.767Z    INFO    migration/migration_to_syncer.go:290    started the deploying: 6a8007b6-8dd9-4b39-b96b-ccee88b27bfc
      2025-09-09T08:08:33.923Z    INFO    migration/migration_to_syncer.go:309    finished syncing migration resources
      2025-09-09T08:08:33.923Z    INFO    migration/migration_to_syncer.go:71    received migration event from global-hub
      2025-09-09T08:08:33.924Z    INFO    migration/migration_to_syncer.go:159    migration Registering started: migrationId=6a8007b6-8dd9-4b39-b96b-ccee88b27bfc, clusters=[vm00006 vm00007 vm00009 vm00012 vm00013 vm00016 vm00018 vm00021 vm00022 vm00023 vm00025 vm00026 vm00029 vm00030 vm00031 vm00033 vm00034 vm00035 vm00041 vm00048 vm00055 vm00056 vm00057 vm00059 vm00060 vm00061 vm00064 vm00065 vm00068 vm00071 vm00073 vm00075 vm00079 vm00082 vm00086 vm00094 vm00095 vm00102 vm00103 vm00104 vm00107 vm00110 vm00112 vm00117 vm00119 vm00126 vm00128 vm00134 vm00139 vm00141 vm00142 vm00144 vm00146 vm00148 vm00157 vm00159 vm00161 vm00168 vm00170 vm00178 vm00181 vm00185 vm00190 vm00192 vm00195 vm00199 vm00201 vm00202 vm00203 vm00204 vm00205 vm00206 vm00207 vm00208 vm00209 vm00210 vm00212 vm00215 vm00216 vm00220 vm00221 vm00222 vm00224 vm00225 vm00227 vm00228 vm00233 vm00234 vm00236 vm00237 vm00240 vm00241 vm00243 vm00249 vm00256 vm00257 vm00259 vm00260 vm00262 vm00263]

       

      Version-Release number of selected component (if applicable):

      How reproducible:

      Steps to Reproduce:

      1.  
      2.  
      3. ...

      Actual results:

      Expected results:

      Additional info:

              rh-ee-myan Meng Yan
              daliu@redhat.com DangPeng Liu
              Hui Chen Hui Chen
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: