-
Bug
-
Resolution: Unresolved
-
Normal
-
Global Hub 1.6.0
-
None
-
Quality / Stability / Reliability
-
False
-
-
False
-
-
-
GH Train-33
-
Moderate
-
None
Description of problem:
In performance env, I restart all agent and manager pod, the new created migration always hang in validating. and all my managed cluster lost.
it looks source hub agent consum old kafka events, There is no mcm in cluster, but the agent still run registing and cleaning process, it makes the cluster deleted.
log in manager:
2025-09-09T08:12:16.812Z INFO migration/migration_pending.go:83 selected migration: migrate-300-cluster-round-1 (phase: Validating)
2025-09-09T08:12:16.812Z INFO migration/migration_controller.go:141 processing migration instance: migrate-300-cluster-round-1
2025-09-09T08:12:16.812Z INFO migration/migration_controller.go:291 set migration: migrate-300-cluster-round-1 timeouts to 10m0s
2025-09-09T08:12:16.812Z INFO migration/migration_validating.go:74 migration: migrate-300-cluster-round-1 validating
2025-09-09T08:12:16.812Z INFO migration/migration_validating.go:80 migration validating
2025-09-09T08:12:16.812Z INFO migration/migration_validating.go:105 migration validating from hub
2025-09-09T08:12:16.812Z INFO migration/migration_validating.go:116 migration validating to hub
2025-09-09T08:12:16.812Z INFO migration/migration_validating.go:129 migration validating clusters
2025-09-09T08:12:21.813Z INFO migration/migration_controller.go:128 reconcile managed cluster migration multicluster-global-hub/migrate-300-cluster-round-1
2025-09-09T08:12:21.813Z INFO migration/migration_pending.go:83 selected migration: migrate-300-cluster-round-1 (phase: Validating)
2025-09-09T08:12:21.813Z INFO migration/migration_controller.go:141 processing migration instance: migrate-300-cluster-round-1
2025-09-09T08:12:21.813Z INFO migration/migration_controller.go:291 set migration: migrate-300-cluster-round-1 timeouts to 10m0s
2025-09-09T08:12:21.813Z INFO migration/migration_validating.go:74 migration: migrate-300-cluster-round-1 validating
2025-09-09T08:12:21.813Z INFO migration/migration_validating.go:80 migration validating
2025-09-09T08:12:21.813Z INFO migration/migration_validating.go:105 migration validating from hub
2025-09-09T08:12:21.813Z INFO migration/migration_validating.go:116 migration validating to hub
2025-09-09T08:12:21.813Z INFO migration/migration_validating.go:129 migration validating clusters
2025-09-09T08:12:24.842Z INFO migration/migration_controller.go:128 reconcile managed cluster migration multicluster-global-hub/migrate-300-cluster-round-1
2025-09-09T08:12:24.842Z INFO migration/migration_controller.go:141 processing migration instance: migrate-300-cluster-round-1
2025-09-09T08:12:24.842Z INFO migration/migration_controller.go:291 set migration: migrate-300-cluster-round-1 timeouts to 10m0s
2025-09-09T08:12:24.842Z INFO migration/migration_eventstatus.go:62 clean up migration status for migrationId: 60e799cb-c04a-4a82-a503-4fddd2f23c8f
2025-09-09T08:12:24.849Z INFO migration/migration_controller.go:223 clean up migration status for migrationId: 60e799cb-c04a-4a82-a503-4fddd2f23c8f
2025-09-09T08:12:24.849Z INFO migration/migration_controller.go:128 reconcile managed cluster migration multicluster-global-hub/migrate-300-cluster-round-1
source hub agent log:(it still run the previous migration registing which already deleted)
2025-09-09T08:08:33.562Z INFO migration/migration_from_syncer.go:672 deleted managed cluster vm00110
2025-09-09T08:08:33.573Z INFO migration/migration_from_syncer.go:672 deleted managed cluster vm00111
2025-09-09T08:08:33.573Z INFO migration/migration_from_syncer.go:149 migration Cleaning completed: migrationId=9aeb253b-2b50-4164-8419-cc623717bbfe
2025-09-09T08:08:33.573Z INFO migration/migration_to_syncer.go:71 received migration event from global-hub
2025-09-09T08:08:33.574Z INFO migration/migration_to_syncer.go:159 migration Initializing started: migrationId=6a8007b6-8dd9-4b39-b96b-ccee88b27bfc, clusters=[]
2025-09-09T08:08:33.585Z INFO migration/migration_to_syncer.go:419 creating migration clusterrole
2025-09-09T08:08:33.622Z INFO migration/migration_to_syncer.go:535 creating subjectaccessreviews clusterrolebinding
2025-09-09T08:08:33.711Z INFO migration/migration_to_syncer.go:477 creating agent registration clusterrolebindingclusterrolebindingglobal-hub-migration-migrate-100-cluster-round-1-bk-registration
2025-09-09T08:08:33.760Z INFO migration/migration_to_syncer.go:167 migration Initializing completed: migrationId=6a8007b6-8dd9-4b39-b96b-ccee88b27bfc
2025-09-09T08:08:33.760Z INFO migration/migration_to_syncer.go:71 received migration event from mh1
2025-09-09T08:08:33.767Z INFO migration/migration_to_syncer.go:290 started the deploying: 6a8007b6-8dd9-4b39-b96b-ccee88b27bfc
2025-09-09T08:08:33.923Z INFO migration/migration_to_syncer.go:309 finished syncing migration resources
2025-09-09T08:08:33.923Z INFO migration/migration_to_syncer.go:71 received migration event from global-hub
2025-09-09T08:08:33.924Z INFO migration/migration_to_syncer.go:159 migration Registering started: migrationId=6a8007b6-8dd9-4b39-b96b-ccee88b27bfc, clusters=[vm00006 vm00007 vm00009 vm00012 vm00013 vm00016 vm00018 vm00021 vm00022 vm00023 vm00025 vm00026 vm00029 vm00030 vm00031 vm00033 vm00034 vm00035 vm00041 vm00048 vm00055 vm00056 vm00057 vm00059 vm00060 vm00061 vm00064 vm00065 vm00068 vm00071 vm00073 vm00075 vm00079 vm00082 vm00086 vm00094 vm00095 vm00102 vm00103 vm00104 vm00107 vm00110 vm00112 vm00117 vm00119 vm00126 vm00128 vm00134 vm00139 vm00141 vm00142 vm00144 vm00146 vm00148 vm00157 vm00159 vm00161 vm00168 vm00170 vm00178 vm00181 vm00185 vm00190 vm00192 vm00195 vm00199 vm00201 vm00202 vm00203 vm00204 vm00205 vm00206 vm00207 vm00208 vm00209 vm00210 vm00212 vm00215 vm00216 vm00220 vm00221 vm00222 vm00224 vm00225 vm00227 vm00228 vm00233 vm00234 vm00236 vm00237 vm00240 vm00241 vm00243 vm00249 vm00256 vm00257 vm00259 vm00260 vm00262 vm00263]
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
- ...