-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
ACM 2.15.0, ACM 2.16.0
-
None
-
Quality / Stability / Reliability
-
False
-
-
False
-
-
-
None
Summary
When migrating resources (managedcluster/clusterdeployment/imageclusterinstall) from ACM 2.13 to ACM 2.15 using GlobalHub, if ClusterInstance fails initial reconciliation due to a validation error (e.g., missing pullSecretRef), the status does not update correctly even after the issue is fixed.
Target hub Environment
- ACM Version: 2.15
- MCE Version: 2.11
Reproduce steps:
1. GlobalHub migrates managedcluster/clusterdeployment/imageclusterinstall resources from ACM 2.13 to ACM 2.15, During migration, ClusterDeployment and ImageClusterInstall status are also updated to target ACM hub
2. After migration completes, manually create ClusterInstance resource
3. If ClusterInstance has an error on first reconcile (e.g., clusterinstance.spec.pullSecretRef references a Secret that does not exist), reconciliation fails with error.
4. Fix the issue (e.g., create the missing pull secret)
5. ClusterInstance status remains incorrect - PROVISIONSTATUS and PROVISIONDETAILS columns are empty
Expected Result
After fixing the validation error and ClusterInstance reconciles successfully:
oc get clusterinstance \-A NAMESPACE NAME PAUSED PROVISIONSTATUS PROVISIONDETAILS AGE vm041 vm041\-clusterinstance Completed Provisioning completed 2d16h
Actual Result
After fixing the validation error:
oc get clusterinstance \-A NAMESPACE NAME PAUSED PROVISIONSTATUS PROVISIONDETAILS AGE vm041 vm041\-clusterinstance 3m26s
The PROVISIONSTATUS and PROVISIONDETAILS columns remain empty even though the ClusterInstance conditions show successful validation and template rendering.
Clusterinstance CR
- apiVersion: siteconfig.open-cluster-management.io/v1alpha1 kind: ClusterInstance metadata: creationTimestamp: "2026-03-06T09:48:39Z" finalizers: - clusterinstance.siteconfig.open-cluster-management.io/finalizer generation: 1 name: vm041-clusterinstance namespace: vm041 resourceVersion: "259971155" uid: e8578083-4fd3-4f27-b1c5-706874eb5aac spec: baseDomain: qe.red-chesterfield.com clusterImageSetNameRef: img4.19.15-x86-64-appsub clusterName: vm041 clusterType: SNO cpuArchitecture: x86_64 cpuPartitioningMode: None extraLabels: ManagedCluster: siteName: eric-vdu-vm sno: "true" sriov: "true" vduCrdNamespace: hv-3-sno-4-ec-gnb-crd vduNamespace: hv-3-sno-4-ec-gnb-001 vendor: OpenShift holdInstallation: false networkType: OVNKubernetes nodes: - automatedCleaningMode: metadata bmcAddress: redfish-virtualmedia://192.168.123.1:8000/redfish/v1/Systems/bb6d3f2c-8ec1-4b71-9aec-581c61fca673 bmcCredentialsName: name: vm041-bmc-secret bootMACAddress: 52:54:00:cf:0b:8f bootMode: UEFI hostName: vm041 ironicInspect: "" role: master templateRefs: - name: ibi-node-templates-v1 namespace: ocm pullSecretRef: name: assisted-deployment-pull-secret templateRefs: - name: ibi-cluster-templates-v1 namespace: ocm status: conditions: - lastTransitionTime: "2026-03-06T09:50:01Z" message: Validation succeeded reason: Completed status: "True" type: ClusterInstanceValidated - lastTransitionTime: "2026-03-06T09:50:01Z" message: Rendered templates successfully reason: Completed status: "True" type: RenderedTemplates - lastTransitionTime: "2026-03-06T09:50:01Z" message: Rendered templates validation succeeded reason: Completed status: "True" type: RenderedTemplatesValidated - lastTransitionTime: "2026-03-06T09:50:01Z" message: Applied site config manifests reason: Completed status: "True" type: RenderedTemplatesApplied manifestsRendered: - apiGroup: metal3.io/v1alpha1 kind: BareMetalHost lastAppliedTime: "2026-03-06T09:50:01Z" name: vm041 namespace: vm041 status: rendered syncWave: 1 - apiGroup: hive.openshift.io/v1 kind: ClusterDeployment lastAppliedTime: "2026-03-06T09:50:01Z" name: vm041 namespace: vm041 status: rendered syncWave: 1 - apiGroup: extensions.hive.openshift.io/v1alpha1 kind: ImageClusterInstall lastAppliedTime: "2026-03-06T09:50:01Z" name: vm041 namespace: vm041 status: rendered syncWave: 1 - apiGroup: agent.open-cluster-management.io/v1 kind: KlusterletAddonConfig lastAppliedTime: "2026-03-06T09:50:01Z" name: vm041 namespace: vm041 status: rendered syncWave: 2 - apiGroup: cluster.open-cluster-management.io/v1 kind: ManagedCluster lastAppliedTime: "2026-03-06T09:50:01Z" name: vm041 status: rendered syncWave: 2 observedGeneration: 1 kind: List metadata: resourceVersion: "" [root@ice02 vm028]# oc get clusterinstance -A -w NAMESPACE NAME PAUSED PROVISIONSTATUS PROVISIONDETAILS AGE vm041 vm041-clusterinstance 3m26s
Siteconfig Operator Logs
2026-03-06T09:49:20.784Z ERROR ClusterInstanceController controller/clusterinstance_controller.go:501 ClusterInstance validation failed due to error {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580", "error": "resource validation failed: failed to validate Pull Secret: [assisted-deployment-pull-secret in namespace vm041], err: Secret \"assisted-deployment-pull-secret\" not found"}
github.com/stolostron/siteconfig/internal/controller.(*ClusterInstanceReconciler).handleValidate
/opt/app-root/src/internal/controller/clusterinstance_controller.go:501
github.com/stolostron/siteconfig/internal/controller.(*ClusterInstanceReconciler).Reconcile
/opt/app-root/src/internal/controller/clusterinstance_controller.go:247
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:216
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:461
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:421
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:296
2026-03-06T09:49:20.784Z INFO ClusterInstanceController controller/clusterinstance_controller.go:514 Finished validation {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580"}
2026-03-06T09:49:20.795Z INFO ClusterInstanceController controller/clusterinstance_controller.go:154 Finished reconciling ClusterInstance {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580"}
2026-03-06T09:49:20Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "clusterinstance", "controllerGroup": "siteconfig.open-cluster-management.io", "controllerKind": "ClusterInstance", "ClusterInstance": {"name":"vm041-clusterinstance","namespace":"vm041"}, "namespace": "vm041", "name": "vm041-clusterinstance", "reconcileID": "dfec2449-5724-4cc7-b8af-f2f38c2fb1d1"}
2026-03-06T09:49:20Z ERROR Reconciler error {"controller": "clusterinstance", "controllerGroup": "siteconfig.open-cluster-management.io", "controllerKind": "ClusterInstance", "ClusterInstance": {"name":"vm041-clusterinstance","namespace":"vm041"}, "namespace": "vm041", "name": "vm041-clusterinstance", "reconcileID": "dfec2449-5724-4cc7-b8af-f2f38c2fb1d1", "error": "resource validation failed: failed to validate Pull Secret: [assisted-deployment-pull-secret in namespace vm041], err: Secret \"assisted-deployment-pull-secret\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:474
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:421
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
/opt/app-root/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:296
2026-03-06T09:50:01.756Z INFO ClusterInstanceController controller/clusterinstance_controller.go:157 Starting reconcile ClusterInstance {"name": "vm041-clusterinstance", "namespace": "vm041"}
2026-03-06T09:50:01.756Z INFO ClusterInstanceController controller/clusterinstance_controller.go:178 Loaded ClusterInstance {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580"}
2026-03-06T09:50:01.756Z INFO ClusterInstanceController controller/clusterinstance_controller.go:498 Starting validation {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580"}
2026-03-06T09:50:01.756Z INFO ClusterInstanceController controller/clusterinstance_controller.go:508 Validation succeeded {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580"}
2026-03-06T09:50:01.756Z INFO ClusterInstanceController controller/clusterinstance_controller.go:514 Finished validation {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580"}
2026-03-06T09:50:01.766Z INFO ClusterInstanceController controller/clusterinstance_controller.go:830 Starting to render templates {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580"}
2026-03-06T09:50:01.766Z INFO ClusterInstanceController controller/clusterinstance_controller.go:536 Starting to render templates {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580"}
2026-03-06T09:50:01.766Z INFO ClusterInstanceController.TemplateEngine.ProcessTemplates clusterinstance/template_engine.go:56 Started processing cluster-level install templates {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580"}
2026-03-06T09:50:01.766Z INFO ClusterInstanceController.TemplateEngine.ProcessTemplates.renderTemplates clusterinstance/template_engine.go:120 Processing templateRef 1 of 1 {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580"}
2026-03-06T09:50:01.767Z INFO ClusterInstanceController.TemplateEngine.ProcessTemplates clusterinstance/template_engine.go:68 Finished processing cluster-level install templates {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580"}
2026-03-06T09:50:01.767Z INFO ClusterInstanceController.TemplateEngine.ProcessTemplates clusterinstance/template_engine.go:73 Started processing node-level install templates [node: 1 of 1] {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580"}
2026-03-06T09:50:01.767Z INFO ClusterInstanceController.TemplateEngine.ProcessTemplates.renderTemplates clusterinstance/template_engine.go:120 Processing templateRef 1 of 1 {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580"}
2026-03-06T09:50:01.768Z INFO ClusterInstanceController.TemplateEngine.ProcessTemplates clusterinstance/template_engine.go:89 Finished processing node-level install templates [node: 1 of 1] {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580"}
2026-03-06T09:50:01.768Z INFO ClusterInstanceController controller/clusterinstance_controller.go:549 Successfully rendered templates {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580"}
2026-03-06T09:50:01.776Z INFO ClusterInstanceController controller/clusterinstance_controller.go:564 Finished rendering templates {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580"}
2026-03-06T09:50:01.776Z INFO ClusterInstanceController.pruneManifests controller/clusterinstance_controller.go:753 No objects to prune; skipping pruning operation {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580"}
2026-03-06T09:50:01.776Z INFO ClusterInstanceController.validateRenderedManifests controller/clusterinstance_controller.go:657 Executing a dry-run validation on the rendered manifests {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580"}
2026-03-06T09:50:01.776Z DEBUG ClusterInstanceController.validateRenderedManifests.applyObject controller/clusterinstance_controller.go:599 Applying object using Server-Side Apply {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580", "name": "vm041", "namespace": "vm041", "kind": "BareMetalHost"}
2026-03-06T09:50:01.794Z DEBUG ClusterInstanceController.validateRenderedManifests.applyObject controller/clusterinstance_controller.go:607 Object applied using Server-Side Apply {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580", "name": "vm041", "namespace": "vm041", "kind": "BareMetalHost"}
2026-03-06T09:50:01.794Z DEBUG ClusterInstanceController.validateRenderedManifests.applyObject controller/clusterinstance_controller.go:599 Applying object using Server-Side Apply {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580", "name": "vm041", "namespace": "vm041", "kind": "ClusterDeployment"}
2026-03-06T09:50:01.817Z DEBUG ClusterInstanceController.validateRenderedManifests.applyObject controller/clusterinstance_controller.go:607 Object applied using Server-Side Apply {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580", "name": "vm041", "namespace": "vm041", "kind": "ClusterDeployment"}
2026-03-06T09:50:01.817Z DEBUG ClusterInstanceController.validateRenderedManifests.applyObject controller/clusterinstance_controller.go:599 Applying object using Server-Side Apply {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580", "name": "vm041", "namespace": "vm041", "kind": "ImageClusterInstall"}
2026-03-06T09:50:01.831Z DEBUG ClusterInstanceController.validateRenderedManifests.applyObject controller/clusterinstance_controller.go:607 Object applied using Server-Side Apply {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580", "name": "vm041", "namespace": "vm041", "kind": "ImageClusterInstall"}
2026-03-06T09:50:01.831Z DEBUG ClusterInstanceController.validateRenderedManifests.applyObject controller/clusterinstance_controller.go:599 Applying object using Server-Side Apply {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580", "name": "vm041", "namespace": "vm041", "kind": "KlusterletAddonConfig"}
2026-03-06T09:50:01.843Z DEBUG ClusterInstanceController.validateRenderedManifests.applyObject controller/clusterinstance_controller.go:607 Object applied using Server-Side Apply {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580", "name": "vm041", "namespace": "vm041", "kind": "KlusterletAddonConfig"}
2026-03-06T09:50:01.843Z DEBUG ClusterInstanceController.validateRenderedManifests.applyObject controller/clusterinstance_controller.go:599 Applying object using Server-Side Apply {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580", "name": "vm041", "namespace": "", "kind": "ManagedCluster"}
2026-03-06T09:50:01.880Z DEBUG ClusterInstanceController.validateRenderedManifests.applyObject controller/clusterinstance_controller.go:607 Object applied using Server-Side Apply {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580", "name": "vm041", "namespace": "", "kind": "ManagedCluster"}
2026-03-06T09:50:01.891Z INFO ClusterInstanceController.validateRenderedManifests controller/clusterinstance_controller.go:693 Finished executing a dry-run validation on the rendered manifests {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580"}
2026-03-06T09:50:01.891Z INFO ClusterInstanceController.applyRenderedManifests controller/clusterinstance_controller.go:705 Applying the rendered manifests {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580"}
2026-03-06T09:50:01.891Z DEBUG ClusterInstanceController.applyRenderedManifests.applyObject controller/clusterinstance_controller.go:599 Applying object using Server-Side Apply {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580", "name": "vm041", "namespace": "vm041", "kind": "BareMetalHost"}
2026-03-06T09:50:01.901Z DEBUG ClusterInstanceController.applyRenderedManifests.applyObject controller/clusterinstance_controller.go:607 Object applied using Server-Side Apply {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580", "name": "vm041", "namespace": "vm041", "kind": "BareMetalHost"}
2026-03-06T09:50:01.901Z DEBUG ClusterInstanceController.applyRenderedManifests.applyObject controller/clusterinstance_controller.go:599 Applying object using Server-Side Apply {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580", "name": "vm041", "namespace": "vm041", "kind": "ClusterDeployment"}
2026-03-06T09:50:01.920Z DEBUG ClusterInstanceController.applyRenderedManifests.applyObject controller/clusterinstance_controller.go:607 Object applied using Server-Side Apply {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580", "name": "vm041", "namespace": "vm041", "kind": "ClusterDeployment"}
2026-03-06T09:50:01.920Z DEBUG ClusterInstanceController.applyRenderedManifests.applyObject controller/clusterinstance_controller.go:599 Applying object using Server-Side Apply {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580", "name": "vm041", "namespace": "vm041", "kind": "ImageClusterInstall"}
2026-03-06T09:50:01.929Z DEBUG ClusterInstanceController.applyRenderedManifests.applyObject controller/clusterinstance_controller.go:607 Object applied using Server-Side Apply {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580", "name": "vm041", "namespace": "vm041", "kind": "ImageClusterInstall"}
2026-03-06T09:50:01.929Z DEBUG ClusterInstanceController.applyRenderedManifests.applyObject controller/clusterinstance_controller.go:599 Applying object using Server-Side Apply {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580", "name": "vm041", "namespace": "vm041", "kind": "KlusterletAddonConfig"}
2026-03-06T09:50:01.938Z DEBUG ClusterInstanceController.applyRenderedManifests.applyObject controller/clusterinstance_controller.go:607 Object applied using Server-Side Apply {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580", "name": "vm041", "namespace": "vm041", "kind": "KlusterletAddonConfig"}
2026-03-06T09:50:01.938Z DEBUG ClusterInstanceController.applyRenderedManifests.applyObject controller/clusterinstance_controller.go:599 Applying object using Server-Side Apply {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580", "name": "vm041", "namespace": "", "kind": "ManagedCluster"}
2026-03-06T09:50:01.964Z DEBUG ClusterInstanceController.applyRenderedManifests.applyObject controller/clusterinstance_controller.go:607 Object applied using Server-Side Apply {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580", "name": "vm041", "namespace": "", "kind": "ManagedCluster"}
2026-03-06T09:50:01.977Z INFO ClusterInstanceController.applyRenderedManifests controller/clusterinstance_controller.go:740 Finished applying the rendered manifests {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580"}
2026-03-06T09:50:01.977Z INFO ClusterInstanceController controller/clusterinstance_controller.go:261 Finished rendering templates {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580"}
2026-03-06T09:50:01.977Z INFO ClusterInstanceController controller/clusterinstance_controller.go:293 Updating clusterinstance.siteconfig.open-cluster-management.io/last-observed-spec annotation to {"clusterName":"vm041","pullSecretRef":{"name":"assisted-deployment-pull-secret"},"clusterImageSetNameRef":"img4.19.15-x86-64-appsub","baseDomain":"qe.red-chesterfield.com","networkType":"OVNKubernetes","extraLabels":{"ManagedCluster":{"acmName":"rchltx-acm-2.faredge.vzwops.com","cluster.open-cluster-management.io/clusterset":"rbac-mcs-sno","cnfIdentifier":"rchltxef-93180000vzwcgnb-y-ec-x-99988880304","common":"true","dataCenter":"richardsonNEC","eric-vdu":"","gnbDuID":"99988880304","group-du-sno":"","hostName":"hv-3-sno-4","infra":"faregde","logRegion":"midwest","logicalGroup":"25b","mcp":"master","pao":"true","server":"VM","siteName":"eric-vdu-vm","sno":"true","sriov":"true","vduCrdNamespace":"hv-3-sno-4-ec-gnb-crd","vduNamespace":"hv-3-sno-4-ec-gnb-001","vendor":"OpenShift"}},"cpuPartitioningMode":"None","cpuArchitecture":"x86_64","clusterType":"SNO","templateRefs":[{"name":"ibi-cluster-templates-v1","namespace":"ocm"}],"nodes":[{"bmcAddress":"redfish-virtualmedia://192.168.123.1:8000/redfish/v1/Systems/bb6d3f2c-8ec1-4b71-9aec-581c61fca673","bmcCredentialsName":{"name":"vm041-bmc-secret"},"bootMACAddress":"52:54:00:cf:0b:8f","automatedCleaningMode":"metadata","hostName":"vm041","bootMode":"UEFI","role":"master","templateRefs":[{"name":"ibi-node-templates-v1","namespace":"ocm"}]}]} {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580"}
2026-03-06T09:50:01Z INFO clusterinstance-webhook validating update request {"name": "vm041-clusterinstance", "namespace": "vm041", "resourceVersion": "259971152"}
2026-03-06T09:50:01Z INFO clusterinstance-webhook Spec change permission level: no changes allowed {"name": "vm041-clusterinstance", "namespace": "vm041", "resourceVersion": "259971152"}
2026-03-06T09:50:01Z INFO clusterinstance-webhook Provisioning or Cluster Reinstall is in progress - no spec changes allowed {"name": "vm041-clusterinstance", "namespace": "vm041", "resourceVersion": "259971152"}
2026-03-06T09:50:02.003Z INFO ClusterInstanceController controller/clusterinstance_controller.go:310 Updating ObservedGeneration to 1 {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580"}
2026-03-06T09:50:02.014Z INFO ClusterInstanceController controller/clusterinstance_controller.go:154 Finished reconciling ClusterInstance {"name": "vm041-clusterinstance", "namespace": "vm041", "version": "259969580"}