Critical Description of problem:
When attempting to install the builds build, I get the following errors in the MCH and MCE pods. The install is stuck as both the MCH and MCE are in a pending state.
MCH operator pod:
2026-01-02T12:01:56.271Z ERROR reconcile failed to get resource {"Kind": "ClusterManagementAddOn", "Name": "application-manager", "error": "no matches for kind \"ClusterManagementAddOn\" in version \"addon.open-cluster-management.io/v1alpha1\""}
github.com/stolostron/multiclusterhub-operator/controllers.(*MultiClusterHubReconciler).logAndSetCondition
/workspace/controllers/templates.go:245
github.com/stolostron/multiclusterhub-operator/controllers.(*MultiClusterHubReconciler).applyTemplate
/workspace/controllers/templates.go:84
github.com/stolostron/multiclusterhub-operator/controllers.(*MultiClusterHubReconciler).ensureComponent
/workspace/controllers/components.go:210
github.com/stolostron/multiclusterhub-operator/controllers.(*MultiClusterHubReconciler).Reconcile
/workspace/controllers/reconcile.go:290
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.4/pkg/internal/controller/controller.go:116
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.4/pkg/internal/controller/controller.go:303
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.4/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.4/pkg/internal/controller/controller.go:224
MCE operator pod:
2026-01-02T11:59:49.586Z ERROR Reconciler error {"controller": "multiclusterengine", "controllerGroup": "multicluster.openshift.io", "controllerKind": "MultiClusterEngine", "MultiClusterEngine": {"name":"multiclusterengine"}, "namespace": "", "name": "multiclusterengine", "reconcileID": "26528502-1d8e-4607-a364-b4ece9cb4e2b", "error": "error applying object Name: work-manager Kind: ClusterManagementAddOn: no matches for kind \"ClusterManagementAddOn\" in version \"addon.open-cluster-management.io/v1alpha1\"", "errorVerbose": "no matches for kind \"ClusterManagementAddOn\" in version \"addon.open-cluster-management.io/v1alpha1\"\nerror applying object Name: work-manager Kind: ClusterManagementAddOn\ngithub.com/stolostron/backplane-operator/controllers.(*MultiClusterEngineReconciler).ensureCustomResources\n\t/workspace/controllers/backplaneconfig_controller.go:1883\ngithub.com/stolostron/backplane-operator/controllers.(*MultiClusterEngineReconciler).DeployAlwaysSubcomponents\n\t/workspace/controllers/backplaneconfig_controller.go:859\ngithub.com/stolostron/backplane-operator/controllers.(*MultiClusterEngineReconciler).Reconcile\n\t/workspace/controllers/backplaneconfig_controller.go:472\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.4/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.4/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.4/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.4/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1700"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.4/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.4/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.4/pkg/internal/controller/controller.go:224
Version-Release number of selected component (if applicable):
2.16.0-DOWNSTREAM-2026-01-01-23-38-03
2.11.0-DOWNSTREAM-2026-01-01-23-38-29
How reproducible:
Always
Steps to Reproduce:
- Install builds above
Actual results:
Install is stuck
Expected results:
Install should work
-------------------------------------------------------------------------------------------------------
QE Hand Off Template (fill out when moving to Review) 1/5/26:
Summary of the Work:
What was implemented or fixed? Include a brief description of the problem (if applicable) and how it was addressed.
e.g., "Updated the UI to show validation errors for the form. The previous implementation did not surface backend validation issues."
- This was a fix that was implemented on the Server Foundation side of things. Related slack thread: https://redhat-internal.slack.com/archives/CTZTHFQRH/p1767102128378869
Key Areas to Verify:
- What functionality should QE focus on? List what was tested or what is most important to validate.
- Ensure the new validation messages appear for required fields
- Confirm the workflow still completes as expected after validation fixes
- Any edge cases or high-risk areas touched by the change
- No changes were made on the Installer side. The test only needs to confirm that the installation functions correctly.
Fix or Feature Availability:
When will this be available in a build?
Code merged on: 2026-01-04
Expected downstream build tag (if known):
ACM: 2.16.0-DOWNSTREAM-2026-01-05-05-16-31
MCE: 2.11.0-DOWNSTREAM-2026-01-05-05-19-55
(Optional) Related PR(s): https://github.com/stolostron/backplane-operator/pull/2413
