Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-30086

CAPM3 does not start

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • MCE 2.11.0
    • MCE 2.11.0
    • Installer
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • False
    • Installer Train 37 - 4
    • Important
    • Approved
    • None

      Description of problem:

      After enabling CAPM3, the MCE is stuck in Progressing status with this error in the MCE operator pod logs.

      2026-02-12T14:27:36.419Z	INFO	reconcile	Reconciling MultiClusterEngine
2026-02-12T14:27:36.440Z	INFO	overrides	Found overrides from environment variables set by OPERAND_IMAGE_ prefix
2026-02-12T14:27:38.082Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "clusterclasses.cluster.x-k8s.io"}
2026-02-12T14:27:38.202Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "clusterresourcesetbindings.addons.cluster.x-k8s.io"}
2026-02-12T14:27:38.283Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "clusterresourcesets.addons.cluster.x-k8s.io"}
2026-02-12T14:27:38.584Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "clusters.cluster.x-k8s.io"}
2026-02-12T14:27:39.179Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "extensionconfigs.runtime.cluster.x-k8s.io"}
2026-02-12T14:27:39.479Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "machinedeployments.cluster.x-k8s.io"}
2026-02-12T14:27:39.777Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "machinedrainrules.cluster.x-k8s.io"}
2026-02-12T14:27:39.878Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "machinehealthchecks.cluster.x-k8s.io"}
2026-02-12T14:27:39.985Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "machinepools.cluster.x-k8s.io"}
2026-02-12T14:27:40.278Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "machines.cluster.x-k8s.io"}
2026-02-12T14:27:40.380Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "machinesets.cluster.x-k8s.io"}
2026-02-12T14:27:40.484Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "ipaddresses.ipam.metal3.io"}
2026-02-12T14:27:40.495Z	ERROR	reconcile	Failed to apply CRD	{"CRD": "ipaddresses.ipam.metal3.io", "error": "error updating CRD 'ipaddresses.ipam.metal3.io': CustomResourceDefinition.apiextensions.k8s.io \"ipaddresses.ipam.metal3.io\" is invalid: [spec.conversion.strategy: Required value, spec.conversion.webhookClientConfig: Forbidden: should not be set when strategy is not set to Webhook]"}
github.com/stolostron/backplane-operator/controllers.(*MultiClusterEngineReconciler).Reconcile
	/workspace/controllers/backplaneconfig_controller.go:475
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
	/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.4/pkg/internal/controller/controller.go:116
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.4/pkg/internal/controller/controller.go:303
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.4/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.4/pkg/internal/controller/controller.go:224
2026-02-12T14:27:40.496Z	INFO	reconcile	Updating status
2026-02-12T14:27:40.578Z	INFO	Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes reqeueuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler	{"controller": "multiclusterengine", "controllerGroup": "multicluster.openshift.io", "controllerKind": "MultiClusterEngine", "MultiClusterEngine": {"name":"multiclusterengine"}, "namespace": "", "name": "multiclusterengine", "reconcileID": "60bc3c27-dc3f-4e08-b33e-bc84e8574d4a"}
2026-02-12T14:27:40.578Z	ERROR	Reconciler error	{"controller": "multiclusterengine", "controllerGroup": "multicluster.openshift.io", "controllerKind": "MultiClusterEngine", "MultiClusterEngine": {"name":"multiclusterengine"}, "namespace": "", "name": "multiclusterengine", "reconcileID": "60bc3c27-dc3f-4e08-b33e-bc84e8574d4a", "error": "error updating CRD 'ipaddresses.ipam.metal3.io': CustomResourceDefinition.apiextensions.k8s.io \"ipaddresses.ipam.metal3.io\" is invalid: [spec.conversion.strategy: Required value, spec.conversion.webhookClientConfig: Forbidden: should not be set when strategy is not set to Webhook]"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.4/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.4/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.4/pkg/internal/controller/controller.go:224

      The above error is seen on a used cluster. On a new cluster, things still get stuck but I see this instead looping over and over.

      2026-02-12T14:49:55.983Z	INFO	reconcile	Reconciling MultiClusterEngine
2026-02-12T14:49:56.005Z	INFO	overrides	Found overrides from environment variables set by OPERAND_IMAGE_ prefix
2026-02-12T14:49:58.045Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "clusterclasses.cluster.x-k8s.io"}
2026-02-12T14:49:58.222Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "clusterresourcesetbindings.addons.cluster.x-k8s.io"}
2026-02-12T14:49:58.248Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "clusterresourcesets.addons.cluster.x-k8s.io"}
2026-02-12T14:49:58.420Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "clusters.cluster.x-k8s.io"}
2026-02-12T14:49:58.704Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "extensionconfigs.runtime.cluster.x-k8s.io"}
2026-02-12T14:49:58.733Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "machinedeployments.cluster.x-k8s.io"}
2026-02-12T14:49:58.811Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "machinedrainrules.cluster.x-k8s.io"}
2026-02-12T14:49:58.906Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "machinehealthchecks.cluster.x-k8s.io"}
2026-02-12T14:49:59.105Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "machinepools.cluster.x-k8s.io"}
2026-02-12T14:49:59.204Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "machines.cluster.x-k8s.io"}
2026-02-12T14:49:59.405Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "machinesets.cluster.x-k8s.io"}
2026-02-12T14:49:59.570Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "metal3clusters.infrastructure.cluster.x-k8s.io"}
2026-02-12T14:49:59.704Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "metal3dataclaims.infrastructure.cluster.x-k8s.io"}
2026-02-12T14:49:59.810Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "metal3datas.infrastructure.cluster.x-k8s.io"}
2026-02-12T14:50:00.012Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "metal3datatemplates.infrastructure.cluster.x-k8s.io"}
2026-02-12T14:50:00.145Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "metal3machines.infrastructure.cluster.x-k8s.io"}
2026-02-12T14:50:00.163Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "metal3machinetemplates.infrastructure.cluster.x-k8s.io"}
2026-02-12T14:50:00.209Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "metal3remediations.infrastructure.cluster.x-k8s.io"}
2026-02-12T14:50:00.220Z	DEBUG	reconcile	Preserved caBundle in CRD update	{"Name": "metal3remediationtemplates.infrastructure.cluster.x-k8s.io"}
2026-02-12T14:50:07.504Z	INFO	reconcile	Check if ManagedCluster CR exists
2026-02-12T14:50:07.510Z	INFO	reconcile	Setting annotations on ManagedCluster CR
2026-02-12T14:50:07.510Z	INFO	reconcile	Removing NodeSelector annotation
2026-02-12T14:50:07.510Z	INFO	reconcile	Adding Tolerations annotation
2026-02-12T14:50:07.510Z	INFO	reconcile	Updating ManagedCluster CR
2026-02-12T14:50:07.563Z	INFO	reconcile	Reconcile completed. Requeuing after 5m0s
2026-02-12T14:50:07.563Z	INFO	reconcile	Updating status

      Version-Release number of selected component (if applicable):

            "acm_image": "quay.io:443/acm-d/acm-dev-catalog:2.16.0-DOWNSTREAM-2026-02-12-05-30-42",
      "mce_image": "quay.io:443/acm-d/mce-dev-catalog:2.11.0-DOWNSTREAM-2026-02-11-23-55-43",

      How reproducible:

      Always

      Steps to Reproduce:

      1. Install builds above
      2. Enable CAPI and CAPM3

      Actual results:

      What I describe above

      Expected results:

      Things should work

              cwall@redhat.com Cameron Wall
              rh-ee-msmigiel Matthew Smigielski
              Matthew Smigielski Matthew Smigielski
              ACM QE Team
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated: