Critical Description of problem:
I'm trying to enable cluster API on a hub running on IPV6, the build
MEC 2.10.0-*DOW*NSTREAM-2025-10-06-01-18-00 was installable on Install tester's env which I think is running on IPV4 usually. but capm3-controller-manager crashes as shown in below on my disconnected IPV6 env
below are the pod description and log
# oc describe pod -n multicluster-engine capm3-controller-manager-6cb966fd8b-pww97
Name: capm3-controller-manager-6cb966fd8b-pww97
Namespace: multicluster-engine
Priority: 0
Service Account: capm3-manager
Node: d16-h14-000-r650/fc00:1006::7
Start Time: Thu, 09 Oct 2025 20:17:02 +0000
Labels: cluster.x-k8s.io/provider=infrastructure-metal3
control-plane=controller-manager
controller-tools.k8s.io=1.0
ocm-antiaffinity-selector=capm3-controller-manager
pod-template-hash=6cb966fd8b
Annotations: k8s.ovn.org/pod-networks:
{"default":{"ip_addresses":["fd01:0:0:3::71/64"],"mac_address":"0a:58:7e:a4:ac:3d","gateway_ips":["fd01:0:0:3::1"],"routes":[{"dest":"fd01...
k8s.v1.cni.cncf.io/network-status:
[{
"name": "ovn-kubernetes",
"interface": "eth0",
"ips": [
"fd01:0:0:3::71"
],
"mac": "0a:58:7e:a4:ac:3d",
"default": true,
"dns": {}
}]
openshift.io/scc: restricted-v2
seccomp.security.alpha.kubernetes.io/pod: runtime/default
security.openshift.io/validated-scc-subject-type: user
Status: Running
SeccompProfile: RuntimeDefault
IP: fd01:0:0:3::71
IPs:
IP: fd01:0:0:3::71
Controlled By: ReplicaSet/capm3-controller-manager-6cb966fd8b
Containers:
manager:
Container ID: cri-o://4380c54c82fd5fec0fd87cef0b314b35e06dfe6ba46e464ed0776146bd942485
Image: registry.redhat.io/openshift4/ose-baremetal-cluster-api-controllers-rhel9@sha256:ce351e3808d9f8cf69433cd090a3c6e46ebaece20c21983fe5e68eff762dd6de
Image ID: registry.redhat.io/openshift4/ose-baremetal-cluster-api-controllers-rhel9@sha256:7481c809c010fd4fcbafb509df96ba07905d55fcf51cedf801b0004bf5ca8022
Ports: 9443/TCP, 9440/TCP, 8443/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Command:
/manager
Args:
--webhook-port=9443
--enableBMHNameBasedPreallocation=false
--diagnostics-address=:8443
--insecure-diagnostics=false
--tls-min-version=VersionTLS13
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Message: 3remediation" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3Remediation"
I1010 11:54:26.296026 1 controller.go:270] "All workers finished" controller="metal3machinetemplate" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3MachineTemplate"
I1010 11:54:26.296028 1 controller.go:270] "All workers finished" controller="metal3datatemplate" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3DataTemplate"
I1010 11:54:26.296035 1 controller.go:270] "All workers finished" controller="clustercache" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster"
I1010 11:54:26.296044 1 controller.go:270] "All workers finished" controller="metal3machine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3Machine"
I1010 11:54:26.296077 1 internal.go:550] "Stopping and waiting for caches"
E1010 11:54:26.298587 1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"IPClaim\" in version \"ipam.metal3.io/v1alpha1\"" logger="controller-runtime.source.EventHandler" kind="IPClaim.ipam.metal3.io"
I1010 11:54:26.298643 1 internal.go:554] "Stopping and waiting for webhooks"
I1010 11:54:26.298728 1 server.go:249] "Shutting down webhook server with timeout of 1 minute" logger="controller-runtime.webhook"
I1010 11:54:26.298793 1 internal.go:557] "Stopping and waiting for HTTP servers"
I1010 11:54:26.298810 1 server.go:254] "Shutting down metrics server with timeout of 1 minute" logger="controller-runtime.metrics"
I1010 11:54:26.298833 1 server.go:68] "shutting down server" name="health probe" addr="[::]:9440"
I1010 11:54:26.298879 1 internal.go:561] "Wait completed, proceeding to shutdown the manager"
E1010 11:54:26.298894 1 main.go:218] "problem running manager" err="failed to wait for metal3data caches to sync kind source: *v1alpha1.IPClaim: timed out waiting for cache to be synced for Kind *v1alpha1.IPClaim" logger="setup"
Exit Code: 1
Started: Fri, 10 Oct 2025 11:52:26 +0000
Finished: Fri, 10 Oct 2025 11:54:26 +0000
Ready: False
Restart Count: 135
Liveness: http-get http://:healthz/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:healthz/readyz delay=0s timeout=1s period=10s #success=1 #failure=3
Environment Variables from:
capm3-capm3fasttrack-configmap ConfigMap Optional: false
Environment:
POD_NAMESPACE: multicluster-engine (v1:metadata.namespace)
Mounts:
/tmp/k8s-webhook-server/serving-certs from cert (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9z85s (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
cert:
Type: Secret (a volume populated by a Secret)
SecretName: capm3-webhook-service-cert
Optional: false
kube-api-access-9z85s:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
Optional: false
DownwardAPI: true
ConfigMapName: openshift-service-ca.crt
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node-role.kubernetes.io/infra:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 26m (x66 over 15h) kubelet Readiness probe failed: Get "http://[fd01:0:0:3::71]:9440/readyz": dial tcp [fd01:0:0:3::71]:9440: connect: connection refused
Normal Created 6m40s (x136 over 15h) kubelet Created container: manager
Normal Pulled 6m40s (x135 over 15h) kubelet Container image "registry.redhat.io/openshift4/ose-baremetal-cluster-api-controllers-rhel9@sha256:ce351e3808d9f8cf69433cd090a3c6e46ebaece20c21983fe5e68eff762dd6de" already present on machine
Warning BackOff 2m (x3267 over 15h) kubelet Back-off restarting failed container manager in pod capm3-controller-manager-6cb966fd8b-pww97_multicluster-engine(b75d7296-92be-40a4-85d1-faa9c5826711)
the pod logs:
# oc logs -n multicluster-engine capm3-controller-manager-6cb966fd8b-pww97 I1010 14:15:22.297226 1 cluster_cache.go:295] "Couldn't find controller Pod metadata, the ClusterCache will always access the cluster it is running on using the regular apiserver endpoint" controller="clustercache" I1010 14:15:22.298259 1 main.go:601] "out-of-service taint" logger="setup" supported=true I1010 14:15:22.298309 1 webhook.go:188] "Registering a mutating webhook" logger="controller-runtime.builder" GVK="infrastructure.cluster.x-k8s.io/v1beta1, Kind=Metal3Cluster" path="/mutate-infrastructure-cluster-x-k8s-io-v1beta1-metal3cluster" I1010 14:15:22.298378 1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/mutate-infrastructure-cluster-x-k8s-io-v1beta1-metal3cluster" I1010 14:15:22.298403 1 webhook.go:226] "Registering a validating webhook" logger="controller-runtime.builder" GVK="infrastructure.cluster.x-k8s.io/v1beta1, Kind=Metal3Cluster" path="/validate-infrastructure-cluster-x-k8s-io-v1beta1-metal3cluster" I1010 14:15:22.298441 1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/validate-infrastructure-cluster-x-k8s-io-v1beta1-metal3cluster" I1010 14:15:22.298487 1 webhook.go:188] "Registering a mutating webhook" logger="controller-runtime.builder" GVK="infrastructure.cluster.x-k8s.io/v1beta1, Kind=Metal3Machine" path="/mutate-infrastructure-cluster-x-k8s-io-v1beta1-metal3machine" I1010 14:15:22.298521 1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/mutate-infrastructure-cluster-x-k8s-io-v1beta1-metal3machine" I1010 14:15:22.298540 1 webhook.go:226] "Registering a validating webhook" logger="controller-runtime.builder" GVK="infrastructure.cluster.x-k8s.io/v1beta1, Kind=Metal3Machine" path="/validate-infrastructure-cluster-x-k8s-io-v1beta1-metal3machine" I1010 14:15:22.298581 1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/validate-infrastructure-cluster-x-k8s-io-v1beta1-metal3machine" I1010 14:15:22.298618 1 webhook.go:188] "Registering a mutating webhook" logger="controller-runtime.builder" GVK="infrastructure.cluster.x-k8s.io/v1beta1, Kind=Metal3MachineTemplate" path="/mutate-infrastructure-cluster-x-k8s-io-v1beta1-metal3machinetemplate" I1010 14:15:22.298654 1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/mutate-infrastructure-cluster-x-k8s-io-v1beta1-metal3machinetemplate" I1010 14:15:22.298676 1 webhook.go:226] "Registering a validating webhook" logger="controller-runtime.builder" GVK="infrastructure.cluster.x-k8s.io/v1beta1, Kind=Metal3MachineTemplate" path="/validate-infrastructure-cluster-x-k8s-io-v1beta1-metal3machinetemplate" I1010 14:15:22.298707 1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/validate-infrastructure-cluster-x-k8s-io-v1beta1-metal3machinetemplate" I1010 14:15:22.298742 1 webhook.go:188] "Registering a mutating webhook" logger="controller-runtime.builder" GVK="infrastructure.cluster.x-k8s.io/v1beta1, Kind=Metal3DataTemplate" path="/mutate-infrastructure-cluster-x-k8s-io-v1beta1-metal3datatemplate" I1010 14:15:22.298775 1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/mutate-infrastructure-cluster-x-k8s-io-v1beta1-metal3datatemplate" I1010 14:15:22.298795 1 webhook.go:226] "Registering a validating webhook" logger="controller-runtime.builder" GVK="infrastructure.cluster.x-k8s.io/v1beta1, Kind=Metal3DataTemplate" path="/validate-infrastructure-cluster-x-k8s-io-v1beta1-metal3datatemplate" I1010 14:15:22.298835 1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/validate-infrastructure-cluster-x-k8s-io-v1beta1-metal3datatemplate" I1010 14:15:22.298870 1 webhook.go:188] "Registering a mutating webhook" logger="controller-runtime.builder" GVK="infrastructure.cluster.x-k8s.io/v1beta1, Kind=Metal3Data" path="/mutate-infrastructure-cluster-x-k8s-io-v1beta1-metal3data" I1010 14:15:22.298904 1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/mutate-infrastructure-cluster-x-k8s-io-v1beta1-metal3data" I1010 14:15:22.298922 1 webhook.go:226] "Registering a validating webhook" logger="controller-runtime.builder" GVK="infrastructure.cluster.x-k8s.io/v1beta1, Kind=Metal3Data" path="/validate-infrastructure-cluster-x-k8s-io-v1beta1-metal3data" I1010 14:15:22.298955 1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/validate-infrastructure-cluster-x-k8s-io-v1beta1-metal3data" I1010 14:15:22.298991 1 webhook.go:188] "Registering a mutating webhook" logger="controller-runtime.builder" GVK="infrastructure.cluster.x-k8s.io/v1beta1, Kind=Metal3DataClaim" path="/mutate-infrastructure-cluster-x-k8s-io-v1beta1-metal3dataclaim" I1010 14:15:22.299023 1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/mutate-infrastructure-cluster-x-k8s-io-v1beta1-metal3dataclaim" I1010 14:15:22.299040 1 webhook.go:226] "Registering a validating webhook" logger="controller-runtime.builder" GVK="infrastructure.cluster.x-k8s.io/v1beta1, Kind=Metal3DataClaim" path="/validate-infrastructure-cluster-x-k8s-io-v1beta1-metal3dataclaim" I1010 14:15:22.299066 1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/validate-infrastructure-cluster-x-k8s-io-v1beta1-metal3dataclaim" I1010 14:15:22.299104 1 webhook.go:188] "Registering a mutating webhook" logger="controller-runtime.builder" GVK="infrastructure.cluster.x-k8s.io/v1beta1, Kind=Metal3Remediation" path="/mutate-infrastructure-cluster-x-k8s-io-v1beta1-metal3remediation" I1010 14:15:22.299134 1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/mutate-infrastructure-cluster-x-k8s-io-v1beta1-metal3remediation" I1010 14:15:22.299152 1 webhook.go:226] "Registering a validating webhook" logger="controller-runtime.builder" GVK="infrastructure.cluster.x-k8s.io/v1beta1, Kind=Metal3Remediation" path="/validate-infrastructure-cluster-x-k8s-io-v1beta1-metal3remediation" I1010 14:15:22.299184 1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/validate-infrastructure-cluster-x-k8s-io-v1beta1-metal3remediation" I1010 14:15:22.299226 1 webhook.go:188] "Registering a mutating webhook" logger="controller-runtime.builder" GVK="infrastructure.cluster.x-k8s.io/v1beta1, Kind=Metal3RemediationTemplate" path="/mutate-infrastructure-cluster-x-k8s-io-v1beta1-metal3remediationtemplate" I1010 14:15:22.299262 1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/mutate-infrastructure-cluster-x-k8s-io-v1beta1-metal3remediationtemplate" I1010 14:15:22.299280 1 webhook.go:226] "Registering a validating webhook" logger="controller-runtime.builder" GVK="infrastructure.cluster.x-k8s.io/v1beta1, Kind=Metal3RemediationTemplate" path="/validate-infrastructure-cluster-x-k8s-io-v1beta1-metal3remediationtemplate" I1010 14:15:22.299315 1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/validate-infrastructure-cluster-x-k8s-io-v1beta1-metal3remediationtemplate" I1010 14:15:22.299356 1 webhook.go:188] "Registering a mutating webhook" logger="controller-runtime.builder" GVK="infrastructure.cluster.x-k8s.io/v1beta1, Kind=Metal3ClusterTemplate" path="/mutate-infrastructure-cluster-x-k8s-io-v1beta1-metal3clustertemplate" I1010 14:15:22.299385 1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/mutate-infrastructure-cluster-x-k8s-io-v1beta1-metal3clustertemplate" I1010 14:15:22.299402 1 webhook.go:226] "Registering a validating webhook" logger="controller-runtime.builder" GVK="infrastructure.cluster.x-k8s.io/v1beta1, Kind=Metal3ClusterTemplate" path="/validate-infrastructure-cluster-x-k8s-io-v1beta1-metal3clustertemplate" I1010 14:15:22.299431 1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/validate-infrastructure-cluster-x-k8s-io-v1beta1-metal3clustertemplate" I1010 14:15:22.299453 1 main.go:216] "starting manager" logger="setup" I1010 14:15:22.299501 1 server.go:208] "Starting metrics server" logger="controller-runtime.metrics" I1010 14:15:22.299526 1 server.go:83] "starting server" name="health probe" addr="[::]:9440" I1010 14:15:22.299594 1 server.go:191] "Starting webhook server" logger="controller-runtime.webhook" I1010 14:15:22.299779 1 controller.go:204] "Starting EventSource" controller="clustercache" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" source="kind source: *v1beta1.Cluster" I1010 14:15:22.299813 1 controller.go:204] "Starting EventSource" controller="metal3cluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3Cluster" source="kind source: *v1beta1.Cluster" I1010 14:15:22.299848 1 controller.go:204] "Starting EventSource" controller="metal3cluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3Cluster" source="kind source: *v1beta1.Metal3Cluster" I1010 14:15:22.299884 1 certwatcher.go:211] "Updated current TLS certificate" logger="controller-runtime.certwatcher" I1010 14:15:22.299948 1 server.go:242] "Serving webhook server" logger="controller-runtime.webhook" host="" port=9443 I1010 14:15:22.299951 1 controller.go:204] "Starting EventSource" controller="metal3remediation" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3Remediation" source="kind source: *v1beta1.Metal3Remediation" I1010 14:15:22.299983 1 controller.go:204] "Starting EventSource" controller="metal3machinetemplate" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3MachineTemplate" source="kind source: *v1beta1.Metal3Machine" I1010 14:15:22.300011 1 certwatcher.go:133] "Starting certificate poll+watcher" logger="controller-runtime.certwatcher" interval="10s" I1010 14:15:22.300014 1 controller.go:204] "Starting EventSource" controller="metal3machinetemplate" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3MachineTemplate" source="kind source: *v1beta1.Metal3MachineTemplate" I1010 14:15:22.300073 1 controller.go:204] "Starting EventSource" controller="metal3datatemplate" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3DataTemplate" source="kind source: *v1beta1.Metal3DataTemplate" I1010 14:15:22.300075 1 controller.go:204] "Starting EventSource" controller="metal3datatemplate" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3DataTemplate" source="kind source: *v1beta1.Metal3DataClaim" I1010 14:15:22.300115 1 controller.go:204] "Starting EventSource" controller="metal3data" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3Data" source="kind source: *v1beta1.Metal3Data" I1010 14:15:22.300115 1 controller.go:204] "Starting EventSource" controller="metal3data" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3Data" source="kind source: *v1alpha1.IPClaim" I1010 14:15:22.300156 1 controller.go:204] "Starting EventSource" controller="metal3machine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3Machine" source="kind source: *v1beta1.Machine" I1010 14:15:22.300155 1 controller.go:204] "Starting EventSource" controller="metal3machine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3Machine" source="kind source: *v1beta1.Metal3Machine" I1010 14:15:22.300163 1 controller.go:204] "Starting EventSource" controller="baremetalhost" controllerGroup="metal3.io" controllerKind="BareMetalHost" source="kind source: *v1beta1.Metal3Cluster" I1010 14:15:22.300168 1 controller.go:204] "Starting EventSource" controller="metal3machine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3Machine" source="kind source: *v1beta1.Metal3Cluster" I1010 14:15:22.300173 1 controller.go:204] "Starting EventSource" controller="baremetalhost" controllerGroup="metal3.io" controllerKind="BareMetalHost" source="kind source: *v1alpha1.BareMetalHost" I1010 14:15:22.300175 1 controller.go:204] "Starting EventSource" controller="metal3machine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3Machine" source="kind source: *v1beta1.Metal3DataClaim" I1010 14:15:22.300173 1 controller.go:204] "Starting EventSource" controller="metal3machine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3Machine" source="kind source: *v1alpha1.BareMetalHost" I1010 14:15:22.300188 1 controller.go:204] "Starting EventSource" controller="metal3machine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3Machine" source="kind source: *v1beta1.Cluster" I1010 14:15:22.300187 1 controller.go:204] "Starting EventSource" controller="metal3machine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3Machine" source="kind source: *v1beta1.Metal3Data" E1010 14:15:22.302820 1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"IPClaim\" in version \"ipam.metal3.io/v1alpha1\"" logger="controller-runtime.source.EventHandler" kind="IPClaim.ipam.metal3.io" I1010 14:15:22.400293 1 controller.go:239] "Starting Controller" controller="clustercache" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" I1010 14:15:22.400308 1 controller.go:248] "Starting workers" controller="clustercache" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" worker count=1 I1010 14:15:22.400363 1 controller.go:239] "Starting Controller" controller="metal3cluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3Cluster" I1010 14:15:22.400370 1 controller.go:248] "Starting workers" controller="metal3cluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3Cluster" worker count=10 I1010 14:15:22.403622 1 controller.go:239] "Starting Controller" controller="metal3machinetemplate" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3MachineTemplate" I1010 14:15:22.403642 1 controller.go:248] "Starting workers" controller="metal3machinetemplate" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3MachineTemplate" worker count=10 I1010 14:15:22.403629 1 controller.go:239] "Starting Controller" controller="metal3remediation" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3Remediation" I1010 14:15:22.403664 1 controller.go:239] "Starting Controller" controller="baremetalhost" controllerGroup="metal3.io" controllerKind="BareMetalHost" I1010 14:15:22.403670 1 controller.go:248] "Starting workers" controller="metal3remediation" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3Remediation" worker count=10 I1010 14:15:22.403670 1 controller.go:239] "Starting Controller" controller="metal3datatemplate" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3DataTemplate" I1010 14:15:22.403684 1 controller.go:248] "Starting workers" controller="metal3datatemplate" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3DataTemplate" worker count=10 I1010 14:15:22.403698 1 controller.go:239] "Starting Controller" controller="metal3machine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3Machine" I1010 14:15:22.403673 1 controller.go:248] "Starting workers" controller="baremetalhost" controllerGroup="metal3.io" controllerKind="BareMetalHost" worker count=10 I1010 14:15:22.403707 1 controller.go:248] "Starting workers" controller="metal3machine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3Machine" worker count=10 I1010 14:15:22.407093 1 metal3labelsync_controller.go:119] "Unknown GroupVersionKind in BareMetalHost Consumer Ref" logger="controllers.Metal3LabelSync.metal3-label-sync-controller" metal3-label-sync="openshift-machine-api/d16-h14-000-r650" groupversion="machine.openshift.io/v1beta1, Kind=Machine" I1010 14:15:22.407211 1 metal3labelsync_controller.go:119] "Unknown GroupVersionKind in BareMetalHost Consumer Ref" logger="controllers.Metal3LabelSync.metal3-label-sync-controller" metal3-label-sync="openshift-machine-api/d16-h11-000-r650" groupversion="machine.openshift.io/v1beta1, Kind=Machine" I1010 14:15:22.407216 1 metal3labelsync_controller.go:119] "Unknown GroupVersionKind in BareMetalHost Consumer Ref" logger="controllers.Metal3LabelSync.metal3-label-sync-controller" metal3-label-sync="openshift-machine-api/d16-h10-000-r650" groupversion="machine.openshift.io/v1beta1, Kind=Machine" I1010 14:15:22.875549 1 server.go:247] "Serving metrics server" logger="controller-runtime.metrics" bindAddress=":8443" secure=true E1010 14:15:32.304144 1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"IPClaim\" in version \"ipam.metal3.io/v1alpha1\"" logger="controller-runtime.source.EventHandler" kind="IPClaim.ipam.metal3.io" E1010 14:15:42.303110 1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"IPClaim\" in version \"ipam.metal3.io/v1alpha1\"" logger="controller-runtime.source.EventHandler" kind="IPClaim.ipam.metal3.io" E1010 14:15:52.303334 1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"IPClaim\" in version \"ipam.metal3.io/v1alpha1\"" logger="controller-runtime.source.EventHandler" kind="IPClaim.ipam.metal3.io" E1010 14:16:02.303878 1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"IPClaim\" in version \"ipam.metal3.io/v1alpha1\"" logger="controller-runtime.source.EventHandler" kind="IPClaim.ipam.metal3.io" E1010 14:16:12.304435 1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"IPClaim\" in version \"ipam.metal3.io/v1alpha1\"" logger="controller-runtime.source.EventHandler" kind="IPClaim.ipam.metal3.io" E1010 14:16:22.304115 1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"IPClaim\" in version \"ipam.metal3.io/v1alpha1\"" logger="controller-runtime.source.EventHandler" kind="IPClaim.ipam.metal3.io" E1010 14:16:32.302719 1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"IPClaim\" in version \"ipam.metal3.io/v1alpha1\"" logger="controller-runtime.source.EventHandler" kind="IPClaim.ipam.metal3.io" E1010 14:16:42.305444 1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"IPClaim\" in version \"ipam.metal3.io/v1alpha1\"" logger="controller-runtime.source.EventHandler" kind="IPClaim.ipam.metal3.io" E1010 14:16:52.302835 1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"IPClaim\" in version \"ipam.metal3.io/v1alpha1\"" logger="controller-runtime.source.EventHandler" kind="IPClaim.ipam.metal3.io" E1010 14:17:02.303670 1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"IPClaim\" in version \"ipam.metal3.io/v1alpha1\"" logger="controller-runtime.source.EventHandler" kind="IPClaim.ipam.metal3.io" E1010 14:17:12.305042 1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"IPClaim\" in version \"ipam.metal3.io/v1alpha1\"" logger="controller-runtime.source.EventHandler" kind="IPClaim.ipam.metal3.io" E1010 14:17:22.300392 1 controller.go:216] "Could not wait for Cache to sync" err="failed to wait for metal3data caches to sync kind source: *v1alpha1.IPClaim: timed out waiting for cache to be synced for Kind *v1alpha1.IPClaim" controller="metal3data" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3Data" source="kind source: *v1alpha1.IPClaim" I1010 14:17:22.300450 1 internal.go:538] "Stopping and waiting for non leader election runnables" I1010 14:17:22.300460 1 internal.go:542] "Stopping and waiting for leader election runnables" I1010 14:17:22.300481 1 controller.go:268] "Shutdown signal received, waiting for all workers to finish" controller="baremetalhost" controllerGroup="metal3.io" controllerKind="BareMetalHost" I1010 14:17:22.300488 1 controller.go:268] "Shutdown signal received, waiting for all workers to finish" controller="metal3machine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3Machine" I1010 14:17:22.300495 1 controller.go:268] "Shutdown signal received, waiting for all workers to finish" controller="metal3datatemplate" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3DataTemplate" I1010 14:17:22.300500 1 controller.go:268] "Shutdown signal received, waiting for all workers to finish" controller="metal3remediation" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3Remediation" I1010 14:17:22.300506 1 controller.go:268] "Shutdown signal received, waiting for all workers to finish" controller="metal3machinetemplate" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3MachineTemplate" I1010 14:17:22.300511 1 controller.go:268] "Shutdown signal received, waiting for all workers to finish" controller="metal3cluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3Cluster" I1010 14:17:22.300517 1 controller.go:268] "Shutdown signal received, waiting for all workers to finish" controller="clustercache" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" I1010 14:17:22.300567 1 controller.go:270] "All workers finished" controller="metal3cluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3Cluster" I1010 14:17:22.300573 1 controller.go:270] "All workers finished" controller="clustercache" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" I1010 14:17:22.300568 1 controller.go:270] "All workers finished" controller="baremetalhost" controllerGroup="metal3.io" controllerKind="BareMetalHost" I1010 14:17:22.300574 1 controller.go:270] "All workers finished" controller="metal3machinetemplate" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3MachineTemplate" I1010 14:17:22.300587 1 controller.go:270] "All workers finished" controller="metal3machine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3Machine" I1010 14:17:22.300586 1 controller.go:270] "All workers finished" controller="metal3remediation" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3Remediation" I1010 14:17:22.300586 1 controller.go:270] "All workers finished" controller="metal3datatemplate" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="Metal3DataTemplate" I1010 14:17:22.300629 1 internal.go:550] "Stopping and waiting for caches" E1010 14:17:22.303631 1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"IPClaim\" in version \"ipam.metal3.io/v1alpha1\"" logger="controller-runtime.source.EventHandler" kind="IPClaim.ipam.metal3.io" I1010 14:17:22.303689 1 internal.go:554] "Stopping and waiting for webhooks" I1010 14:17:22.303718 1 server.go:249] "Shutting down webhook server with timeout of 1 minute" logger="controller-runtime.webhook" I1010 14:17:22.303765 1 internal.go:557] "Stopping and waiting for HTTP servers" I1010 14:17:22.303781 1 server.go:254] "Shutting down metrics server with timeout of 1 minute" logger="controller-runtime.metrics" I1010 14:17:22.303787 1 server.go:68] "shutting down server" name="health probe" addr="[::]:9440" I1010 14:17:22.303838 1 internal.go:561] "Wait completed, proceeding to shutdown the manager" E1010 14:17:22.303853 1 main.go:218] "problem running manager" err="failed to wait for metal3data caches to sync kind source: *v1alpha1.IPClaim: timed out waiting for cache to be synced for Kind *v1alpha1.IPClaim" logger="setup"
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
- Install MCE downstream build 2.10.0-*DOW*NSTREAM-2025-10-06-01-18-00
- enable all cluster-api pre-view features in MCE
- all cluster-api pod should be running
Actual results:
all other cluster-api pods are running, but capm3-controller-manager crashes
# oc get pod -n multicluster-engine | grep ca capa-controller-manager-799f5d7874-l6zt6 1/1 Running 0 18h capi-controller-manager-6b8bb57446-m2k7v 1/1 Running 0 18h capm3-controller-manager-6cb966fd8b-pww97 0/1 CrashLoopBackOff 160 (7s ago) 18h capoa-bootstrap-controller-manager-68866c998f-wccpj 1/1 Running 0 18h capoa-controlplane-controller-manager-7dbd469c89-tswqw 1/1 Running 0 18h mce-capi-webhook-config-76bd49c444-t89hr
