Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-29732

hive-operator pod failing to start

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • MCE 2.11.0
    • MCE 2.11.0
    • Hive, Installer
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • False
    • Installer Train 36 - 3
    • Critical
    • None

      Description of problem:

      The hive-operator pod is failing to start due to this error.

      time="2026-02-04T13:58:37Z" level=info msg="Version: openshift/hive "
time="2026-02-04T13:58:37Z" level=info msg="Starting /healthz and /readyz endpoints"
time="2026-02-04T13:58:37Z" level=info msg="generated leader election ID" id=79c580a1-f240-4c49-a36b-ea38eadb676c
I0204 13:58:37.736579       1 leaderelection.go:257] attempting to acquire leader lease multicluster-engine/hive-operator-leader...
I0204 13:58:37.752064       1 leaderelection.go:271] successfully acquired lease multicluster-engine/hive-operator-leader
time="2026-02-04T13:58:37Z" level=info msg="became leader" id=79c580a1-f240-4c49-a36b-ea38eadb676c
time="2026-02-04T13:58:37Z" level=info msg="Registering Components."
time="2026-02-04T13:58:37Z" level=info msg="hive operator NS: multicluster-engine"
time="2026-02-04T13:58:37Z" level=info msg="Starting the Cmd."
time="2026-02-04T13:58:37Z" level=info msg="started metrics calculator goroutine"
time="2026-02-04T13:58:37Z" level=info msg="calculating metrics for all Hive" controller=metrics
time="2026-02-04T13:58:37Z" level=info msg="reconcile complete" controller=metrics elapsedMillis=0 elapsedMillisGT=0 outcome=unspecified
E0204 13:58:37.885257       1 reflector.go:205] "Failed to watch" err="failed to list *v1.APIServer: apiservers.config.openshift.io is forbidden: User \"system:serviceaccount:multicluster-engine:hive-operator\" cannot list resource \"apiservers\" in API group \"config.openshift.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go@v0.34.2/tools/cache/reflector.go:290" type="*v1.APIServer"
E0204 13:58:38.968943       1 reflector.go:205] "Failed to watch" err="failed to list *v1.APIServer: apiservers.config.openshift.io is forbidden: User \"system:serviceaccount:multicluster-engine:hive-operator\" cannot list resource \"apiservers\" in API group \"config.openshift.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go@v0.34.2/tools/cache/reflector.go:290" type="*v1.APIServer"
E0204 13:58:40.653598       1 reflector.go:205] "Failed to watch" err="failed to list *v1.APIServer: apiservers.config.openshift.io is forbidden: User \"system:serviceaccount:multicluster-engine:hive-operator\" cannot list resource \"apiservers\" in API group \"config.openshift.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go@v0.34.2/tools/cache/reflector.go:290" type="*v1.APIServer"
E0204 13:58:45.056495       1 reflector.go:205] "Failed to watch" err="failed to list *v1.APIServer: apiservers.config.openshift.io is forbidden: User \"system:serviceaccount:multicluster-engine:hive-operator\" cannot list resource \"apiservers\" in API group \"config.openshift.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go@v0.34.2/tools/cache/reflector.go:290" type="*v1.APIServer"
E0204 13:58:52.436516       1 reflector.go:205] "Failed to watch" err="failed to list *v1.APIServer: apiservers.config.openshift.io is forbidden: User \"system:serviceaccount:multicluster-engine:hive-operator\" cannot list resource \"apiservers\" in API group \"config.openshift.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go@v0.34.2/tools/cache/reflector.go:290" type="*v1.APIServer"
E0204 13:59:14.945916       1 reflector.go:205] "Failed to watch" err="failed to list *v1.APIServer: apiservers.config.openshift.io is forbidden: User \"system:serviceaccount:multicluster-engine:hive-operator\" cannot list resource \"apiservers\" in API group \"config.openshift.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go@v0.34.2/tools/cache/reflector.go:290" type="*v1.APIServer"
E0204 13:59:51.507222       1 reflector.go:205] "Failed to watch" err="failed to list *v1.APIServer: apiservers.config.openshift.io is forbidden: User \"system:serviceaccount:multicluster-engine:hive-operator\" cannot list resource \"apiservers\" in API group \"config.openshift.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go@v0.34.2/tools/cache/reflector.go:290" type="*v1.APIServer"
E0204 14:00:30.587055       1 reflector.go:205] "Failed to watch" err="failed to list *v1.APIServer: apiservers.config.openshift.io is forbidden: User \"system:serviceaccount:multicluster-engine:hive-operator\" cannot list resource \"apiservers\" in API group \"config.openshift.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go@v0.34.2/tools/cache/reflector.go:290" type="*v1.APIServer"
time="2026-02-04T14:00:37Z" level=info msg="calculating metrics for all Hive" controller=metrics
time="2026-02-04T14:00:37Z" level=info msg="reconcile complete" controller=metrics elapsedMillis=0 elapsedMillisGT=0 outcome=unspecified
time="2026-02-04T14:00:37Z" level=error msg="Could not wait for Cache to sync" controller=hive-controller error="failed to wait for hive-controller caches to sync: timed out waiting for cache to be synced for Kind *v1.DaemonSet"
[controller-runtime] log.SetLogger(...) was never called; logs will not be displayed.
Detected at:
	>  goroutine 245 [running]:
	>  runtime/debug.Stack()
	>  	runtime/debug/stack.go:26 +0x5e
	>  sigs.k8s.io/controller-runtime/pkg/log.eventuallyFulfillRoot()
	>  	sigs.k8s.io/controller-runtime@v0.22.3/pkg/log/log.go:60 +0xcd
	>  sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error(0xc000412840, {0x58cd9c0, 0xc0018d77c0}, {0x51adcfd, 0x21}, {0x0, 0x0, 0x0})
	>  	sigs.k8s.io/controller-runtime@v0.22.3/pkg/log/deleg.go:139 +0x5d
	>  github.com/go-logr/logr.Logger.Error({{0x5915ac0?, 0xc000412840?}, 0x0?}, {0x58cd9c0, 0xc0018d77c0}, {0x51adcfd, 0x21}, {0x0, 0x0, 0x0})
	>  	github.com/go-logr/logr@v1.4.3/logr.go:301 +0x145
	>  sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1()
	>  	sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/source/kind.go:76 +0x1a9
	>  k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func1({0x590b3d8?, 0xc00088fa90?}, 0xc0008fbe80?)
	>  	k8s.io/apimachinery@v0.34.2/pkg/util/wait/loop.go:53 +0x62
	>  k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext({0x590b3d8, 0xc00088fa90}, {0x58ef2f8, 0xc0004c9920}, 0x1, 0x0, 0xc000c29fa8)
	>  	k8s.io/apimachinery@v0.34.2/pkg/util/wait/loop.go:54 +0x115
	>  k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel({0x590b3d8, 0xc00088fa90}, 0x0?, 0x1, 0xc0008fbfa8)
	>  	k8s.io/apimachinery@v0.34.2/pkg/util/wait/poll.go:33 +0x56
	>  sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1()
	>  	sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/source/kind.go:64 +0xba
	>  created by sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start in goroutine 232
	>  	sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/source/kind.go:56 +0x194
I0204 14:00:37.875653       1 reflector.go:568] "Warning: watch ended with error" reflector="k8s.io/client-go@v0.34.2/tools/cache/reflector.go:290" type="*v1.Proxy" err="an error on the server (\"unable to decode an event from the watch stream: context canceled\") has prevented the request from succeeding"
I0204 14:00:37.875727       1 reflector.go:568] "Warning: watch ended with error" reflector="k8s.io/client-go@v0.34.2/tools/cache/reflector.go:290" type="*v1.CustomResourceDefinition" err="an error on the server (\"unable to decode an event from the watch stream: context canceled\") has prevented the request from succeeding"
time="2026-02-04T14:00:37Z" level=error msg="error running manager" error="failed to wait for hive-controller caches to sync: timed out waiting for cache to be synced for Kind *v1.DaemonSet"
time="2026-02-04T14:00:37Z" level=info msg="leader lost" id=79c580a1-f240-4c49-a36b-ea38eadb676c

      Version-Release number of selected component (if applicable):

      This is a regression that started around this build:
      quay.io:443/acm-d/mce-dev-catalog:2.11.0-DOWNSTREAM-2026-02-04-05-25-54

      How reproducible:

      Always

      Steps to Reproduce:

      1. Install the build above

      Actual results:

      The hive-operator pod fails to start

      Expected results:

      Pod should start

      -------------------------------------------------------------------------------------------------------
      QE Hand Off Template (fill out when moving to Review) 2/5/26:

      Summary of the Work:
      What was implemented or fixed? Include a brief description of the problem (if applicable) and how it was addressed.
      e.g., "Updated the UI to show validation errors for the form. The previous implementation did not surface backend validation issues."

      Key Areas to Verify:

      1. What functionality should QE focus on? List what was tested or what is most important to validate.
      2. Ensure the new validation messages appear for required fields
      3. Confirm the workflow still completes as expected after validation fixes
      4. Any edge cases or high-risk areas touched by the change

      Fix or Feature Availability:
      When will this be available in a build?
      Code merged on: YYYY-MM-DD
      Expected downstream build tag (if known): example-build-tag
      (Optional) Related PR(s): Link

              dbennett@redhat.com Disaiah Bennett
              rh-ee-msmigiel Matthew Smigielski
              Matthew Smigielski Matthew Smigielski
              ACM QE Team
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: