❯ make test-e2e GOBIN=/Users/anahas/Github/forks/kueue-operator/kueue-operator/bin GO111MODULE=on go install -mod=mod github.com/onsi/ginkgo/v2/ginkgo@v2.1.4 /Users/anahas/Github/forks/kueue-operator/kueue-operator/bin/ginkgo --keep-going --flake-attempts=3 --label-filter="!disruptive" -v ./test/e2e/... Starting kueue operator suite Running Suite: e2e suite - /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e ============================================================================================ Random Seed: 1772034796 Will run 51 of 53 specs ------------------------------ [BeforeSuite] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_suite_test.go:68 STEP: Deploying Kueue operand for all test suites @ 02/25/26 12:53:21.917 I0225 12:53:21.917673 45539 e2e_operator_test.go:1648] Creating Kueue instance I0225 12:53:23.006921 45539 e2e_operator_test.go:1671] Checking pod: kueue-controller-manager-5db787648b-khrrc, phase: Running, deletionTS: STEP: Waiting for all Kueue CRDs to be registered @ 02/25/26 12:53:23.006 I0225 12:53:25.835517 45539 e2e_operator_test.go:1739] All 9 Kueue CRDs are registered and established STEP: Waiting for webhook to handle requests successfully @ 02/25/26 12:53:25.835 I0225 12:53:26.201819 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-ggdc4 I0225 12:53:26.377120 45539 e2e_operator_test.go:1494] Webhook success 1/3 I0225 12:53:29.741211 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-tx2bm I0225 12:53:29.929014 45539 e2e_operator_test.go:1494] Webhook success 2/3 I0225 12:53:33.288426 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-6ds68 [BeforeSuite] PASSED [11.548 seconds] I0225 12:53:33.456714 45539 e2e_operator_test.go:1491] Webhook stable after 3 consecutive successes ------------------------------ Kueue Operator when installs operator pods should be ready [operator] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:90 I0225 12:53:33.952999 45539 e2e_operator_test.go:102] Checking pod: openshift-kueue-operator-56945f7b7c-5kq48, phase: Running, deletionTS: • [0.496 seconds] ------------------------------ Kueue Operator when installs kueue pods should be ready [operator] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:110 I0225 12:53:34.132445 45539 e2e_operator_test.go:122] Checking pod: kueue-controller-manager-5db787648b-khrrc, phase: Running, deletionTS: • [0.179 seconds] ------------------------------ Kueue Operator when installs should set ReadyReplicas in operator status and handle degraded condition [operator] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:137 • [0.334 seconds] ------------------------------ Kueue Operator when installs kueue operator deployment should contain priority class [operator] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:173 • [0.175 seconds] ------------------------------ Kueue Operator when installs kueue deployment should contain priority class and have no resource limits set [operator] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:187 • [0.173 seconds] ------------------------------ Kueue Operator when installs Verifying that no v1alpha Kueue CRDs are installed [operator] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:205 • [1.489 seconds] ------------------------------ Kueue Operator when installs verify webhook readiness [operator] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:239 • [0.663 seconds] ------------------------------ Kueue Operator when installs verify that deny-all network policy is present [operator] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:309 • [0.532 seconds] ------------------------------ Kueue Operator when installs verify kueue-config-map is updated with operators configuration [operator] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:361 STEP: check if JobSet is present @ 02/25/26 12:53:37.498 STEP: check if LeaderWorkerSet is present @ 02/25/26 12:53:37.672 • [0.334 seconds] ------------------------------ Kueue Operator when enable webhook via opt-in namespaces should suspend jobs only in labeled namespaces when labelPolicy=None [operator] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:486 STEP: Creating LocalQueue test-queue in namespace kueue-managed-test @ 02/25/26 12:53:38.345 STEP: Feching Kueue Instance @ 02/25/26 12:53:40.468 STEP: Updating Kueue config @ 02/25/26 12:53:40.826 STEP: Skipping deployment update wait - config unchanged @ 02/25/26 12:53:41.017 STEP: Checking if deployment replicas: 2 matches amount of ready replicas: 2 @ 02/25/26 12:53:41.196 STEP: Waiting for webhook configurations to exist @ 02/25/26 12:53:41.197 STEP: Waiting for webhook to handle requests successfully @ 02/25/26 12:53:41.557 I0225 12:53:41.940042 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-pwpmm I0225 12:53:42.186025 45539 e2e_operator_test.go:1494] Webhook success 1/3 I0225 12:53:58.219351 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-r5xsd I0225 12:53:58.393362 45539 e2e_operator_test.go:1494] Webhook success 1/3 I0225 12:54:01.739918 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-gmt69 I0225 12:54:01.910921 45539 e2e_operator_test.go:1494] Webhook success 2/3 I0225 12:54:05.280938 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-nvgvj I0225 12:54:05.451812 45539 e2e_operator_test.go:1491] Webhook stable after 3 consecutive successes STEP: creating job in labeled namespace @ 02/25/26 12:54:05.451 STEP: creating job in unlabeled namespace @ 02/25/26 12:54:26.462 STEP: Feching Kueue Instance @ 02/25/26 12:54:26.81 STEP: Updating Kueue config @ 02/25/26 12:54:27.352 STEP: Destroying job test-job-zwrrm @ 02/25/26 12:54:27.532 STEP: Destroying job test-job-mwprf @ 02/25/26 12:54:27.895 [FAILED] in [It] - /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:1372 @ 02/25/26 12:54:28.247 STEP: Test failed - dumping kueue-controller-manager logs @ 02/25/26 12:54:28.247 === Logs from pod kueue-controller-manager-76d4586765-gt7g5 === --- Container: manager --- {"level":"info","ts":"2026-02-25T15:53:44.340461016Z","logger":"setup","caller":"kueue/main.go:579","msg":"Successfully loaded configuration","config":"apiVersion: config.kueue.x-k8s.io/v1beta2\nclientConnection:\n burst: 100\n qps: 50\ncontroller:\n groupKindConcurrency:\n ClusterQueue.kueue.x-k8s.io: 1\n Job.batch: 5\n LocalQueue.kueue.x-k8s.io: 1\n Pod: 5\n ResourceFlavor.kueue.x-k8s.io: 1\n Workload.kueue.x-k8s.io: 5\nhealth:\n healthProbeBindAddress: :8081\nintegrations:\n frameworks:\n - batch/job\n - pod\n - deployment\n - statefulset\n - leaderworkerset.x-k8s.io/leaderworkerset\n - jobset.x-k8s.io/jobset\ninternalCertManagement:\n enable: false\nkind: Configuration\nleaderElection:\n leaderElect: true\n leaseDuration: 2m17s\n renewDeadline: 1m47s\n resourceLock: leases\n resourceName: c1f6bfd2.kueue.x-k8s.io\n resourceNamespace: \"\"\n retryPeriod: 26s\nmanageJobsWithoutQueueName: true\nmanagedJobsNamespaceSelector:\n matchLabels:\n kueue.openshift.io/managed: \"true\"\nmetrics:\n bindAddress: :8443\n enableClusterQueueResources: true\nmultiKueue:\n dispatcherName: kueue.x-k8s.io/multikueue-dispatcher-all-at-once\n gcInterval: 1m0s\n origin: multikueue\n workerLostTimeout: 15m0s\nnamespace: openshift-kueue-operator\nwebhook:\n certDir: /tmp/k8s-webhook-server/serving-certs\n port: 9443\n"} {"level":"info","ts":"2026-02-25T15:53:44.340957399Z","logger":"setup","caller":"kueue/main.go:160","msg":"Initializing","gitVersion":"50b346f-dirty","gitCommit":"50b346fb172b592ffe564959eb2cb59527765417","buildDate":"2026-02-22T04:31:17Z"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.341086985Z","logger":"setup","caller":"features/kube_features.go:406","msg":"Loaded feature gates","featureGates":{"AdmissionFairSharing":true,"DynamicResourceAllocation":false,"ElasticJobsViaWorkloadSlices":false,"FailureRecoveryPolicy":false,"FlavorFungibility":true,"HierarchicalCohorts":true,"LendingLimit":true,"LocalQueueDefaulting":true,"LocalQueueMetrics":false,"ManagedJobsNamespaceSelectorAlwaysRespected":true,"MultiKueue":true,"MultiKueueAdaptersForCustomJobs":true,"MultiKueueAllowInsecureKubeconfigs":false,"MultiKueueBatchJobWithManagedBy":true,"MultiKueueClusterProfile":false,"MultiKueueRedoAdmissionOnEvictionInWorker":true,"MultiKueueWaitForWorkloadAdmitted":true,"ObjectRetentionPolicies":true,"PartialAdmission":true,"PrioritySortingWithinCohort":true,"PropagateBatchJobLabelsToWorkload":true,"ReclaimablePods":true,"RemoveFinalizersWithStrictPatch":true,"SanitizePodSets":true,"SkipFinalizersForPodsSuspendedByParent":true,"TASBalancedPlacement":false,"TASFailedNodeReplacement":true,"TASFailedNodeReplacementFailFast":true,"TASProfileLeastFreeCapacity":false,"TASProfileMixed":true,"TASReplaceNodeOnPodTermination":true,"TLSOptions":true,"TopologyAwareScheduling":true,"VisibilityOnDemand":true,"WorkloadRequestUseMergePatch":false}} {"level":"info","ts":"2026-02-25T15:53:44.341168181Z","logger":"setup","caller":"kueue/main.go:195","msg":"Initializing metrics certificate watcher using provided certificates","metrics-cert-path":"/etc/kueue/metrics/certs"} {"level":"info","ts":"2026-02-25T15:53:44.341395682Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:214","msg":"Updated current TLS certificate","cert":"/etc/kueue/metrics/certs/tls.crt","key":"/etc/kueue/metrics/certs/tls.key"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.341595861Z","logger":"setup","caller":"kueue/main.go:228","msg":"K8S Client","qps":50,"burst":100} {"level":"info","ts":"2026-02-25T15:53:44.356912395Z","logger":"setup","caller":"kueue/main.go:549","msg":"RoleTracker: leader election enabled"} {"level":"error","ts":"2026-02-25T15:53:44.358927783Z","logger":"setup","caller":"kueue/main.go:364","msg":"Skipping admission check controller setup: Provisioning Requests not supported (Possible cause: missing or unsupported cluster-autoscaler)","error":"no matches for kind \"ProvisioningRequest\" in version \"autoscaling.x-k8s.io/v1\"","stacktrace":"main.setupIndexes\n\t/workspace/upstream/kueue/src/cmd/kueue/main.go:364\nmain.main\n\t/workspace/upstream/kueue/src/cmd/kueue/main.go:298\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:285"} {"level":"info","ts":"2026-02-25T15:53:44.36036001Z","logger":"setup","caller":"kueue/main.go:506","msg":"Probe endpoints are configured on healthz and readyz"} {"level":"info","ts":"2026-02-25T15:53:44.363733741Z","logger":"setup","caller":"kueue/main.go:399","msg":"Skipping provisioning controller setup: Provisioning Requests not supported (Possible cause: missing or unsupported cluster-autoscaler)"} {"level":"info","ts":"2026-02-25T15:53:44.363993494Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"workload.codeflare.dev/v1beta2, Kind=AppWrapper","path":"/mutate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.364200276Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.364298679Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"workload.codeflare.dev/v1beta2, Kind=AppWrapper","path":"/validate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.364489979Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.364596082Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"apps/v1, Kind=Deployment","path":"/mutate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.364648156Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.364674283Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"apps/v1, Kind=Deployment","path":"/validate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.364711319Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.364825818Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"batch/v1, Kind=Job","path":"/mutate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.364868354Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.364892466Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"batch/v1, Kind=Job","path":"/validate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.364927808Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.36505134Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"jobset.x-k8s.io/v1alpha2, Kind=JobSet","path":"/mutate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.365097013Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.36512396Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"jobset.x-k8s.io/v1alpha2, Kind=JobSet","path":"/validate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.365163907Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.365236227Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=JAXJob","path":"/mutate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.365279072Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.365304314Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=JAXJob","path":"/validate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.365342609Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.365384023Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=PaddleJob","path":"/mutate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.365420423Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.365443718Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=PaddleJob","path":"/validate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.365478474Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.365522616Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=PyTorchJob","path":"/mutate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.365560908Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.365586148Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=PyTorchJob","path":"/validate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.365621753Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.365667208Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=TFJob","path":"/mutate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.365709175Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.365732952Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=TFJob","path":"/validate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.365768288Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.365807752Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=XGBoostJob","path":"/mutate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.365843187Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.365867943Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=XGBoostJob","path":"/validate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.36590428Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.365944172Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v2beta1, Kind=MPIJob","path":"/mutate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.365981326Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.36600501Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v2beta1, Kind=MPIJob","path":"/validate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.366058861Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.366129671Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"/v1, Kind=Pod","path":"/mutate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.366168454Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.366192602Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"/v1, Kind=Pod","path":"/validate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.366227307Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.366335588Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"leaderworkerset.x-k8s.io/v1, Kind=LeaderWorkerSet","path":"/mutate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.366376357Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.366401096Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"leaderworkerset.x-k8s.io/v1, Kind=LeaderWorkerSet","path":"/validate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.36643766Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.366508072Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"ray.io/v1, Kind=RayCluster","path":"/mutate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.366544727Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.366569756Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"ray.io/v1, Kind=RayCluster","path":"/validate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.366605781Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.366650474Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"ray.io/v1, Kind=RayJob","path":"/mutate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.36668701Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.366708993Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"ray.io/v1, Kind=RayJob","path":"/validate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.366746876Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.366874134Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"apps/v1, Kind=StatefulSet","path":"/mutate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.366947915Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.366974333Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"apps/v1, Kind=StatefulSet","path":"/validate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.367012744Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.367111915Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"trainer.kubeflow.org/v1alpha1, Kind=TrainJob","path":"/mutate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.367151586Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.367175979Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"trainer.kubeflow.org/v1alpha1, Kind=TrainJob","path":"/validate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.367211817Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.367251848Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=Workload","path":"/mutate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.367288031Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.36731295Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=Workload","path":"/validate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.367349904Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.367412599Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/convert"} {"level":"info","ts":"2026-02-25T15:53:44.367420421Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=Workload"} {"level":"info","ts":"2026-02-25T15:53:44.367442335Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ResourceFlavor","path":"/mutate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.367478578Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.367502036Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ResourceFlavor","path":"/validate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.36753712Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.367585005Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ClusterQueue","path":"/mutate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.367621525Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.367645385Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ClusterQueue","path":"/validate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.367681207Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.367711727Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.367737372Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=Cohort","path":"/validate-kueue-x-k8s-io-v1beta2-cohort"} {"level":"info","ts":"2026-02-25T15:53:44.367772249Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-cohort"} {"level":"info","ts":"2026-02-25T15:53:44.367801074Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=Cohort"} {"level":"info","ts":"2026-02-25T15:53:44.367830265Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=LocalQueue"} {"level":"info","ts":"2026-02-25T15:53:44.36785223Z","logger":"setup","caller":"kueue/main.go:349","msg":"Starting manager"} {"level":"info","ts":"2026-02-25T15:53:44.367907557Z","logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} {"level":"info","ts":"2026-02-25T15:53:44.368055438Z","logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":8443","secure":true} {"level":"info","ts":"2026-02-25T15:53:44.368056519Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:191","msg":"Starting webhook server"} {"level":"info","ts":"2026-02-25T15:53:44.368433564Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:214","msg":"Updated current TLS certificate","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key"} {"level":"info","ts":"2026-02-25T15:53:44.368550355Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:242","msg":"Serving webhook server","host":"","port":9443} {"level":"info","ts":"2026-02-25T15:53:44.368039059Z","caller":"manager/server.go:83","msg":"starting server","name":"health probe","addr":"[::]:8081"} {"level":"info","ts":"2026-02-25T15:53:44.368687314Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:136","msg":"Starting certificate poll+watcher","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key","interval":10} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.372585545Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.MultiKueueConfig","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.373057275Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.AdmissionCheck","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.373358959Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Job","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.373366805Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.MultiKueueCluster","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.374965877Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.LimitRange","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.37519478Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.ResourceFlavor","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.375658616Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.LocalQueue","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.376417194Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Node","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.377615608Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.Workload","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.402616147Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"info","ts":"2026-02-25T15:53:44.469051647Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:136","msg":"Starting certificate poll+watcher","cert":"/etc/kueue/metrics/certs/tls.crt","key":"/etc/kueue/metrics/certs/tls.key","interval":10} {"level":"info","ts":"2026-02-25T15:53:44.469189233Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0005323f0"} {"level":"info","ts":"2026-02-25T15:53:44.469314418Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.AdmissionCheck"} {"level":"info","ts":"2026-02-25T15:53:44.469208079Z","logger":"localqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0005324d0"} {"level":"info","ts":"2026-02-25T15:53:44.469385041Z","logger":"localqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.469405585Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.469356113Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc000532700"} {"level":"info","ts":"2026-02-25T15:53:44.469359686Z","logger":"workloadpriorityclass-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.WorkloadPriorityClass"} {"level":"info","ts":"2026-02-25T15:53:44.469386427Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Namespace"} {"level":"info","ts":"2026-02-25T15:53:44.469343721Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ResourceFlavor"} {"level":"info","ts":"2026-02-25T15:53:44.469528143Z","logger":"localqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.LocalQueue"} {"level":"info","ts":"2026-02-25T15:53:44.469312616Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc000532310"} {"level":"info","ts":"2026-02-25T15:53:44.469614645Z","logger":"cohort-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0005325b0"} {"level":"info","ts":"2026-02-25T15:53:44.469668395Z","logger":"cohort-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Cohort"} {"level":"info","ts":"2026-02-25T15:53:44.469515708Z","logger":"tas-topology-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Topology"} {"level":"info","ts":"2026-02-25T15:53:44.469707353Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0005327e0"} {"level":"info","ts":"2026-02-25T15:53:44.469744939Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.LimitRange"} {"level":"info","ts":"2026-02-25T15:53:44.469791414Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.RuntimeClass"} {"level":"info","ts":"2026-02-25T15:53:44.469841502Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.469884951Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.LocalQueue"} {"level":"info","ts":"2026-02-25T15:53:44.469591047Z","logger":"tas-topology-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ResourceFlavor"} {"level":"info","ts":"2026-02-25T15:53:44.469945773Z","logger":"tas-topology-ungater","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-02-25T15:53:44.469919433Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:53:44.469997455Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Node"} {"level":"info","ts":"2026-02-25T15:53:44.469977487Z","logger":"tas-non-tas-usage-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"standalone","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-02-25T15:53:44.470091876Z","logger":"tas-node-failure-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Node"} {"level":"info","ts":"2026-02-25T15:53:44.469892497Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ResourceFlavor"} {"level":"info","ts":"2026-02-25T15:53:44.46980967Z","logger":"tas-topology-ungater","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} I0225 15:53:44.470329 1 leaderelection.go:258] "Attempting to acquire leader lease..." lock="openshift-kueue-operator/c1f6bfd2.kueue.x-k8s.io" {"level":"Level(-2)","ts":"2026-02-25T15:53:44.470399883Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:141","msg":"ResourceFlavor create event","replica-role":"follower","resourceFlavor":{"name":"default"}} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.470488034Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:215","msg":"LocalQueue create event","replica-role":"follower","localQueue":{"name":"test-queue","namespace":"kueue-managed-test"}} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.470636107Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:141","msg":"ResourceFlavor create event","replica-role":"follower","resourceFlavor":{"name":"default-flavor"}} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.470689046Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:141","msg":"ResourceFlavor create event","replica-role":"follower","resourceFlavor":{"name":"default-flavor2"}} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.475644953Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.ClusterQueue","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.475967039Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.RuntimeClass","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.476247521Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.WorkloadPriorityClass","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.4764467Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.Cohort","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.477298878Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.Topology","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.480087522Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Namespace","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} I0225 15:53:44.568784 1 plugins.go:157] Loaded 2 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,MutatingAdmissionPolicy. {"level":"info","ts":"2026-02-25T15:53:44.56993691Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.570001145Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.570206671Z","logger":"cohort-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.570244158Z","logger":"cohort-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.570328551Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:313","msg":"ClusterQueue create event","replica-role":"follower","clusterQueue":{"name":"cluster-queue"}} {"level":"info","ts":"2026-02-25T15:53:44.570435838Z","logger":"localqueue-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.570454802Z","logger":"localqueue-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.570494443Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:313","msg":"ClusterQueue create event","replica-role":"follower","clusterQueue":{"name":"cluster-queue2"}} {"level":"info","ts":"2026-02-25T15:53:44.570559442Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.570575526Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.57064351Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:313","msg":"ClusterQueue create event","replica-role":"follower","clusterQueue":{"name":"test-clusterqueue"}} {"level":"info","ts":"2026-02-25T15:53:44.570676238Z","logger":"tas-non-tas-usage-controller","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"standalone"} {"level":"info","ts":"2026-02-25T15:53:44.570690809Z","logger":"tas-non-tas-usage-controller","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"standalone","worker count":5} {"level":"info","ts":"2026-02-25T15:53:44.570821375Z","logger":"workloadpriorityclass-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.570851465Z","logger":"workloadpriorityclass-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.571494162Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.571551869Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.573127464Z","logger":"tas-node-failure-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.573192531Z","logger":"tas-node-failure-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.574116015Z","logger":"workload-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.574172258Z","logger":"workload-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":5} {"level":"info","ts":"2026-02-25T15:53:44.574296766Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.574326301Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} I0225 15:53:44.575096 1 handler.go:304] Adding GroupVersion visibility.kueue.x-k8s.io v1beta2 to ResourceManager I0225 15:53:44.575551 1 handler.go:304] Adding GroupVersion visibility.kueue.x-k8s.io v1beta1 to ResourceManager {"level":"info","ts":"2026-02-25T15:53:44.584440915Z","logger":"tas-topology-ungater","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.584552636Z","logger":"tas-topology-ungater","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":5} {"level":"info","ts":"2026-02-25T15:53:44.584778753Z","logger":"tas-topology-controller","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.58483679Z","logger":"tas-topology-controller","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} I0225 15:53:44.588925 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0225 15:53:44.588950 1 shared_informer.go:370] "Waiting for caches to sync" I0225 15:53:44.588968 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0225 15:53:44.588975 1 shared_informer.go:370] "Waiting for caches to sync" I0225 15:53:44.588986 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0225 15:53:44.588992 1 shared_informer.go:370] "Waiting for caches to sync" I0225 15:53:44.589364 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/visibility/tls.crt::/visibility/tls.key" I0225 15:53:44.589555 1 secure_serving.go:211] Serving securely on [::]:8082 I0225 15:53:44.589583 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" I0225 15:53:44.589632 1 apf_controller.go:377] Starting API Priority and Fairness config controller I0225 15:53:44.689891 1 apf_controller.go:382] Running API Priority and Fairness config worker I0225 15:53:44.689968 1 apf_controller.go:385] Running API Priority and Fairness periodic rebalancing process I0225 15:53:44.689922 1 shared_informer.go:377] "Caches are synced" I0225 15:53:44.689935 1 shared_informer.go:377] "Caches are synced" I0225 15:53:44.689938 1 shared_informer.go:377] "Caches are synced" {"level":"Level(-2)","ts":"2026-02-25T15:54:23.590111739Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"follower","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.590207371Z","logger":"workload-reconciler","caller":"core/workload_controller.go:831","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.608000117Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.608087502Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.19812975Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.198195966Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.212175285Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"follower","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} === Logs from pod kueue-controller-manager-76d4586765-w7lmz === --- Container: manager --- {"level":"info","ts":"2026-02-25T15:53:44.279621583Z","logger":"setup","caller":"kueue/main.go:579","msg":"Successfully loaded configuration","config":"apiVersion: config.kueue.x-k8s.io/v1beta2\nclientConnection:\n burst: 100\n qps: 50\ncontroller:\n groupKindConcurrency:\n ClusterQueue.kueue.x-k8s.io: 1\n Job.batch: 5\n LocalQueue.kueue.x-k8s.io: 1\n Pod: 5\n ResourceFlavor.kueue.x-k8s.io: 1\n Workload.kueue.x-k8s.io: 5\nhealth:\n healthProbeBindAddress: :8081\nintegrations:\n frameworks:\n - batch/job\n - pod\n - deployment\n - statefulset\n - leaderworkerset.x-k8s.io/leaderworkerset\n - jobset.x-k8s.io/jobset\ninternalCertManagement:\n enable: false\nkind: Configuration\nleaderElection:\n leaderElect: true\n leaseDuration: 2m17s\n renewDeadline: 1m47s\n resourceLock: leases\n resourceName: c1f6bfd2.kueue.x-k8s.io\n resourceNamespace: \"\"\n retryPeriod: 26s\nmanageJobsWithoutQueueName: true\nmanagedJobsNamespaceSelector:\n matchLabels:\n kueue.openshift.io/managed: \"true\"\nmetrics:\n bindAddress: :8443\n enableClusterQueueResources: true\nmultiKueue:\n dispatcherName: kueue.x-k8s.io/multikueue-dispatcher-all-at-once\n gcInterval: 1m0s\n origin: multikueue\n workerLostTimeout: 15m0s\nnamespace: openshift-kueue-operator\nwebhook:\n certDir: /tmp/k8s-webhook-server/serving-certs\n port: 9443\n"} {"level":"info","ts":"2026-02-25T15:53:44.279747887Z","logger":"setup","caller":"kueue/main.go:160","msg":"Initializing","gitVersion":"50b346f-dirty","gitCommit":"50b346fb172b592ffe564959eb2cb59527765417","buildDate":"2026-02-22T04:31:17Z"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.281674849Z","logger":"setup","caller":"features/kube_features.go:406","msg":"Loaded feature gates","featureGates":{"AdmissionFairSharing":true,"DynamicResourceAllocation":false,"ElasticJobsViaWorkloadSlices":false,"FailureRecoveryPolicy":false,"FlavorFungibility":true,"HierarchicalCohorts":true,"LendingLimit":true,"LocalQueueDefaulting":true,"LocalQueueMetrics":false,"ManagedJobsNamespaceSelectorAlwaysRespected":true,"MultiKueue":true,"MultiKueueAdaptersForCustomJobs":true,"MultiKueueAllowInsecureKubeconfigs":false,"MultiKueueBatchJobWithManagedBy":true,"MultiKueueClusterProfile":false,"MultiKueueRedoAdmissionOnEvictionInWorker":true,"MultiKueueWaitForWorkloadAdmitted":true,"ObjectRetentionPolicies":true,"PartialAdmission":true,"PrioritySortingWithinCohort":true,"PropagateBatchJobLabelsToWorkload":true,"ReclaimablePods":true,"RemoveFinalizersWithStrictPatch":true,"SanitizePodSets":true,"SkipFinalizersForPodsSuspendedByParent":true,"TASBalancedPlacement":false,"TASFailedNodeReplacement":true,"TASFailedNodeReplacementFailFast":true,"TASProfileLeastFreeCapacity":false,"TASProfileMixed":true,"TASReplaceNodeOnPodTermination":true,"TLSOptions":true,"TopologyAwareScheduling":true,"VisibilityOnDemand":true,"WorkloadRequestUseMergePatch":false}} {"level":"info","ts":"2026-02-25T15:53:44.281778129Z","logger":"setup","caller":"kueue/main.go:195","msg":"Initializing metrics certificate watcher using provided certificates","metrics-cert-path":"/etc/kueue/metrics/certs"} {"level":"info","ts":"2026-02-25T15:53:44.282148722Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:214","msg":"Updated current TLS certificate","cert":"/etc/kueue/metrics/certs/tls.crt","key":"/etc/kueue/metrics/certs/tls.key"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.282434386Z","logger":"setup","caller":"kueue/main.go:228","msg":"K8S Client","qps":50,"burst":100} {"level":"info","ts":"2026-02-25T15:53:44.30091856Z","logger":"setup","caller":"kueue/main.go:549","msg":"RoleTracker: leader election enabled"} {"level":"error","ts":"2026-02-25T15:53:44.303506867Z","logger":"setup","caller":"kueue/main.go:364","msg":"Skipping admission check controller setup: Provisioning Requests not supported (Possible cause: missing or unsupported cluster-autoscaler)","error":"no matches for kind \"ProvisioningRequest\" in version \"autoscaling.x-k8s.io/v1\"","stacktrace":"main.setupIndexes\n\t/workspace/upstream/kueue/src/cmd/kueue/main.go:364\nmain.main\n\t/workspace/upstream/kueue/src/cmd/kueue/main.go:298\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:285"} {"level":"info","ts":"2026-02-25T15:53:44.305857965Z","logger":"setup","caller":"kueue/main.go:506","msg":"Probe endpoints are configured on healthz and readyz"} {"level":"info","ts":"2026-02-25T15:53:44.307849898Z","logger":"setup","caller":"kueue/main.go:399","msg":"Skipping provisioning controller setup: Provisioning Requests not supported (Possible cause: missing or unsupported cluster-autoscaler)"} {"level":"info","ts":"2026-02-25T15:53:44.308011898Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"workload.codeflare.dev/v1beta2, Kind=AppWrapper","path":"/mutate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.308091229Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.308135679Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"workload.codeflare.dev/v1beta2, Kind=AppWrapper","path":"/validate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.308186638Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.308277059Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"apps/v1, Kind=Deployment","path":"/mutate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.308325654Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.308385596Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"apps/v1, Kind=Deployment","path":"/validate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.308431886Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.3085592Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"batch/v1, Kind=Job","path":"/mutate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.308602344Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.308621786Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"batch/v1, Kind=Job","path":"/validate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.308658489Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.308757891Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"jobset.x-k8s.io/v1alpha2, Kind=JobSet","path":"/mutate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.308797523Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.308815349Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"jobset.x-k8s.io/v1alpha2, Kind=JobSet","path":"/validate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.308863601Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.308934062Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=JAXJob","path":"/mutate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.308968472Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.308992373Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=JAXJob","path":"/validate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.309025871Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.309077677Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=PaddleJob","path":"/mutate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.309129509Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.309163402Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=PaddleJob","path":"/validate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.309200142Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.309264058Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=PyTorchJob","path":"/mutate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.309298958Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.309324593Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=PyTorchJob","path":"/validate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.30936498Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.309430139Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=TFJob","path":"/mutate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.309520135Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.309549091Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=TFJob","path":"/validate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.309603256Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.309682302Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=XGBoostJob","path":"/mutate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.309751505Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.309788546Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=XGBoostJob","path":"/validate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.309834512Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.30988365Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v2beta1, Kind=MPIJob","path":"/mutate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.309924823Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.309954839Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v2beta1, Kind=MPIJob","path":"/validate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.309995907Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.310079654Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"/v1, Kind=Pod","path":"/mutate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.310139549Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.310164284Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"/v1, Kind=Pod","path":"/validate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.310214291Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.310381361Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"leaderworkerset.x-k8s.io/v1, Kind=LeaderWorkerSet","path":"/mutate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.31051083Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.31056551Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"leaderworkerset.x-k8s.io/v1, Kind=LeaderWorkerSet","path":"/validate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.310626805Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.310751222Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"ray.io/v1, Kind=RayCluster","path":"/mutate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.310817583Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.310856509Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"ray.io/v1, Kind=RayCluster","path":"/validate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.310918893Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.310990165Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"ray.io/v1, Kind=RayJob","path":"/mutate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.311051318Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.311088179Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"ray.io/v1, Kind=RayJob","path":"/validate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.311499756Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.312242487Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"apps/v1, Kind=StatefulSet","path":"/mutate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.31249166Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.31253884Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"apps/v1, Kind=StatefulSet","path":"/validate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.312589856Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.312885251Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"trainer.kubeflow.org/v1alpha1, Kind=TrainJob","path":"/mutate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.312941616Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.312986347Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"trainer.kubeflow.org/v1alpha1, Kind=TrainJob","path":"/validate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.313049129Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.313214607Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=Workload","path":"/mutate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.313306305Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.313376373Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=Workload","path":"/validate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.313456132Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.313629481Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/convert"} {"level":"info","ts":"2026-02-25T15:53:44.313659096Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=Workload"} {"level":"info","ts":"2026-02-25T15:53:44.313710299Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ResourceFlavor","path":"/mutate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.313787755Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.313837229Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ResourceFlavor","path":"/validate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.313908388Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.314048405Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ClusterQueue","path":"/mutate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.314725043Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.314801746Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ClusterQueue","path":"/validate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.314860746Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.31494798Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.314990652Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=Cohort","path":"/validate-kueue-x-k8s-io-v1beta2-cohort"} {"level":"info","ts":"2026-02-25T15:53:44.315034476Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-cohort"} {"level":"info","ts":"2026-02-25T15:53:44.315147712Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=Cohort"} {"level":"info","ts":"2026-02-25T15:53:44.315236212Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=LocalQueue"} {"level":"info","ts":"2026-02-25T15:53:44.315258026Z","logger":"setup","caller":"kueue/main.go:349","msg":"Starting manager"} {"level":"info","ts":"2026-02-25T15:53:44.31533328Z","logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} {"level":"info","ts":"2026-02-25T15:53:44.315471423Z","logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":8443","secure":true} {"level":"info","ts":"2026-02-25T15:53:44.316260303Z","caller":"manager/server.go:83","msg":"starting server","name":"health probe","addr":"[::]:8081"} {"level":"info","ts":"2026-02-25T15:53:44.31633774Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:191","msg":"Starting webhook server"} {"level":"info","ts":"2026-02-25T15:53:44.31671217Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:214","msg":"Updated current TLS certificate","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key"} {"level":"info","ts":"2026-02-25T15:53:44.316798854Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:242","msg":"Serving webhook server","host":"","port":9443} {"level":"info","ts":"2026-02-25T15:53:44.317317806Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:136","msg":"Starting certificate poll+watcher","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key","interval":10} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.322565594Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.LimitRange","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.324181786Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Job","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.324418016Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.AdmissionCheck","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.334771556Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.LocalQueue","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.335081697Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.MultiKueueCluster","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.33950372Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.MultiKueueConfig","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.340560729Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Node","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.341150449Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.ResourceFlavor","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.354242079Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.Workload","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.392841688Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"info","ts":"2026-02-25T15:53:44.417945012Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:136","msg":"Starting certificate poll+watcher","cert":"/etc/kueue/metrics/certs/tls.crt","key":"/etc/kueue/metrics/certs/tls.key","interval":10} {"level":"info","ts":"2026-02-25T15:53:44.418063438Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002de850"} {"level":"info","ts":"2026-02-25T15:53:44.418190033Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002de700"} {"level":"info","ts":"2026-02-25T15:53:44.418248952Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ResourceFlavor"} {"level":"info","ts":"2026-02-25T15:53:44.418306847Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.418323341Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.418336783Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.AdmissionCheck"} {"level":"info","ts":"2026-02-25T15:53:44.418364917Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.418379128Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.418412681Z","logger":"localqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002de930"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.418457955Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:141","msg":"ResourceFlavor create event","replica-role":"follower","resourceFlavor":{"name":"default"}} {"level":"info","ts":"2026-02-25T15:53:44.418543799Z","logger":"workloadpriorityclass-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.WorkloadPriorityClass"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.418599616Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:141","msg":"ResourceFlavor create event","replica-role":"follower","resourceFlavor":{"name":"default-flavor"}} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.418627025Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:141","msg":"ResourceFlavor create event","replica-role":"follower","resourceFlavor":{"name":"default-flavor2"}} {"level":"info","ts":"2026-02-25T15:53:44.41866598Z","logger":"localqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.LocalQueue"} {"level":"info","ts":"2026-02-25T15:53:44.418793947Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002debd0"} {"level":"info","ts":"2026-02-25T15:53:44.418835076Z","logger":"localqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.418934927Z","logger":"cohort-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002dea10"} {"level":"info","ts":"2026-02-25T15:53:44.41902137Z","logger":"tas-topology-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Topology"} {"level":"info","ts":"2026-02-25T15:53:44.419069043Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Namespace"} {"level":"info","ts":"2026-02-25T15:53:44.419259296Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.419330123Z","logger":"cohort-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Cohort"} {"level":"info","ts":"2026-02-25T15:53:44.419517674Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002decb0"} {"level":"info","ts":"2026-02-25T15:53:44.419615479Z","logger":"tas-topology-ungater","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:53:44.419680116Z","logger":"tas-topology-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ResourceFlavor"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.419850793Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:215","msg":"LocalQueue create event","replica-role":"follower","localQueue":{"name":"test-queue","namespace":"kueue-managed-test"}} {"level":"info","ts":"2026-02-25T15:53:44.420079153Z","logger":"tas-topology-ungater","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-02-25T15:53:44.420087499Z","logger":"tas-non-tas-usage-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"standalone","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-02-25T15:53:44.420203472Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.LimitRange"} {"level":"info","ts":"2026-02-25T15:53:44.420264114Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.RuntimeClass"} {"level":"info","ts":"2026-02-25T15:53:44.420312116Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.42035359Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.LocalQueue"} {"level":"info","ts":"2026-02-25T15:53:44.420394273Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:53:44.420513225Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ResourceFlavor"} {"level":"info","ts":"2026-02-25T15:53:44.42061171Z","logger":"tas-node-failure-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Node"} {"level":"info","ts":"2026-02-25T15:53:44.420691072Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Node"} I0225 15:53:44.420838 1 leaderelection.go:258] "Attempting to acquire leader lease..." lock="openshift-kueue-operator/c1f6bfd2.kueue.x-k8s.io" {"level":"Level(-2)","ts":"2026-02-25T15:53:44.424441062Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.Topology","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.425208312Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.Cohort","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.425494622Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.WorkloadPriorityClass","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.425642886Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.RuntimeClass","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.425729818Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.ClusterQueue","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.429091863Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Namespace","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"info","ts":"2026-02-25T15:53:44.519303959Z","logger":"workloadpriorityclass-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.519420284Z","logger":"workloadpriorityclass-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.519413528Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:313","msg":"ClusterQueue create event","replica-role":"follower","clusterQueue":{"name":"cluster-queue"}} {"level":"info","ts":"2026-02-25T15:53:44.519598564Z","logger":"localqueue-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.519657428Z","logger":"localqueue-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.519611865Z","logger":"cohort-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.519742302Z","logger":"cohort-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.519623116Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:313","msg":"ClusterQueue create event","replica-role":"follower","clusterQueue":{"name":"cluster-queue2"}} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.519898159Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:313","msg":"ClusterQueue create event","replica-role":"follower","clusterQueue":{"name":"test-clusterqueue"}} {"level":"info","ts":"2026-02-25T15:53:44.520142016Z","logger":"tas-topology-controller","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.520162874Z","logger":"tas-topology-controller","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.520715763Z","logger":"tas-topology-ungater","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.520735396Z","logger":"tas-topology-ungater","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":5} {"level":"info","ts":"2026-02-25T15:53:44.52193053Z","logger":"workload-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.522003655Z","logger":"workload-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":5} {"level":"info","ts":"2026-02-25T15:53:44.522003161Z","logger":"tas-non-tas-usage-controller","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"standalone"} {"level":"info","ts":"2026-02-25T15:53:44.522044978Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.522092774Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.521979407Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.522012377Z","logger":"tas-node-failure-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.522128959Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.522134259Z","logger":"tas-node-failure-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.522052297Z","logger":"tas-non-tas-usage-controller","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"standalone","worker count":5} I0225 15:53:44.640704 1 plugins.go:157] Loaded 2 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,MutatingAdmissionPolicy. I0225 15:53:44.644255 1 handler.go:304] Adding GroupVersion visibility.kueue.x-k8s.io v1beta2 to ResourceManager I0225 15:53:44.644803 1 handler.go:304] Adding GroupVersion visibility.kueue.x-k8s.io v1beta1 to ResourceManager I0225 15:53:44.653678 1 secure_serving.go:211] Serving securely on [::]:8082 I0225 15:53:44.653813 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0225 15:53:44.653846 1 shared_informer.go:370] "Waiting for caches to sync" I0225 15:53:44.653881 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/visibility/tls.crt::/visibility/tls.key" I0225 15:53:44.653979 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" I0225 15:53:44.654449 1 apf_controller.go:377] Starting API Priority and Fairness config controller I0225 15:53:44.654499 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0225 15:53:44.654529 1 shared_informer.go:370] "Waiting for caches to sync" I0225 15:53:44.654578 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0225 15:53:44.654589 1 shared_informer.go:370] "Waiting for caches to sync" I0225 15:53:44.753987 1 shared_informer.go:377] "Caches are synced" I0225 15:53:44.755178 1 shared_informer.go:377] "Caches are synced" I0225 15:53:44.755183 1 shared_informer.go:377] "Caches are synced" I0225 15:53:44.755194 1 apf_controller.go:382] Running API Priority and Fairness config worker I0225 15:53:44.755303 1 apf_controller.go:385] Running API Priority and Fairness periodic rebalancing process I0225 15:54:23.478467 1 leaderelection.go:272] "Successfully acquired lease" lock="openshift-kueue-operator/c1f6bfd2.kueue.x-k8s.io" {"level":"debug","ts":"2026-02-25T15:54:23.47848655Z","logger":"events","caller":"recorder/recorder.go:104","msg":"kueue-controller-manager-76d4586765-w7lmz_e29d34fc-6f59-4925-af31-0047b63f4a24 became leader","type":"Normal","object":{"kind":"Lease","namespace":"openshift-kueue-operator","name":"c1f6bfd2.kueue.x-k8s.io","uid":"71c5dacc-f277-46a1-89d4-36b6039cf3f3","apiVersion":"coordination.k8s.io/v1","resourceVersion":"142117"},"reason":"LeaderElection"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.47863183Z","logger":"MultiKueueGC","caller":"multikueue/multikueuecluster.go:661","msg":"Starting Garbage Collector"} {"level":"info","ts":"2026-02-25T15:54:23.478798117Z","logger":"multikueue-admissioncheck","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.MultiKueueCluster"} {"level":"info","ts":"2026-02-25T15:54:23.47887902Z","logger":"multikueue-admissioncheck","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.AdmissionCheck"} {"level":"info","ts":"2026-02-25T15:54:23.478911067Z","logger":"multikueue-admissioncheck","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.MultiKueueConfig"} {"level":"info","ts":"2026-02-25T15:54:23.478876419Z","logger":"job","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:54:23.478937512Z","logger":"job","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Job"} {"level":"info","ts":"2026-02-25T15:54:23.47899078Z","logger":"job","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:54:23.479009867Z","logger":"leaderworkerset","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.LeaderWorkerSet"} {"level":"info","ts":"2026-02-25T15:54:23.479034679Z","logger":"job","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:54:23.479048968Z","logger":"job","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":5} {"level":"info","ts":"2026-02-25T15:54:23.479065076Z","logger":"multikueue-workload","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:54:23.479098312Z","logger":"multikueue-workload","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.MultiKueueConfig"} {"level":"info","ts":"2026-02-25T15:54:23.478935795Z","logger":"jobset","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:54:23.479202659Z","logger":"v1_pod","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:54:23.479218633Z","logger":"statefulset","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-02-25T15:54:23.479269984Z","logger":"multikueue-cluster","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002df0a0"} {"level":"info","ts":"2026-02-25T15:54:23.479312024Z","logger":"jobset","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1alpha2.JobSet"} {"level":"info","ts":"2026-02-25T15:54:23.478990311Z","logger":"multikueue-workload","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002df110"} {"level":"info","ts":"2026-02-25T15:54:23.479035148Z","logger":"leaderworkerset_pod","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-02-25T15:54:23.479407314Z","logger":"setup","caller":"roletracker/tracker.go:59","msg":"RoleTracker: transitioned to leader"} {"level":"info","ts":"2026-02-25T15:54:23.479279008Z","logger":"v1_pod","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-02-25T15:54:23.479430208Z","logger":"statefulset","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.StatefulSet"} {"level":"info","ts":"2026-02-25T15:54:23.479441315Z","logger":"multikueue-cluster","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.MultiKueueCluster"} {"level":"info","ts":"2026-02-25T15:54:23.479463892Z","logger":"multikueue-cluster","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Secret"} {"level":"info","ts":"2026-02-25T15:54:23.479584497Z","logger":"multikueue-cluster","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002df030"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.479790735Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"follower","namespace":"openshift-marketplace","name":"46657514f1e649ebaa51da741ac70392e61d69e281cd3964becce7ed5ab42c6","reconcileID":"4b63c326-7e10-4ff5-b4f6-8b586cbbe217","job":"openshift-marketplace/46657514f1e649ebaa51da741ac70392e61d69e281cd3964becce7ed5ab42c6","gvk":"batch/v1, Kind=Job","namespace":"openshift-marketplace"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.479805015Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-marketplace","name":"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76ac9","reconcileID":"a10ce01f-3c78-4c59-be98-b1c835b22d11","job":"openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76ac9","gvk":"batch/v1, Kind=Job","namespace":"openshift-marketplace"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.479848208Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-marketplace","name":"925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e56eb0c","reconcileID":"9263f448-2761-4f65-b891-1a83beb3b10f","job":"openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e56eb0c","gvk":"batch/v1, Kind=Job","namespace":"openshift-marketplace"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.480032346Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-marketplace","name":"c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78e8d070","reconcileID":"d2a96b26-2e64-441e-94de-692fef5cec4d","job":"openshift-marketplace/c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78e8d070","gvk":"batch/v1, Kind=Job","namespace":"openshift-marketplace"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.480050993Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-mwprf","reconcileID":"d4748d7e-84a5-4d4e-ab24-bf13177d201c","job":"kueue-managed-test/test-job-mwprf","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.492986141Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.JobSet","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.494641998Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.LeaderWorkerSet","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.495230317Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Secret","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.495339101Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.PriorityClass","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.495920814Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.StatefulSet","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"info","ts":"2026-02-25T15:54:23.580006576Z","logger":"jobset","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.58004821Z","logger":"jobset","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":1} {"level":"info","ts":"2026-02-25T15:54:23.580006578Z","logger":"multikueue-admissioncheck","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.580069816Z","logger":"multikueue-admissioncheck","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":1} {"level":"info","ts":"2026-02-25T15:54:23.580006225Z","logger":"leaderworkerset","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.580080288Z","logger":"leaderworkerset","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":1} {"level":"info","ts":"2026-02-25T15:54:23.580037828Z","logger":"statefulset","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.580091225Z","logger":"statefulset","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":1} {"level":"info","ts":"2026-02-25T15:54:23.580130816Z","logger":"multikueue-cluster","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.580138534Z","logger":"multikueue-cluster","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":1} {"level":"info","ts":"2026-02-25T15:54:23.580162296Z","logger":"leaderworkerset_pod","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.580169185Z","logger":"leaderworkerset_pod","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":5} {"level":"info","ts":"2026-02-25T15:54:23.580207148Z","logger":"multikueue-workload","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.580217794Z","logger":"multikueue-workload","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":5} {"level":"info","ts":"2026-02-25T15:54:23.581448472Z","logger":"v1_pod","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.581486301Z","logger":"v1_pod","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":5} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.595909321Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"14d0250c-88b6-4e65-8ab0-b09c6532a655"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.596063088Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"14d0250c-88b6-4e65-8ab0-b09c6532a655","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.59727002Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"leader","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.59739504Z","logger":"workload-reconciler","caller":"core/workload_controller.go:831","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.597499396Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"63e8c91b-221e-43b4-a3b7-c067d831381b"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.597516723Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-mwprf","reconcileID":"c2b98f22-2720-41d0-a6da-ab2b3398c53a","job":"kueue-managed-test/test-job-mwprf","gvk":"batch/v1, Kind=Job"} {"level":"debug","ts":"2026-02-25T15:54:23.59753245Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Created Workload: kueue-managed-test/job-test-job-mwprf-48c0e","type":"Normal","object":{"kind":"Job","namespace":"kueue-managed-test","name":"test-job-mwprf","uid":"eae93dbb-3e05-49df-b9c2-902ef552dd23","apiVersion":"batch/v1","resourceVersion":"142003"},"reason":"CreatedWorkload"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.608007747Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"fbdee5b1-6dd2-4c5f-8f86-b1fcb5c100eb"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.608167464Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"fbdee5b1-6dd2-4c5f-8f86-b1fcb5c100eb","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.6079943Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.608220862Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-mwprf","reconcileID":"531b9311-4487-473b-8594-708f2b9f6b60","job":"kueue-managed-test/test-job-mwprf","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.608250847Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.608316107Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"c8228417-97d6-4201-bee4-42f747ed20a0"} {"level":"Level(-2)","ts":"2026-02-25T15:54:26.584906078Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"kueue-unmanaged-test","name":"test-job-zwrrm","reconcileID":"abde3b87-0f40-4f82-84e9-bd573936408d","job":"kueue-unmanaged-test/test-job-zwrrm","gvk":"batch/v1, Kind=Job","namespace":"kueue-unmanaged-test"} {"level":"Level(-2)","ts":"2026-02-25T15:54:26.609606098Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"kueue-unmanaged-test","name":"test-job-zwrrm","reconcileID":"0b006413-f499-46bf-bf78-153b541a9638","job":"kueue-unmanaged-test/test-job-zwrrm","gvk":"batch/v1, Kind=Job","namespace":"kueue-unmanaged-test"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.198310364Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"551aeca3-1383-4804-9c23-1954904a284a"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.198408741Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.198536738Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.198653572Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"956a7f34-83e7-4968-8fcd-b4ef5b1e4b04"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.198443892Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"551aeca3-1383-4804-9c23-1954904a284a","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.212995169Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"leader","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.213453876Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"48722856-20b6-47fd-8095-8e8615d6c8eb"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.213543812Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"48722856-20b6-47fd-8095-8e8615d6c8eb","isDeleted":true} Attempt #1 Failed. Retrying ↺ @ 02/25/26 12:54:28.979 STEP: Feching Kueue Instance @ 02/25/26 12:54:29.501 STEP: Updating Kueue config @ 02/25/26 12:54:29.822 STEP: Skipping deployment update wait - config unchanged @ 02/25/26 12:54:30.008 STEP: Checking if deployment replicas: 2 matches amount of ready replicas: 2 @ 02/25/26 12:54:30.179 STEP: Waiting for webhook configurations to exist @ 02/25/26 12:54:30.179 STEP: Waiting for webhook to handle requests successfully @ 02/25/26 12:54:30.536 I0225 12:54:30.892800 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-dzl7v I0225 12:54:31.073915 45539 e2e_operator_test.go:1494] Webhook success 1/3 I0225 12:54:34.450707 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-dv8fw I0225 12:54:34.631247 45539 e2e_operator_test.go:1494] Webhook success 2/3 I0225 12:54:37.992378 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-lrf58 I0225 12:54:38.172823 45539 e2e_operator_test.go:1491] Webhook stable after 3 consecutive successes STEP: creating job in labeled namespace @ 02/25/26 12:54:38.172 STEP: creating job in unlabeled namespace @ 02/25/26 12:54:38.7 STEP: Feching Kueue Instance @ 02/25/26 12:54:39.053 STEP: Updating Kueue config @ 02/25/26 12:54:39.55 STEP: Skipping deployment update wait - config unchanged @ 02/25/26 12:54:39.736 STEP: Checking if deployment replicas: 2 matches amount of ready replicas: 2 @ 02/25/26 12:54:39.918 STEP: Waiting for webhook configurations to exist @ 02/25/26 12:54:39.919 STEP: Waiting for webhook to handle requests successfully @ 02/25/26 12:54:40.264 I0225 12:54:40.637335 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-g9lg6 I0225 12:54:40.818154 45539 e2e_operator_test.go:1494] Webhook success 1/3 I0225 12:54:44.179832 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-tw6tr I0225 12:54:44.337120 45539 e2e_operator_test.go:1494] Webhook success 2/3 I0225 12:54:47.697106 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-2h4xr I0225 12:54:47.876929 45539 e2e_operator_test.go:1491] Webhook stable after 3 consecutive successes STEP: Destroying job test-job-c7mms @ 02/25/26 12:54:47.876 STEP: Destroying job test-job-ptq5g @ 02/25/26 12:54:48.225 ↺ [FLAKEY TEST - TOOK 2 ATTEMPTS TO PASS] [70.749 seconds] ------------------------------ Kueue Operator when enable webhook via opt-in namespaces should manage jobs only in labeled namespaces [operator] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:533 STEP: creating job in labeled namespace @ 02/25/26 12:54:49.047 STEP: verifying workload created in namespace kueue-managed-test and uid 56c63b2f-0462-44a1-8757-92af7ef51831 @ 02/25/26 12:54:49.241 STEP: creating job in labeled namespace not managed by Kueue @ 02/25/26 12:54:49.605 STEP: Destroying job test-job-fw8gz @ 02/25/26 12:57:49.774 STEP: Destroying job test-job-b7dnm @ 02/25/26 12:57:50.125 [FAILED] in [It] - /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:559 @ 02/25/26 12:57:50.469 STEP: Test failed - dumping kueue-controller-manager logs @ 02/25/26 12:57:50.469 === Logs from pod kueue-controller-manager-76d4586765-gt7g5 === --- Container: manager --- {"level":"info","ts":"2026-02-25T15:53:44.340461016Z","logger":"setup","caller":"kueue/main.go:579","msg":"Successfully loaded configuration","config":"apiVersion: config.kueue.x-k8s.io/v1beta2\nclientConnection:\n burst: 100\n qps: 50\ncontroller:\n groupKindConcurrency:\n ClusterQueue.kueue.x-k8s.io: 1\n Job.batch: 5\n LocalQueue.kueue.x-k8s.io: 1\n Pod: 5\n ResourceFlavor.kueue.x-k8s.io: 1\n Workload.kueue.x-k8s.io: 5\nhealth:\n healthProbeBindAddress: :8081\nintegrations:\n frameworks:\n - batch/job\n - pod\n - deployment\n - statefulset\n - leaderworkerset.x-k8s.io/leaderworkerset\n - jobset.x-k8s.io/jobset\ninternalCertManagement:\n enable: false\nkind: Configuration\nleaderElection:\n leaderElect: true\n leaseDuration: 2m17s\n renewDeadline: 1m47s\n resourceLock: leases\n resourceName: c1f6bfd2.kueue.x-k8s.io\n resourceNamespace: \"\"\n retryPeriod: 26s\nmanageJobsWithoutQueueName: true\nmanagedJobsNamespaceSelector:\n matchLabels:\n kueue.openshift.io/managed: \"true\"\nmetrics:\n bindAddress: :8443\n enableClusterQueueResources: true\nmultiKueue:\n dispatcherName: kueue.x-k8s.io/multikueue-dispatcher-all-at-once\n gcInterval: 1m0s\n origin: multikueue\n workerLostTimeout: 15m0s\nnamespace: openshift-kueue-operator\nwebhook:\n certDir: /tmp/k8s-webhook-server/serving-certs\n port: 9443\n"} {"level":"info","ts":"2026-02-25T15:53:44.340957399Z","logger":"setup","caller":"kueue/main.go:160","msg":"Initializing","gitVersion":"50b346f-dirty","gitCommit":"50b346fb172b592ffe564959eb2cb59527765417","buildDate":"2026-02-22T04:31:17Z"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.341086985Z","logger":"setup","caller":"features/kube_features.go:406","msg":"Loaded feature gates","featureGates":{"AdmissionFairSharing":true,"DynamicResourceAllocation":false,"ElasticJobsViaWorkloadSlices":false,"FailureRecoveryPolicy":false,"FlavorFungibility":true,"HierarchicalCohorts":true,"LendingLimit":true,"LocalQueueDefaulting":true,"LocalQueueMetrics":false,"ManagedJobsNamespaceSelectorAlwaysRespected":true,"MultiKueue":true,"MultiKueueAdaptersForCustomJobs":true,"MultiKueueAllowInsecureKubeconfigs":false,"MultiKueueBatchJobWithManagedBy":true,"MultiKueueClusterProfile":false,"MultiKueueRedoAdmissionOnEvictionInWorker":true,"MultiKueueWaitForWorkloadAdmitted":true,"ObjectRetentionPolicies":true,"PartialAdmission":true,"PrioritySortingWithinCohort":true,"PropagateBatchJobLabelsToWorkload":true,"ReclaimablePods":true,"RemoveFinalizersWithStrictPatch":true,"SanitizePodSets":true,"SkipFinalizersForPodsSuspendedByParent":true,"TASBalancedPlacement":false,"TASFailedNodeReplacement":true,"TASFailedNodeReplacementFailFast":true,"TASProfileLeastFreeCapacity":false,"TASProfileMixed":true,"TASReplaceNodeOnPodTermination":true,"TLSOptions":true,"TopologyAwareScheduling":true,"VisibilityOnDemand":true,"WorkloadRequestUseMergePatch":false}} {"level":"info","ts":"2026-02-25T15:53:44.341168181Z","logger":"setup","caller":"kueue/main.go:195","msg":"Initializing metrics certificate watcher using provided certificates","metrics-cert-path":"/etc/kueue/metrics/certs"} {"level":"info","ts":"2026-02-25T15:53:44.341395682Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:214","msg":"Updated current TLS certificate","cert":"/etc/kueue/metrics/certs/tls.crt","key":"/etc/kueue/metrics/certs/tls.key"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.341595861Z","logger":"setup","caller":"kueue/main.go:228","msg":"K8S Client","qps":50,"burst":100} {"level":"info","ts":"2026-02-25T15:53:44.356912395Z","logger":"setup","caller":"kueue/main.go:549","msg":"RoleTracker: leader election enabled"} {"level":"error","ts":"2026-02-25T15:53:44.358927783Z","logger":"setup","caller":"kueue/main.go:364","msg":"Skipping admission check controller setup: Provisioning Requests not supported (Possible cause: missing or unsupported cluster-autoscaler)","error":"no matches for kind \"ProvisioningRequest\" in version \"autoscaling.x-k8s.io/v1\"","stacktrace":"main.setupIndexes\n\t/workspace/upstream/kueue/src/cmd/kueue/main.go:364\nmain.main\n\t/workspace/upstream/kueue/src/cmd/kueue/main.go:298\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:285"} {"level":"info","ts":"2026-02-25T15:53:44.36036001Z","logger":"setup","caller":"kueue/main.go:506","msg":"Probe endpoints are configured on healthz and readyz"} {"level":"info","ts":"2026-02-25T15:53:44.363733741Z","logger":"setup","caller":"kueue/main.go:399","msg":"Skipping provisioning controller setup: Provisioning Requests not supported (Possible cause: missing or unsupported cluster-autoscaler)"} {"level":"info","ts":"2026-02-25T15:53:44.363993494Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"workload.codeflare.dev/v1beta2, Kind=AppWrapper","path":"/mutate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.364200276Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.364298679Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"workload.codeflare.dev/v1beta2, Kind=AppWrapper","path":"/validate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.364489979Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.364596082Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"apps/v1, Kind=Deployment","path":"/mutate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.364648156Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.364674283Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"apps/v1, Kind=Deployment","path":"/validate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.364711319Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.364825818Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"batch/v1, Kind=Job","path":"/mutate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.364868354Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.364892466Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"batch/v1, Kind=Job","path":"/validate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.364927808Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.36505134Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"jobset.x-k8s.io/v1alpha2, Kind=JobSet","path":"/mutate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.365097013Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.36512396Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"jobset.x-k8s.io/v1alpha2, Kind=JobSet","path":"/validate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.365163907Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.365236227Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=JAXJob","path":"/mutate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.365279072Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.365304314Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=JAXJob","path":"/validate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.365342609Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.365384023Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=PaddleJob","path":"/mutate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.365420423Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.365443718Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=PaddleJob","path":"/validate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.365478474Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.365522616Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=PyTorchJob","path":"/mutate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.365560908Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.365586148Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=PyTorchJob","path":"/validate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.365621753Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.365667208Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=TFJob","path":"/mutate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.365709175Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.365732952Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=TFJob","path":"/validate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.365768288Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.365807752Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=XGBoostJob","path":"/mutate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.365843187Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.365867943Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=XGBoostJob","path":"/validate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.36590428Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.365944172Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v2beta1, Kind=MPIJob","path":"/mutate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.365981326Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.36600501Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v2beta1, Kind=MPIJob","path":"/validate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.366058861Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.366129671Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"/v1, Kind=Pod","path":"/mutate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.366168454Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.366192602Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"/v1, Kind=Pod","path":"/validate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.366227307Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.366335588Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"leaderworkerset.x-k8s.io/v1, Kind=LeaderWorkerSet","path":"/mutate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.366376357Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.366401096Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"leaderworkerset.x-k8s.io/v1, Kind=LeaderWorkerSet","path":"/validate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.36643766Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.366508072Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"ray.io/v1, Kind=RayCluster","path":"/mutate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.366544727Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.366569756Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"ray.io/v1, Kind=RayCluster","path":"/validate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.366605781Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.366650474Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"ray.io/v1, Kind=RayJob","path":"/mutate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.36668701Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.366708993Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"ray.io/v1, Kind=RayJob","path":"/validate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.366746876Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.366874134Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"apps/v1, Kind=StatefulSet","path":"/mutate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.366947915Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.366974333Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"apps/v1, Kind=StatefulSet","path":"/validate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.367012744Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.367111915Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"trainer.kubeflow.org/v1alpha1, Kind=TrainJob","path":"/mutate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.367151586Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.367175979Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"trainer.kubeflow.org/v1alpha1, Kind=TrainJob","path":"/validate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.367211817Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.367251848Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=Workload","path":"/mutate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.367288031Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.36731295Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=Workload","path":"/validate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.367349904Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.367412599Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/convert"} {"level":"info","ts":"2026-02-25T15:53:44.367420421Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=Workload"} {"level":"info","ts":"2026-02-25T15:53:44.367442335Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ResourceFlavor","path":"/mutate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.367478578Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.367502036Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ResourceFlavor","path":"/validate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.36753712Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.367585005Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ClusterQueue","path":"/mutate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.367621525Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.367645385Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ClusterQueue","path":"/validate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.367681207Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.367711727Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.367737372Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=Cohort","path":"/validate-kueue-x-k8s-io-v1beta2-cohort"} {"level":"info","ts":"2026-02-25T15:53:44.367772249Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-cohort"} {"level":"info","ts":"2026-02-25T15:53:44.367801074Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=Cohort"} {"level":"info","ts":"2026-02-25T15:53:44.367830265Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=LocalQueue"} {"level":"info","ts":"2026-02-25T15:53:44.36785223Z","logger":"setup","caller":"kueue/main.go:349","msg":"Starting manager"} {"level":"info","ts":"2026-02-25T15:53:44.367907557Z","logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} {"level":"info","ts":"2026-02-25T15:53:44.368055438Z","logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":8443","secure":true} {"level":"info","ts":"2026-02-25T15:53:44.368056519Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:191","msg":"Starting webhook server"} {"level":"info","ts":"2026-02-25T15:53:44.368433564Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:214","msg":"Updated current TLS certificate","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key"} {"level":"info","ts":"2026-02-25T15:53:44.368550355Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:242","msg":"Serving webhook server","host":"","port":9443} {"level":"info","ts":"2026-02-25T15:53:44.368039059Z","caller":"manager/server.go:83","msg":"starting server","name":"health probe","addr":"[::]:8081"} {"level":"info","ts":"2026-02-25T15:53:44.368687314Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:136","msg":"Starting certificate poll+watcher","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key","interval":10} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.372585545Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.MultiKueueConfig","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.373057275Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.AdmissionCheck","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.373358959Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Job","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.373366805Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.MultiKueueCluster","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.374965877Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.LimitRange","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.37519478Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.ResourceFlavor","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.375658616Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.LocalQueue","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.376417194Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Node","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.377615608Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.Workload","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.402616147Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"info","ts":"2026-02-25T15:53:44.469051647Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:136","msg":"Starting certificate poll+watcher","cert":"/etc/kueue/metrics/certs/tls.crt","key":"/etc/kueue/metrics/certs/tls.key","interval":10} {"level":"info","ts":"2026-02-25T15:53:44.469189233Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0005323f0"} {"level":"info","ts":"2026-02-25T15:53:44.469314418Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.AdmissionCheck"} {"level":"info","ts":"2026-02-25T15:53:44.469208079Z","logger":"localqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0005324d0"} {"level":"info","ts":"2026-02-25T15:53:44.469385041Z","logger":"localqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.469405585Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.469356113Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc000532700"} {"level":"info","ts":"2026-02-25T15:53:44.469359686Z","logger":"workloadpriorityclass-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.WorkloadPriorityClass"} {"level":"info","ts":"2026-02-25T15:53:44.469386427Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Namespace"} {"level":"info","ts":"2026-02-25T15:53:44.469343721Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ResourceFlavor"} {"level":"info","ts":"2026-02-25T15:53:44.469528143Z","logger":"localqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.LocalQueue"} {"level":"info","ts":"2026-02-25T15:53:44.469312616Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc000532310"} {"level":"info","ts":"2026-02-25T15:53:44.469614645Z","logger":"cohort-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0005325b0"} {"level":"info","ts":"2026-02-25T15:53:44.469668395Z","logger":"cohort-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Cohort"} {"level":"info","ts":"2026-02-25T15:53:44.469515708Z","logger":"tas-topology-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Topology"} {"level":"info","ts":"2026-02-25T15:53:44.469707353Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0005327e0"} {"level":"info","ts":"2026-02-25T15:53:44.469744939Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.LimitRange"} {"level":"info","ts":"2026-02-25T15:53:44.469791414Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.RuntimeClass"} {"level":"info","ts":"2026-02-25T15:53:44.469841502Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.469884951Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.LocalQueue"} {"level":"info","ts":"2026-02-25T15:53:44.469591047Z","logger":"tas-topology-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ResourceFlavor"} {"level":"info","ts":"2026-02-25T15:53:44.469945773Z","logger":"tas-topology-ungater","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-02-25T15:53:44.469919433Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:53:44.469997455Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Node"} {"level":"info","ts":"2026-02-25T15:53:44.469977487Z","logger":"tas-non-tas-usage-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"standalone","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-02-25T15:53:44.470091876Z","logger":"tas-node-failure-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Node"} {"level":"info","ts":"2026-02-25T15:53:44.469892497Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ResourceFlavor"} {"level":"info","ts":"2026-02-25T15:53:44.46980967Z","logger":"tas-topology-ungater","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} I0225 15:53:44.470329 1 leaderelection.go:258] "Attempting to acquire leader lease..." lock="openshift-kueue-operator/c1f6bfd2.kueue.x-k8s.io" {"level":"Level(-2)","ts":"2026-02-25T15:53:44.470399883Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:141","msg":"ResourceFlavor create event","replica-role":"follower","resourceFlavor":{"name":"default"}} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.470488034Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:215","msg":"LocalQueue create event","replica-role":"follower","localQueue":{"name":"test-queue","namespace":"kueue-managed-test"}} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.470636107Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:141","msg":"ResourceFlavor create event","replica-role":"follower","resourceFlavor":{"name":"default-flavor"}} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.470689046Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:141","msg":"ResourceFlavor create event","replica-role":"follower","resourceFlavor":{"name":"default-flavor2"}} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.475644953Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.ClusterQueue","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.475967039Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.RuntimeClass","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.476247521Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.WorkloadPriorityClass","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.4764467Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.Cohort","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.477298878Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.Topology","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.480087522Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Namespace","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} I0225 15:53:44.568784 1 plugins.go:157] Loaded 2 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,MutatingAdmissionPolicy. {"level":"info","ts":"2026-02-25T15:53:44.56993691Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.570001145Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.570206671Z","logger":"cohort-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.570244158Z","logger":"cohort-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.570328551Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:313","msg":"ClusterQueue create event","replica-role":"follower","clusterQueue":{"name":"cluster-queue"}} {"level":"info","ts":"2026-02-25T15:53:44.570435838Z","logger":"localqueue-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.570454802Z","logger":"localqueue-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.570494443Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:313","msg":"ClusterQueue create event","replica-role":"follower","clusterQueue":{"name":"cluster-queue2"}} {"level":"info","ts":"2026-02-25T15:53:44.570559442Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.570575526Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.57064351Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:313","msg":"ClusterQueue create event","replica-role":"follower","clusterQueue":{"name":"test-clusterqueue"}} {"level":"info","ts":"2026-02-25T15:53:44.570676238Z","logger":"tas-non-tas-usage-controller","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"standalone"} {"level":"info","ts":"2026-02-25T15:53:44.570690809Z","logger":"tas-non-tas-usage-controller","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"standalone","worker count":5} {"level":"info","ts":"2026-02-25T15:53:44.570821375Z","logger":"workloadpriorityclass-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.570851465Z","logger":"workloadpriorityclass-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.571494162Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.571551869Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.573127464Z","logger":"tas-node-failure-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.573192531Z","logger":"tas-node-failure-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.574116015Z","logger":"workload-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.574172258Z","logger":"workload-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":5} {"level":"info","ts":"2026-02-25T15:53:44.574296766Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.574326301Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} I0225 15:53:44.575096 1 handler.go:304] Adding GroupVersion visibility.kueue.x-k8s.io v1beta2 to ResourceManager I0225 15:53:44.575551 1 handler.go:304] Adding GroupVersion visibility.kueue.x-k8s.io v1beta1 to ResourceManager {"level":"info","ts":"2026-02-25T15:53:44.584440915Z","logger":"tas-topology-ungater","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.584552636Z","logger":"tas-topology-ungater","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":5} {"level":"info","ts":"2026-02-25T15:53:44.584778753Z","logger":"tas-topology-controller","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.58483679Z","logger":"tas-topology-controller","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} I0225 15:53:44.588925 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0225 15:53:44.588950 1 shared_informer.go:370] "Waiting for caches to sync" I0225 15:53:44.588968 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0225 15:53:44.588975 1 shared_informer.go:370] "Waiting for caches to sync" I0225 15:53:44.588986 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0225 15:53:44.588992 1 shared_informer.go:370] "Waiting for caches to sync" I0225 15:53:44.589364 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/visibility/tls.crt::/visibility/tls.key" I0225 15:53:44.589555 1 secure_serving.go:211] Serving securely on [::]:8082 I0225 15:53:44.589583 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" I0225 15:53:44.589632 1 apf_controller.go:377] Starting API Priority and Fairness config controller I0225 15:53:44.689891 1 apf_controller.go:382] Running API Priority and Fairness config worker I0225 15:53:44.689968 1 apf_controller.go:385] Running API Priority and Fairness periodic rebalancing process I0225 15:53:44.689922 1 shared_informer.go:377] "Caches are synced" I0225 15:53:44.689935 1 shared_informer.go:377] "Caches are synced" I0225 15:53:44.689938 1 shared_informer.go:377] "Caches are synced" {"level":"Level(-2)","ts":"2026-02-25T15:54:23.590111739Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"follower","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.590207371Z","logger":"workload-reconciler","caller":"core/workload_controller.go:831","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.608000117Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.608087502Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.19812975Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.198195966Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.212175285Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"follower","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.30798512Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"follower","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.308068183Z","logger":"workload-reconciler","caller":"core/workload_controller.go:831","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.317818492Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.317858668Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.534874458Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.534950465Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.548692336Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"follower","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.188895592Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"follower","workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"queue":"test-queue","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.197532922Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted","prevStatus":"pending","clusterQueue":"test-clusterqueue"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.733111904Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"follower","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.733173254Z","logger":"workload-reconciler","caller":"core/workload_controller.go:831","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.745537428Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.745598368Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:50.198748709Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:254","msg":"Queue update event","replica-role":"follower","localQueue":{"name":"test-queue","namespace":"kueue-managed-test"}} {"level":"Level(-2)","ts":"2026-02-25T15:54:50.199456463Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:346","msg":"ClusterQueue update event","replica-role":"follower","clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.084404451Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.08447968Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.094073562Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"follower","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.42723648Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted","clusterQueue":"test-clusterqueue"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.455541678Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"follower","workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted"} === Logs from pod kueue-controller-manager-76d4586765-w7lmz === --- Container: manager --- {"level":"info","ts":"2026-02-25T15:53:44.279621583Z","logger":"setup","caller":"kueue/main.go:579","msg":"Successfully loaded configuration","config":"apiVersion: config.kueue.x-k8s.io/v1beta2\nclientConnection:\n burst: 100\n qps: 50\ncontroller:\n groupKindConcurrency:\n ClusterQueue.kueue.x-k8s.io: 1\n Job.batch: 5\n LocalQueue.kueue.x-k8s.io: 1\n Pod: 5\n ResourceFlavor.kueue.x-k8s.io: 1\n Workload.kueue.x-k8s.io: 5\nhealth:\n healthProbeBindAddress: :8081\nintegrations:\n frameworks:\n - batch/job\n - pod\n - deployment\n - statefulset\n - leaderworkerset.x-k8s.io/leaderworkerset\n - jobset.x-k8s.io/jobset\ninternalCertManagement:\n enable: false\nkind: Configuration\nleaderElection:\n leaderElect: true\n leaseDuration: 2m17s\n renewDeadline: 1m47s\n resourceLock: leases\n resourceName: c1f6bfd2.kueue.x-k8s.io\n resourceNamespace: \"\"\n retryPeriod: 26s\nmanageJobsWithoutQueueName: true\nmanagedJobsNamespaceSelector:\n matchLabels:\n kueue.openshift.io/managed: \"true\"\nmetrics:\n bindAddress: :8443\n enableClusterQueueResources: true\nmultiKueue:\n dispatcherName: kueue.x-k8s.io/multikueue-dispatcher-all-at-once\n gcInterval: 1m0s\n origin: multikueue\n workerLostTimeout: 15m0s\nnamespace: openshift-kueue-operator\nwebhook:\n certDir: /tmp/k8s-webhook-server/serving-certs\n port: 9443\n"} {"level":"info","ts":"2026-02-25T15:53:44.279747887Z","logger":"setup","caller":"kueue/main.go:160","msg":"Initializing","gitVersion":"50b346f-dirty","gitCommit":"50b346fb172b592ffe564959eb2cb59527765417","buildDate":"2026-02-22T04:31:17Z"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.281674849Z","logger":"setup","caller":"features/kube_features.go:406","msg":"Loaded feature gates","featureGates":{"AdmissionFairSharing":true,"DynamicResourceAllocation":false,"ElasticJobsViaWorkloadSlices":false,"FailureRecoveryPolicy":false,"FlavorFungibility":true,"HierarchicalCohorts":true,"LendingLimit":true,"LocalQueueDefaulting":true,"LocalQueueMetrics":false,"ManagedJobsNamespaceSelectorAlwaysRespected":true,"MultiKueue":true,"MultiKueueAdaptersForCustomJobs":true,"MultiKueueAllowInsecureKubeconfigs":false,"MultiKueueBatchJobWithManagedBy":true,"MultiKueueClusterProfile":false,"MultiKueueRedoAdmissionOnEvictionInWorker":true,"MultiKueueWaitForWorkloadAdmitted":true,"ObjectRetentionPolicies":true,"PartialAdmission":true,"PrioritySortingWithinCohort":true,"PropagateBatchJobLabelsToWorkload":true,"ReclaimablePods":true,"RemoveFinalizersWithStrictPatch":true,"SanitizePodSets":true,"SkipFinalizersForPodsSuspendedByParent":true,"TASBalancedPlacement":false,"TASFailedNodeReplacement":true,"TASFailedNodeReplacementFailFast":true,"TASProfileLeastFreeCapacity":false,"TASProfileMixed":true,"TASReplaceNodeOnPodTermination":true,"TLSOptions":true,"TopologyAwareScheduling":true,"VisibilityOnDemand":true,"WorkloadRequestUseMergePatch":false}} {"level":"info","ts":"2026-02-25T15:53:44.281778129Z","logger":"setup","caller":"kueue/main.go:195","msg":"Initializing metrics certificate watcher using provided certificates","metrics-cert-path":"/etc/kueue/metrics/certs"} {"level":"info","ts":"2026-02-25T15:53:44.282148722Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:214","msg":"Updated current TLS certificate","cert":"/etc/kueue/metrics/certs/tls.crt","key":"/etc/kueue/metrics/certs/tls.key"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.282434386Z","logger":"setup","caller":"kueue/main.go:228","msg":"K8S Client","qps":50,"burst":100} {"level":"info","ts":"2026-02-25T15:53:44.30091856Z","logger":"setup","caller":"kueue/main.go:549","msg":"RoleTracker: leader election enabled"} {"level":"error","ts":"2026-02-25T15:53:44.303506867Z","logger":"setup","caller":"kueue/main.go:364","msg":"Skipping admission check controller setup: Provisioning Requests not supported (Possible cause: missing or unsupported cluster-autoscaler)","error":"no matches for kind \"ProvisioningRequest\" in version \"autoscaling.x-k8s.io/v1\"","stacktrace":"main.setupIndexes\n\t/workspace/upstream/kueue/src/cmd/kueue/main.go:364\nmain.main\n\t/workspace/upstream/kueue/src/cmd/kueue/main.go:298\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:285"} {"level":"info","ts":"2026-02-25T15:53:44.305857965Z","logger":"setup","caller":"kueue/main.go:506","msg":"Probe endpoints are configured on healthz and readyz"} {"level":"info","ts":"2026-02-25T15:53:44.307849898Z","logger":"setup","caller":"kueue/main.go:399","msg":"Skipping provisioning controller setup: Provisioning Requests not supported (Possible cause: missing or unsupported cluster-autoscaler)"} {"level":"info","ts":"2026-02-25T15:53:44.308011898Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"workload.codeflare.dev/v1beta2, Kind=AppWrapper","path":"/mutate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.308091229Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.308135679Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"workload.codeflare.dev/v1beta2, Kind=AppWrapper","path":"/validate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.308186638Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.308277059Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"apps/v1, Kind=Deployment","path":"/mutate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.308325654Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.308385596Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"apps/v1, Kind=Deployment","path":"/validate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.308431886Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.3085592Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"batch/v1, Kind=Job","path":"/mutate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.308602344Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.308621786Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"batch/v1, Kind=Job","path":"/validate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.308658489Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.308757891Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"jobset.x-k8s.io/v1alpha2, Kind=JobSet","path":"/mutate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.308797523Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.308815349Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"jobset.x-k8s.io/v1alpha2, Kind=JobSet","path":"/validate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.308863601Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.308934062Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=JAXJob","path":"/mutate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.308968472Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.308992373Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=JAXJob","path":"/validate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.309025871Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.309077677Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=PaddleJob","path":"/mutate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.309129509Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.309163402Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=PaddleJob","path":"/validate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.309200142Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.309264058Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=PyTorchJob","path":"/mutate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.309298958Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.309324593Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=PyTorchJob","path":"/validate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.30936498Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.309430139Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=TFJob","path":"/mutate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.309520135Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.309549091Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=TFJob","path":"/validate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.309603256Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.309682302Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=XGBoostJob","path":"/mutate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.309751505Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.309788546Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=XGBoostJob","path":"/validate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.309834512Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.30988365Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v2beta1, Kind=MPIJob","path":"/mutate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.309924823Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.309954839Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v2beta1, Kind=MPIJob","path":"/validate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.309995907Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.310079654Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"/v1, Kind=Pod","path":"/mutate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.310139549Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.310164284Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"/v1, Kind=Pod","path":"/validate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.310214291Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.310381361Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"leaderworkerset.x-k8s.io/v1, Kind=LeaderWorkerSet","path":"/mutate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.31051083Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.31056551Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"leaderworkerset.x-k8s.io/v1, Kind=LeaderWorkerSet","path":"/validate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.310626805Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.310751222Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"ray.io/v1, Kind=RayCluster","path":"/mutate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.310817583Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.310856509Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"ray.io/v1, Kind=RayCluster","path":"/validate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.310918893Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.310990165Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"ray.io/v1, Kind=RayJob","path":"/mutate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.311051318Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.311088179Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"ray.io/v1, Kind=RayJob","path":"/validate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.311499756Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.312242487Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"apps/v1, Kind=StatefulSet","path":"/mutate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.31249166Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.31253884Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"apps/v1, Kind=StatefulSet","path":"/validate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.312589856Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.312885251Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"trainer.kubeflow.org/v1alpha1, Kind=TrainJob","path":"/mutate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.312941616Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.312986347Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"trainer.kubeflow.org/v1alpha1, Kind=TrainJob","path":"/validate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.313049129Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.313214607Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=Workload","path":"/mutate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.313306305Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.313376373Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=Workload","path":"/validate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.313456132Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.313629481Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/convert"} {"level":"info","ts":"2026-02-25T15:53:44.313659096Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=Workload"} {"level":"info","ts":"2026-02-25T15:53:44.313710299Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ResourceFlavor","path":"/mutate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.313787755Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.313837229Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ResourceFlavor","path":"/validate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.313908388Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.314048405Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ClusterQueue","path":"/mutate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.314725043Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.314801746Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ClusterQueue","path":"/validate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.314860746Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.31494798Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.314990652Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=Cohort","path":"/validate-kueue-x-k8s-io-v1beta2-cohort"} {"level":"info","ts":"2026-02-25T15:53:44.315034476Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-cohort"} {"level":"info","ts":"2026-02-25T15:53:44.315147712Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=Cohort"} {"level":"info","ts":"2026-02-25T15:53:44.315236212Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=LocalQueue"} {"level":"info","ts":"2026-02-25T15:53:44.315258026Z","logger":"setup","caller":"kueue/main.go:349","msg":"Starting manager"} {"level":"info","ts":"2026-02-25T15:53:44.31533328Z","logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} {"level":"info","ts":"2026-02-25T15:53:44.315471423Z","logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":8443","secure":true} {"level":"info","ts":"2026-02-25T15:53:44.316260303Z","caller":"manager/server.go:83","msg":"starting server","name":"health probe","addr":"[::]:8081"} {"level":"info","ts":"2026-02-25T15:53:44.31633774Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:191","msg":"Starting webhook server"} {"level":"info","ts":"2026-02-25T15:53:44.31671217Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:214","msg":"Updated current TLS certificate","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key"} {"level":"info","ts":"2026-02-25T15:53:44.316798854Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:242","msg":"Serving webhook server","host":"","port":9443} {"level":"info","ts":"2026-02-25T15:53:44.317317806Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:136","msg":"Starting certificate poll+watcher","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key","interval":10} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.322565594Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.LimitRange","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.324181786Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Job","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.324418016Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.AdmissionCheck","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.334771556Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.LocalQueue","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.335081697Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.MultiKueueCluster","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.33950372Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.MultiKueueConfig","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.340560729Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Node","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.341150449Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.ResourceFlavor","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.354242079Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.Workload","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.392841688Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"info","ts":"2026-02-25T15:53:44.417945012Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:136","msg":"Starting certificate poll+watcher","cert":"/etc/kueue/metrics/certs/tls.crt","key":"/etc/kueue/metrics/certs/tls.key","interval":10} {"level":"info","ts":"2026-02-25T15:53:44.418063438Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002de850"} {"level":"info","ts":"2026-02-25T15:53:44.418190033Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002de700"} {"level":"info","ts":"2026-02-25T15:53:44.418248952Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ResourceFlavor"} {"level":"info","ts":"2026-02-25T15:53:44.418306847Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.418323341Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.418336783Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.AdmissionCheck"} {"level":"info","ts":"2026-02-25T15:53:44.418364917Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.418379128Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.418412681Z","logger":"localqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002de930"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.418457955Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:141","msg":"ResourceFlavor create event","replica-role":"follower","resourceFlavor":{"name":"default"}} {"level":"info","ts":"2026-02-25T15:53:44.418543799Z","logger":"workloadpriorityclass-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.WorkloadPriorityClass"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.418599616Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:141","msg":"ResourceFlavor create event","replica-role":"follower","resourceFlavor":{"name":"default-flavor"}} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.418627025Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:141","msg":"ResourceFlavor create event","replica-role":"follower","resourceFlavor":{"name":"default-flavor2"}} {"level":"info","ts":"2026-02-25T15:53:44.41866598Z","logger":"localqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.LocalQueue"} {"level":"info","ts":"2026-02-25T15:53:44.418793947Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002debd0"} {"level":"info","ts":"2026-02-25T15:53:44.418835076Z","logger":"localqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.418934927Z","logger":"cohort-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002dea10"} {"level":"info","ts":"2026-02-25T15:53:44.41902137Z","logger":"tas-topology-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Topology"} {"level":"info","ts":"2026-02-25T15:53:44.419069043Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Namespace"} {"level":"info","ts":"2026-02-25T15:53:44.419259296Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.419330123Z","logger":"cohort-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Cohort"} {"level":"info","ts":"2026-02-25T15:53:44.419517674Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002decb0"} {"level":"info","ts":"2026-02-25T15:53:44.419615479Z","logger":"tas-topology-ungater","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:53:44.419680116Z","logger":"tas-topology-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ResourceFlavor"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.419850793Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:215","msg":"LocalQueue create event","replica-role":"follower","localQueue":{"name":"test-queue","namespace":"kueue-managed-test"}} {"level":"info","ts":"2026-02-25T15:53:44.420079153Z","logger":"tas-topology-ungater","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-02-25T15:53:44.420087499Z","logger":"tas-non-tas-usage-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"standalone","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-02-25T15:53:44.420203472Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.LimitRange"} {"level":"info","ts":"2026-02-25T15:53:44.420264114Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.RuntimeClass"} {"level":"info","ts":"2026-02-25T15:53:44.420312116Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.42035359Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.LocalQueue"} {"level":"info","ts":"2026-02-25T15:53:44.420394273Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:53:44.420513225Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ResourceFlavor"} {"level":"info","ts":"2026-02-25T15:53:44.42061171Z","logger":"tas-node-failure-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Node"} {"level":"info","ts":"2026-02-25T15:53:44.420691072Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Node"} I0225 15:53:44.420838 1 leaderelection.go:258] "Attempting to acquire leader lease..." lock="openshift-kueue-operator/c1f6bfd2.kueue.x-k8s.io" {"level":"Level(-2)","ts":"2026-02-25T15:53:44.424441062Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.Topology","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.425208312Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.Cohort","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.425494622Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.WorkloadPriorityClass","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.425642886Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.RuntimeClass","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.425729818Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.ClusterQueue","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.429091863Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Namespace","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"info","ts":"2026-02-25T15:53:44.519303959Z","logger":"workloadpriorityclass-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.519420284Z","logger":"workloadpriorityclass-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.519413528Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:313","msg":"ClusterQueue create event","replica-role":"follower","clusterQueue":{"name":"cluster-queue"}} {"level":"info","ts":"2026-02-25T15:53:44.519598564Z","logger":"localqueue-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.519657428Z","logger":"localqueue-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.519611865Z","logger":"cohort-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.519742302Z","logger":"cohort-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.519623116Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:313","msg":"ClusterQueue create event","replica-role":"follower","clusterQueue":{"name":"cluster-queue2"}} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.519898159Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:313","msg":"ClusterQueue create event","replica-role":"follower","clusterQueue":{"name":"test-clusterqueue"}} {"level":"info","ts":"2026-02-25T15:53:44.520142016Z","logger":"tas-topology-controller","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.520162874Z","logger":"tas-topology-controller","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.520715763Z","logger":"tas-topology-ungater","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.520735396Z","logger":"tas-topology-ungater","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":5} {"level":"info","ts":"2026-02-25T15:53:44.52193053Z","logger":"workload-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.522003655Z","logger":"workload-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":5} {"level":"info","ts":"2026-02-25T15:53:44.522003161Z","logger":"tas-non-tas-usage-controller","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"standalone"} {"level":"info","ts":"2026-02-25T15:53:44.522044978Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.522092774Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.521979407Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.522012377Z","logger":"tas-node-failure-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.522128959Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.522134259Z","logger":"tas-node-failure-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.522052297Z","logger":"tas-non-tas-usage-controller","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"standalone","worker count":5} I0225 15:53:44.640704 1 plugins.go:157] Loaded 2 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,MutatingAdmissionPolicy. I0225 15:53:44.644255 1 handler.go:304] Adding GroupVersion visibility.kueue.x-k8s.io v1beta2 to ResourceManager I0225 15:53:44.644803 1 handler.go:304] Adding GroupVersion visibility.kueue.x-k8s.io v1beta1 to ResourceManager I0225 15:53:44.653678 1 secure_serving.go:211] Serving securely on [::]:8082 I0225 15:53:44.653813 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0225 15:53:44.653846 1 shared_informer.go:370] "Waiting for caches to sync" I0225 15:53:44.653881 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/visibility/tls.crt::/visibility/tls.key" I0225 15:53:44.653979 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" I0225 15:53:44.654449 1 apf_controller.go:377] Starting API Priority and Fairness config controller I0225 15:53:44.654499 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0225 15:53:44.654529 1 shared_informer.go:370] "Waiting for caches to sync" I0225 15:53:44.654578 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0225 15:53:44.654589 1 shared_informer.go:370] "Waiting for caches to sync" I0225 15:53:44.753987 1 shared_informer.go:377] "Caches are synced" I0225 15:53:44.755178 1 shared_informer.go:377] "Caches are synced" I0225 15:53:44.755183 1 shared_informer.go:377] "Caches are synced" I0225 15:53:44.755194 1 apf_controller.go:382] Running API Priority and Fairness config worker I0225 15:53:44.755303 1 apf_controller.go:385] Running API Priority and Fairness periodic rebalancing process I0225 15:54:23.478467 1 leaderelection.go:272] "Successfully acquired lease" lock="openshift-kueue-operator/c1f6bfd2.kueue.x-k8s.io" {"level":"debug","ts":"2026-02-25T15:54:23.47848655Z","logger":"events","caller":"recorder/recorder.go:104","msg":"kueue-controller-manager-76d4586765-w7lmz_e29d34fc-6f59-4925-af31-0047b63f4a24 became leader","type":"Normal","object":{"kind":"Lease","namespace":"openshift-kueue-operator","name":"c1f6bfd2.kueue.x-k8s.io","uid":"71c5dacc-f277-46a1-89d4-36b6039cf3f3","apiVersion":"coordination.k8s.io/v1","resourceVersion":"142117"},"reason":"LeaderElection"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.47863183Z","logger":"MultiKueueGC","caller":"multikueue/multikueuecluster.go:661","msg":"Starting Garbage Collector"} {"level":"info","ts":"2026-02-25T15:54:23.478798117Z","logger":"multikueue-admissioncheck","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.MultiKueueCluster"} {"level":"info","ts":"2026-02-25T15:54:23.47887902Z","logger":"multikueue-admissioncheck","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.AdmissionCheck"} {"level":"info","ts":"2026-02-25T15:54:23.478911067Z","logger":"multikueue-admissioncheck","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.MultiKueueConfig"} {"level":"info","ts":"2026-02-25T15:54:23.478876419Z","logger":"job","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:54:23.478937512Z","logger":"job","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Job"} {"level":"info","ts":"2026-02-25T15:54:23.47899078Z","logger":"job","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:54:23.479009867Z","logger":"leaderworkerset","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.LeaderWorkerSet"} {"level":"info","ts":"2026-02-25T15:54:23.479034679Z","logger":"job","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:54:23.479048968Z","logger":"job","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":5} {"level":"info","ts":"2026-02-25T15:54:23.479065076Z","logger":"multikueue-workload","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:54:23.479098312Z","logger":"multikueue-workload","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.MultiKueueConfig"} {"level":"info","ts":"2026-02-25T15:54:23.478935795Z","logger":"jobset","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:54:23.479202659Z","logger":"v1_pod","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:54:23.479218633Z","logger":"statefulset","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-02-25T15:54:23.479269984Z","logger":"multikueue-cluster","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002df0a0"} {"level":"info","ts":"2026-02-25T15:54:23.479312024Z","logger":"jobset","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1alpha2.JobSet"} {"level":"info","ts":"2026-02-25T15:54:23.478990311Z","logger":"multikueue-workload","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002df110"} {"level":"info","ts":"2026-02-25T15:54:23.479035148Z","logger":"leaderworkerset_pod","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-02-25T15:54:23.479407314Z","logger":"setup","caller":"roletracker/tracker.go:59","msg":"RoleTracker: transitioned to leader"} {"level":"info","ts":"2026-02-25T15:54:23.479279008Z","logger":"v1_pod","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-02-25T15:54:23.479430208Z","logger":"statefulset","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.StatefulSet"} {"level":"info","ts":"2026-02-25T15:54:23.479441315Z","logger":"multikueue-cluster","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.MultiKueueCluster"} {"level":"info","ts":"2026-02-25T15:54:23.479463892Z","logger":"multikueue-cluster","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Secret"} {"level":"info","ts":"2026-02-25T15:54:23.479584497Z","logger":"multikueue-cluster","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002df030"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.479790735Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"follower","namespace":"openshift-marketplace","name":"46657514f1e649ebaa51da741ac70392e61d69e281cd3964becce7ed5ab42c6","reconcileID":"4b63c326-7e10-4ff5-b4f6-8b586cbbe217","job":"openshift-marketplace/46657514f1e649ebaa51da741ac70392e61d69e281cd3964becce7ed5ab42c6","gvk":"batch/v1, Kind=Job","namespace":"openshift-marketplace"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.479805015Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-marketplace","name":"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76ac9","reconcileID":"a10ce01f-3c78-4c59-be98-b1c835b22d11","job":"openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76ac9","gvk":"batch/v1, Kind=Job","namespace":"openshift-marketplace"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.479848208Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-marketplace","name":"925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e56eb0c","reconcileID":"9263f448-2761-4f65-b891-1a83beb3b10f","job":"openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e56eb0c","gvk":"batch/v1, Kind=Job","namespace":"openshift-marketplace"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.480032346Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-marketplace","name":"c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78e8d070","reconcileID":"d2a96b26-2e64-441e-94de-692fef5cec4d","job":"openshift-marketplace/c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78e8d070","gvk":"batch/v1, Kind=Job","namespace":"openshift-marketplace"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.480050993Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-mwprf","reconcileID":"d4748d7e-84a5-4d4e-ab24-bf13177d201c","job":"kueue-managed-test/test-job-mwprf","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.492986141Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.JobSet","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.494641998Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.LeaderWorkerSet","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.495230317Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Secret","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.495339101Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.PriorityClass","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.495920814Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.StatefulSet","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"info","ts":"2026-02-25T15:54:23.580006576Z","logger":"jobset","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.58004821Z","logger":"jobset","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":1} {"level":"info","ts":"2026-02-25T15:54:23.580006578Z","logger":"multikueue-admissioncheck","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.580069816Z","logger":"multikueue-admissioncheck","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":1} {"level":"info","ts":"2026-02-25T15:54:23.580006225Z","logger":"leaderworkerset","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.580080288Z","logger":"leaderworkerset","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":1} {"level":"info","ts":"2026-02-25T15:54:23.580037828Z","logger":"statefulset","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.580091225Z","logger":"statefulset","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":1} {"level":"info","ts":"2026-02-25T15:54:23.580130816Z","logger":"multikueue-cluster","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.580138534Z","logger":"multikueue-cluster","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":1} {"level":"info","ts":"2026-02-25T15:54:23.580162296Z","logger":"leaderworkerset_pod","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.580169185Z","logger":"leaderworkerset_pod","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":5} {"level":"info","ts":"2026-02-25T15:54:23.580207148Z","logger":"multikueue-workload","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.580217794Z","logger":"multikueue-workload","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":5} {"level":"info","ts":"2026-02-25T15:54:23.581448472Z","logger":"v1_pod","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.581486301Z","logger":"v1_pod","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":5} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.595909321Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"14d0250c-88b6-4e65-8ab0-b09c6532a655"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.596063088Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"14d0250c-88b6-4e65-8ab0-b09c6532a655","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.59727002Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"leader","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.59739504Z","logger":"workload-reconciler","caller":"core/workload_controller.go:831","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.597499396Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"63e8c91b-221e-43b4-a3b7-c067d831381b"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.597516723Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-mwprf","reconcileID":"c2b98f22-2720-41d0-a6da-ab2b3398c53a","job":"kueue-managed-test/test-job-mwprf","gvk":"batch/v1, Kind=Job"} {"level":"debug","ts":"2026-02-25T15:54:23.59753245Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Created Workload: kueue-managed-test/job-test-job-mwprf-48c0e","type":"Normal","object":{"kind":"Job","namespace":"kueue-managed-test","name":"test-job-mwprf","uid":"eae93dbb-3e05-49df-b9c2-902ef552dd23","apiVersion":"batch/v1","resourceVersion":"142003"},"reason":"CreatedWorkload"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.608007747Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"fbdee5b1-6dd2-4c5f-8f86-b1fcb5c100eb"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.608167464Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"fbdee5b1-6dd2-4c5f-8f86-b1fcb5c100eb","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.6079943Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.608220862Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-mwprf","reconcileID":"531b9311-4487-473b-8594-708f2b9f6b60","job":"kueue-managed-test/test-job-mwprf","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.608250847Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.608316107Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"c8228417-97d6-4201-bee4-42f747ed20a0"} {"level":"Level(-2)","ts":"2026-02-25T15:54:26.584906078Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"kueue-unmanaged-test","name":"test-job-zwrrm","reconcileID":"abde3b87-0f40-4f82-84e9-bd573936408d","job":"kueue-unmanaged-test/test-job-zwrrm","gvk":"batch/v1, Kind=Job","namespace":"kueue-unmanaged-test"} {"level":"Level(-2)","ts":"2026-02-25T15:54:26.609606098Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"kueue-unmanaged-test","name":"test-job-zwrrm","reconcileID":"0b006413-f499-46bf-bf78-153b541a9638","job":"kueue-unmanaged-test/test-job-zwrrm","gvk":"batch/v1, Kind=Job","namespace":"kueue-unmanaged-test"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.198310364Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"551aeca3-1383-4804-9c23-1954904a284a"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.198408741Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.198536738Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.198653572Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"956a7f34-83e7-4968-8fcd-b4ef5b1e4b04"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.198443892Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"551aeca3-1383-4804-9c23-1954904a284a","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.212995169Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"leader","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.213453876Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"48722856-20b6-47fd-8095-8e8615d6c8eb"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.213543812Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"48722856-20b6-47fd-8095-8e8615d6c8eb","isDeleted":true} {"level":"Level(-2)","ts":"2026-02-25T15:54:30.835235027Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-dzl7v","reconcileID":"e752aa62-1aad-4717-bdfd-80dad67f456b","job":"openshift-kueue-operator/webhook-test-dzl7v","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:30.858796259Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-dzl7v","reconcileID":"15ac08f1-aaf6-4f72-b41b-c08fcc605878","job":"openshift-kueue-operator/webhook-test-dzl7v","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:34.393557978Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-dv8fw","reconcileID":"075ec704-995d-4370-8186-200f9c5542f1","job":"openshift-kueue-operator/webhook-test-dv8fw","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:34.413218051Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-dv8fw","reconcileID":"a303ebb0-0572-4258-9f37-d1a898c7f9e0","job":"openshift-kueue-operator/webhook-test-dv8fw","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:37.933881218Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-lrf58","reconcileID":"e4bbcdc9-5a76-4728-9eb7-b1b9959d445b","job":"openshift-kueue-operator/webhook-test-lrf58","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:37.955948082Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-lrf58","reconcileID":"acc8109f-e6b5-44d7-908e-031481504976","job":"openshift-kueue-operator/webhook-test-lrf58","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.297363222Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-ptq5g","reconcileID":"9bbf8981-4531-4e68-a567-e90da6cc7407","job":"kueue-managed-test/test-job-ptq5g","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.308004411Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"leader","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.308172898Z","logger":"workload-reconciler","caller":"core/workload_controller.go:831","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.308183112Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"9016c0e9-fbfa-47ff-b7f8-92657a9ede3f"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.308247305Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"9016c0e9-fbfa-47ff-b7f8-92657a9ede3f","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.308293487Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"072e3bcc-5d53-4d32-814a-17a5d1d0dbf5"} {"level":"debug","ts":"2026-02-25T15:54:38.308317696Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Created Workload: kueue-managed-test/job-test-job-ptq5g-d05b9","type":"Normal","object":{"kind":"Job","namespace":"kueue-managed-test","name":"test-job-ptq5g","uid":"5f6fcec3-8336-41b4-9a05-df83c6dbc4b4","apiVersion":"batch/v1","resourceVersion":"142270"},"reason":"CreatedWorkload"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.308350501Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-ptq5g","reconcileID":"c75832e9-ffba-4b76-b76b-d3320a212e9f","job":"kueue-managed-test/test-job-ptq5g","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.317758394Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.317828202Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.317874821Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"11048109-02d0-41d9-b41e-5dd4b02b6273"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.317777881Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"6bd89d4c-62cb-4d68-b4de-abdaa513a90f"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.317917803Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-ptq5g","reconcileID":"822f6940-afd7-45ed-9048-e8e8cded7f82","job":"kueue-managed-test/test-job-ptq5g","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.317940658Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"6bd89d4c-62cb-4d68-b4de-abdaa513a90f","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.822975545Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"kueue-unmanaged-test","name":"test-job-c7mms","reconcileID":"094cfa3b-6221-4875-9fe7-dc32f4260835","job":"kueue-unmanaged-test/test-job-c7mms","gvk":"batch/v1, Kind=Job","namespace":"kueue-unmanaged-test"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.843891064Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"kueue-unmanaged-test","name":"test-job-c7mms","reconcileID":"ce6e50c3-3bec-4ff9-9d18-16b83d9da8c1","job":"kueue-unmanaged-test/test-job-c7mms","gvk":"batch/v1, Kind=Job","namespace":"kueue-unmanaged-test"} {"level":"Level(-2)","ts":"2026-02-25T15:54:40.5800551Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-g9lg6","reconcileID":"2b85fe17-9b4a-46b0-87f1-f6b9f5e62648","job":"openshift-kueue-operator/webhook-test-g9lg6","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:40.598508257Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-g9lg6","reconcileID":"28c77f18-4284-4185-9586-7e5067bd0b15","job":"openshift-kueue-operator/webhook-test-g9lg6","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:41.66072042Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"kueue-unmanaged-test","name":"test-job-c7mms","reconcileID":"0901a6b3-b4cb-4dfd-879c-eca4b34ebd95","job":"kueue-unmanaged-test/test-job-c7mms","gvk":"batch/v1, Kind=Job","namespace":"kueue-unmanaged-test"} {"level":"Level(-2)","ts":"2026-02-25T15:54:44.121870782Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-tw6tr","reconcileID":"d9e4ee13-f09a-4de4-87a9-6a96f90670c4","job":"openshift-kueue-operator/webhook-test-tw6tr","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:44.142095921Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-tw6tr","reconcileID":"da121fbe-c306-4c8c-9061-f5882cffb292","job":"openshift-kueue-operator/webhook-test-tw6tr","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:47.636945527Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-2h4xr","reconcileID":"77e27f5b-a7c9-4a91-a417-57d9553a1829","job":"openshift-kueue-operator/webhook-test-2h4xr","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:47.656685073Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-2h4xr","reconcileID":"e0faeb79-5f5d-40ac-9f06-8ec791e4376d","job":"openshift-kueue-operator/webhook-test-2h4xr","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.535152469Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"0890d2dc-380f-45db-a161-111c7327b96d"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.535166412Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.535303136Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.535252072Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"0890d2dc-380f-45db-a161-111c7327b96d","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.535381625Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"bdc53d6e-4500-406a-9a17-66877cddd1c1"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.548638338Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"leader","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.548650975Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"3f429d71-7e77-48a7-8711-3a30a7c896e7"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.54868315Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"3f429d71-7e77-48a7-8711-3a30a7c896e7","isDeleted":true} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.179007304Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-b7dnm","reconcileID":"b1f8c609-cc34-4e4a-b463-2b43d2d2781f","job":"kueue-managed-test/test-job-b7dnm","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.187570212Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"leader","workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"queue":"test-queue","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.187579308Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"3308fff7-811d-44f4-a8c9-d2e5444b7cde"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.187672048Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"3308fff7-811d-44f4-a8c9-d2e5444b7cde","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.187736613Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"5af4000f-f01e-4e5f-9387-5efa7824087e"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.187875963Z","logger":"scheduler","caller":"scheduler/scheduler.go:248","msg":"Attempting to schedule workload","replica-role":"follower","schedulingCycle":1,"workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.187927084Z","logger":"scheduler","caller":"scheduler/scheduler.go:697","msg":"Workload assumed in the cache","replica-role":"follower","schedulingCycle":1,"workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.188975493Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-b7dnm","reconcileID":"f311f793-d174-4ee7-83c4-05f8a87d6344","job":"kueue-managed-test/test-job-b7dnm","gvk":"batch/v1, Kind=Job"} {"level":"debug","ts":"2026-02-25T15:54:49.188987061Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Created Workload: kueue-managed-test/job-test-job-b7dnm-99f6f","type":"Normal","object":{"kind":"Job","namespace":"kueue-managed-test","name":"test-job-b7dnm","uid":"56c63b2f-0462-44a1-8757-92af7ef51831","apiVersion":"batch/v1","resourceVersion":"142431"},"reason":"CreatedWorkload"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.197613757Z","logger":"scheduler","caller":"scheduler/scheduler.go:660","msg":"Workload successfully admitted and assigned flavors","replica-role":"follower","schedulingCycle":1,"workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"clusterQueue":{"name":"test-clusterqueue"},"assignments":[{"name":"main","flavors":{"cpu":"default","memory":"default"},"resourceUsage":{"cpu":"200m","memory":"512Mi"},"count":1}]} {"level":"debug","ts":"2026-02-25T15:54:49.197729126Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Quota reserved in ClusterQueue test-clusterqueue, wait time since queued was 0s","type":"Normal","object":{"kind":"Workload","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","uid":"d4ed6467-b308-417e-a3db-ba7c4d554b0c","apiVersion":"kueue.x-k8s.io/v1beta2","resourceVersion":"142436"},"reason":"QuotaReserved"} {"level":"debug","ts":"2026-02-25T15:54:49.197950237Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Admitted by ClusterQueue test-clusterqueue, wait time since reservation was 0s","type":"Normal","object":{"kind":"Workload","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","uid":"d4ed6467-b308-417e-a3db-ba7c4d554b0c","apiVersion":"kueue.x-k8s.io/v1beta2","resourceVersion":"142436"},"reason":"Admitted"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.197982053Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"aba0751d-4afb-4f88-bdea-0bd660268a40"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.198033929Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"aba0751d-4afb-4f88-bdea-0bd660268a40","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.198049313Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted","prevStatus":"pending","clusterQueue":"test-clusterqueue"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.19816536Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-b7dnm","reconcileID":"85c41867-e185-49c3-9fef-2d92cfbcbde3","job":"kueue-managed-test/test-job-b7dnm","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.198272806Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"29d8e8b0-050a-4299-b3d0-d239db21fa9b"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.198263743Z","logger":"job","caller":"jobframework/reconciler.go:598","msg":"Job admitted, unsuspending","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-b7dnm","reconcileID":"85c41867-e185-49c3-9fef-2d92cfbcbde3","job":"kueue-managed-test/test-job-b7dnm","gvk":"batch/v1, Kind=Job"} {"level":"debug","ts":"2026-02-25T15:54:49.206567237Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Admitted by clusterQueue test-clusterqueue","type":"Normal","object":{"kind":"Job","namespace":"kueue-managed-test","name":"test-job-b7dnm","uid":"56c63b2f-0462-44a1-8757-92af7ef51831","apiVersion":"batch/v1","resourceVersion":"142438"},"reason":"Started"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.206971061Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-b7dnm","reconcileID":"4776d43d-087e-404c-8214-2893aa1127f0","job":"kueue-managed-test/test-job-b7dnm","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.243256359Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-b7dnm","reconcileID":"ba747b46-40ac-48c4-833b-6ad7ef303167","job":"kueue-managed-test/test-job-b7dnm","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.722378742Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-fw8gz","reconcileID":"e8634250-df79-458c-b6e0-20d6818064e0","job":"kueue-managed-test/test-job-fw8gz","gvk":"batch/v1, Kind=Job"} {"level":"debug","ts":"2026-02-25T15:54:49.733313598Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Created Workload: kueue-managed-test/job-test-job-fw8gz-cb598","type":"Normal","object":{"kind":"Job","namespace":"kueue-managed-test","name":"test-job-fw8gz","uid":"2143be44-4cc1-4a53-bd4c-846481cb3829","apiVersion":"batch/v1","resourceVersion":"142456"},"reason":"CreatedWorkload"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.733306929Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-fw8gz","reconcileID":"8c098a2d-c793-4810-aa1d-7c6e04d826f0","job":"kueue-managed-test/test-job-fw8gz","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.7340941Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"leader","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.734141311Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"a1c60a28-4e53-4025-bfae-4fb88cf2e30a"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.734163011Z","logger":"workload-reconciler","caller":"core/workload_controller.go:831","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.734165496Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"a1c60a28-4e53-4025-bfae-4fb88cf2e30a","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.734221126Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"2c057638-7caf-4779-87f5-938e00920b62"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.743795981Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"34ff050d-8d8b-47c4-99b4-abd07d22b62d"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.743842066Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"34ff050d-8d8b-47c4-99b4-abd07d22b62d","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.743799222Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.743886353Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.743936362Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"d4251cfc-7f58-423b-9a64-c877a7c8dc6b"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.745699197Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-fw8gz","reconcileID":"50ab82f2-fadb-4c3f-ba24-4d75c683c672","job":"kueue-managed-test/test-job-fw8gz","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.745805592Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-fw8gz","reconcileID":"ed961e85-a380-4254-bec8-dc1bb48d1505","job":"kueue-managed-test/test-job-fw8gz","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:50.188819001Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:167","msg":"Reconcile LocalQueue","replica-role":"leader","namespace":"kueue-managed-test","name":"test-queue","reconcileID":"df47b88f-0e0b-47d5-abd8-3493e771286c"} {"level":"Level(-2)","ts":"2026-02-25T15:54:50.188849945Z","logger":"clusterqueue-reconciler","caller":"core/clusterqueue_controller.go:161","msg":"Reconcile ClusterQueue","replica-role":"leader","name":"test-clusterqueue","reconcileID":"465c365e-f43c-40c8-889b-de3874d2b4e0"} {"level":"Level(-2)","ts":"2026-02-25T15:54:50.199373861Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:254","msg":"Queue update event","replica-role":"leader","localQueue":{"name":"test-queue","namespace":"kueue-managed-test"}} {"level":"Level(-2)","ts":"2026-02-25T15:54:50.199608309Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:346","msg":"ClusterQueue update event","replica-role":"leader","clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T15:54:50.199806984Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:167","msg":"Reconcile LocalQueue","replica-role":"leader","namespace":"kueue-managed-test","name":"test-queue","reconcileID":"7eecdd52-f7b3-4c0c-b099-f38a83c402fc"} {"level":"Level(-2)","ts":"2026-02-25T15:54:50.19990456Z","logger":"clusterqueue-reconciler","caller":"core/clusterqueue_controller.go:161","msg":"Reconcile ClusterQueue","replica-role":"leader","name":"test-clusterqueue","reconcileID":"ba17753b-8293-4318-9d43-63f0760709e9"} {"level":"Level(-2)","ts":"2026-02-25T15:54:55.925348272Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-b7dnm","reconcileID":"12851701-68ac-4b69-a60b-d44b853b1ec7","job":"kueue-managed-test/test-job-b7dnm","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.418695762Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:95","msg":"Reconcile ResourceFlavor","replica-role":"leader","name":"default","reconcileID":"17974fed-cd8e-4ee6-9120-c14ddc74aa84"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.419682372Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:95","msg":"Reconcile ResourceFlavor","replica-role":"leader","name":"default-flavor","reconcileID":"be068d1b-f680-44e6-8993-bf0484560a81"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.419727144Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:95","msg":"Reconcile ResourceFlavor","replica-role":"leader","name":"default-flavor2","reconcileID":"41129b42-6671-4fc3-a591-b2983da17939"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.52325002Z","logger":"clusterqueue-reconciler","caller":"core/clusterqueue_controller.go:161","msg":"Reconcile ClusterQueue","replica-role":"leader","name":"cluster-queue","reconcileID":"3ca0422d-b657-4ea8-94fc-f04c1f08b7a1"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.523220996Z","logger":"tas-node-failure-reconciler","caller":"tas/node_failure_controller.go:73","msg":"Reconcile Node","replica-role":"leader","name":"ip-10-0-10-243.us-east-2.compute.internal","reconcileID":"16cc0c68-b045-48dd-83b4-62516fa7cb6f"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.523494845Z","logger":"tas-node-failure-reconciler","caller":"tas/node_failure_controller.go:73","msg":"Reconcile Node","replica-role":"leader","name":"ip-10-0-30-56.us-east-2.compute.internal","reconcileID":"2a94d5c1-d2fa-4f75-bace-53ab74350824"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.523559524Z","logger":"tas-node-failure-reconciler","caller":"tas/node_failure_controller.go:73","msg":"Reconcile Node","replica-role":"leader","name":"ip-10-0-32-68.us-east-2.compute.internal","reconcileID":"39c76157-b61d-486e-8657-2e4facf4296e"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.523615638Z","logger":"tas-node-failure-reconciler","caller":"tas/node_failure_controller.go:73","msg":"Reconcile Node","replica-role":"leader","name":"ip-10-0-48-252.us-east-2.compute.internal","reconcileID":"4f2f77b6-b983-4e85-ab88-caed448a0336"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.52367458Z","logger":"tas-node-failure-reconciler","caller":"tas/node_failure_controller.go:73","msg":"Reconcile Node","replica-role":"leader","name":"ip-10-0-66-156.us-east-2.compute.internal","reconcileID":"306dac96-1460-4da1-ab7f-1d03c587257e"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.523719741Z","logger":"tas-node-failure-reconciler","caller":"tas/node_failure_controller.go:73","msg":"Reconcile Node","replica-role":"leader","name":"ip-10-0-94-129.us-east-2.compute.internal","reconcileID":"8a0ff90c-f2e8-406a-a064-fe9991064b70"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.533833314Z","logger":"clusterqueue-reconciler","caller":"core/clusterqueue_controller.go:161","msg":"Reconcile ClusterQueue","replica-role":"leader","name":"cluster-queue2","reconcileID":"66f989ac-1097-448b-8d87-d2e45c147959"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.084278552Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.084439381Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.084338738Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"3c463d5e-0853-4834-a033-7c5c4c2e90a4"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.084537368Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"3c463d5e-0853-4834-a033-7c5c4c2e90a4","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.084503955Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"4d2a674b-ccdc-41dc-8052-98cfc109865d"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.094103263Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"leader","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.094103197Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"6dba1531-6475-4ed2-b7fd-e67f42243334"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.094256343Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"6dba1531-6475-4ed2-b7fd-e67f42243334","isDeleted":true} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.427221366Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"36711661-2013-44f7-bba5-60d02a8e509f"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.427295639Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"36711661-2013-44f7-bba5-60d02a8e509f","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.427238968Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted","clusterQueue":"test-clusterqueue"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.427470842Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"e9cb182b-cec6-4e9b-b8fb-116e861cdd2e"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.455526204Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"3498d9c9-0061-4439-83b5-cb57d068d9a8"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.455565004Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"leader","workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.455594574Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"3498d9c9-0061-4439-83b5-cb57d068d9a8","isDeleted":true} Attempt #1 Failed. Retrying ↺ @ 02/25/26 12:57:51.351 STEP: creating job in labeled namespace @ 02/25/26 12:57:51.695 STEP: verifying workload created in namespace kueue-managed-test and uid e677d513-5e59-49b3-a4a6-8031f1657633 @ 02/25/26 12:57:51.877 STEP: creating job in labeled namespace not managed by Kueue @ 02/25/26 12:57:52.236 STEP: Destroying job test-job-c74bb @ 02/25/26 13:00:52.407 STEP: Destroying job test-job-8xmxz @ 02/25/26 13:00:52.743 [FAILED] in [It] - /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:559 @ 02/25/26 13:00:53.084 STEP: Test failed - dumping kueue-controller-manager logs @ 02/25/26 13:00:53.084 === Logs from pod kueue-controller-manager-76d4586765-gt7g5 === --- Container: manager --- {"level":"info","ts":"2026-02-25T15:53:44.340461016Z","logger":"setup","caller":"kueue/main.go:579","msg":"Successfully loaded configuration","config":"apiVersion: config.kueue.x-k8s.io/v1beta2\nclientConnection:\n burst: 100\n qps: 50\ncontroller:\n groupKindConcurrency:\n ClusterQueue.kueue.x-k8s.io: 1\n Job.batch: 5\n LocalQueue.kueue.x-k8s.io: 1\n Pod: 5\n ResourceFlavor.kueue.x-k8s.io: 1\n Workload.kueue.x-k8s.io: 5\nhealth:\n healthProbeBindAddress: :8081\nintegrations:\n frameworks:\n - batch/job\n - pod\n - deployment\n - statefulset\n - leaderworkerset.x-k8s.io/leaderworkerset\n - jobset.x-k8s.io/jobset\ninternalCertManagement:\n enable: false\nkind: Configuration\nleaderElection:\n leaderElect: true\n leaseDuration: 2m17s\n renewDeadline: 1m47s\n resourceLock: leases\n resourceName: c1f6bfd2.kueue.x-k8s.io\n resourceNamespace: \"\"\n retryPeriod: 26s\nmanageJobsWithoutQueueName: true\nmanagedJobsNamespaceSelector:\n matchLabels:\n kueue.openshift.io/managed: \"true\"\nmetrics:\n bindAddress: :8443\n enableClusterQueueResources: true\nmultiKueue:\n dispatcherName: kueue.x-k8s.io/multikueue-dispatcher-all-at-once\n gcInterval: 1m0s\n origin: multikueue\n workerLostTimeout: 15m0s\nnamespace: openshift-kueue-operator\nwebhook:\n certDir: /tmp/k8s-webhook-server/serving-certs\n port: 9443\n"} {"level":"info","ts":"2026-02-25T15:53:44.340957399Z","logger":"setup","caller":"kueue/main.go:160","msg":"Initializing","gitVersion":"50b346f-dirty","gitCommit":"50b346fb172b592ffe564959eb2cb59527765417","buildDate":"2026-02-22T04:31:17Z"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.341086985Z","logger":"setup","caller":"features/kube_features.go:406","msg":"Loaded feature gates","featureGates":{"AdmissionFairSharing":true,"DynamicResourceAllocation":false,"ElasticJobsViaWorkloadSlices":false,"FailureRecoveryPolicy":false,"FlavorFungibility":true,"HierarchicalCohorts":true,"LendingLimit":true,"LocalQueueDefaulting":true,"LocalQueueMetrics":false,"ManagedJobsNamespaceSelectorAlwaysRespected":true,"MultiKueue":true,"MultiKueueAdaptersForCustomJobs":true,"MultiKueueAllowInsecureKubeconfigs":false,"MultiKueueBatchJobWithManagedBy":true,"MultiKueueClusterProfile":false,"MultiKueueRedoAdmissionOnEvictionInWorker":true,"MultiKueueWaitForWorkloadAdmitted":true,"ObjectRetentionPolicies":true,"PartialAdmission":true,"PrioritySortingWithinCohort":true,"PropagateBatchJobLabelsToWorkload":true,"ReclaimablePods":true,"RemoveFinalizersWithStrictPatch":true,"SanitizePodSets":true,"SkipFinalizersForPodsSuspendedByParent":true,"TASBalancedPlacement":false,"TASFailedNodeReplacement":true,"TASFailedNodeReplacementFailFast":true,"TASProfileLeastFreeCapacity":false,"TASProfileMixed":true,"TASReplaceNodeOnPodTermination":true,"TLSOptions":true,"TopologyAwareScheduling":true,"VisibilityOnDemand":true,"WorkloadRequestUseMergePatch":false}} {"level":"info","ts":"2026-02-25T15:53:44.341168181Z","logger":"setup","caller":"kueue/main.go:195","msg":"Initializing metrics certificate watcher using provided certificates","metrics-cert-path":"/etc/kueue/metrics/certs"} {"level":"info","ts":"2026-02-25T15:53:44.341395682Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:214","msg":"Updated current TLS certificate","cert":"/etc/kueue/metrics/certs/tls.crt","key":"/etc/kueue/metrics/certs/tls.key"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.341595861Z","logger":"setup","caller":"kueue/main.go:228","msg":"K8S Client","qps":50,"burst":100} {"level":"info","ts":"2026-02-25T15:53:44.356912395Z","logger":"setup","caller":"kueue/main.go:549","msg":"RoleTracker: leader election enabled"} {"level":"error","ts":"2026-02-25T15:53:44.358927783Z","logger":"setup","caller":"kueue/main.go:364","msg":"Skipping admission check controller setup: Provisioning Requests not supported (Possible cause: missing or unsupported cluster-autoscaler)","error":"no matches for kind \"ProvisioningRequest\" in version \"autoscaling.x-k8s.io/v1\"","stacktrace":"main.setupIndexes\n\t/workspace/upstream/kueue/src/cmd/kueue/main.go:364\nmain.main\n\t/workspace/upstream/kueue/src/cmd/kueue/main.go:298\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:285"} {"level":"info","ts":"2026-02-25T15:53:44.36036001Z","logger":"setup","caller":"kueue/main.go:506","msg":"Probe endpoints are configured on healthz and readyz"} {"level":"info","ts":"2026-02-25T15:53:44.363733741Z","logger":"setup","caller":"kueue/main.go:399","msg":"Skipping provisioning controller setup: Provisioning Requests not supported (Possible cause: missing or unsupported cluster-autoscaler)"} {"level":"info","ts":"2026-02-25T15:53:44.363993494Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"workload.codeflare.dev/v1beta2, Kind=AppWrapper","path":"/mutate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.364200276Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.364298679Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"workload.codeflare.dev/v1beta2, Kind=AppWrapper","path":"/validate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.364489979Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.364596082Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"apps/v1, Kind=Deployment","path":"/mutate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.364648156Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.364674283Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"apps/v1, Kind=Deployment","path":"/validate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.364711319Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.364825818Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"batch/v1, Kind=Job","path":"/mutate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.364868354Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.364892466Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"batch/v1, Kind=Job","path":"/validate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.364927808Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.36505134Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"jobset.x-k8s.io/v1alpha2, Kind=JobSet","path":"/mutate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.365097013Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.36512396Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"jobset.x-k8s.io/v1alpha2, Kind=JobSet","path":"/validate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.365163907Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.365236227Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=JAXJob","path":"/mutate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.365279072Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.365304314Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=JAXJob","path":"/validate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.365342609Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.365384023Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=PaddleJob","path":"/mutate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.365420423Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.365443718Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=PaddleJob","path":"/validate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.365478474Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.365522616Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=PyTorchJob","path":"/mutate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.365560908Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.365586148Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=PyTorchJob","path":"/validate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.365621753Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.365667208Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=TFJob","path":"/mutate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.365709175Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.365732952Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=TFJob","path":"/validate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.365768288Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.365807752Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=XGBoostJob","path":"/mutate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.365843187Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.365867943Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=XGBoostJob","path":"/validate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.36590428Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.365944172Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v2beta1, Kind=MPIJob","path":"/mutate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.365981326Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.36600501Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v2beta1, Kind=MPIJob","path":"/validate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.366058861Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.366129671Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"/v1, Kind=Pod","path":"/mutate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.366168454Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.366192602Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"/v1, Kind=Pod","path":"/validate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.366227307Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.366335588Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"leaderworkerset.x-k8s.io/v1, Kind=LeaderWorkerSet","path":"/mutate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.366376357Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.366401096Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"leaderworkerset.x-k8s.io/v1, Kind=LeaderWorkerSet","path":"/validate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.36643766Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.366508072Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"ray.io/v1, Kind=RayCluster","path":"/mutate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.366544727Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.366569756Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"ray.io/v1, Kind=RayCluster","path":"/validate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.366605781Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.366650474Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"ray.io/v1, Kind=RayJob","path":"/mutate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.36668701Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.366708993Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"ray.io/v1, Kind=RayJob","path":"/validate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.366746876Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.366874134Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"apps/v1, Kind=StatefulSet","path":"/mutate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.366947915Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.366974333Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"apps/v1, Kind=StatefulSet","path":"/validate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.367012744Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.367111915Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"trainer.kubeflow.org/v1alpha1, Kind=TrainJob","path":"/mutate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.367151586Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.367175979Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"trainer.kubeflow.org/v1alpha1, Kind=TrainJob","path":"/validate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.367211817Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.367251848Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=Workload","path":"/mutate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.367288031Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.36731295Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=Workload","path":"/validate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.367349904Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.367412599Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/convert"} {"level":"info","ts":"2026-02-25T15:53:44.367420421Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=Workload"} {"level":"info","ts":"2026-02-25T15:53:44.367442335Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ResourceFlavor","path":"/mutate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.367478578Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.367502036Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ResourceFlavor","path":"/validate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.36753712Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.367585005Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ClusterQueue","path":"/mutate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.367621525Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.367645385Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ClusterQueue","path":"/validate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.367681207Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.367711727Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.367737372Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=Cohort","path":"/validate-kueue-x-k8s-io-v1beta2-cohort"} {"level":"info","ts":"2026-02-25T15:53:44.367772249Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-cohort"} {"level":"info","ts":"2026-02-25T15:53:44.367801074Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=Cohort"} {"level":"info","ts":"2026-02-25T15:53:44.367830265Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=LocalQueue"} {"level":"info","ts":"2026-02-25T15:53:44.36785223Z","logger":"setup","caller":"kueue/main.go:349","msg":"Starting manager"} {"level":"info","ts":"2026-02-25T15:53:44.367907557Z","logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} {"level":"info","ts":"2026-02-25T15:53:44.368055438Z","logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":8443","secure":true} {"level":"info","ts":"2026-02-25T15:53:44.368056519Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:191","msg":"Starting webhook server"} {"level":"info","ts":"2026-02-25T15:53:44.368433564Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:214","msg":"Updated current TLS certificate","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key"} {"level":"info","ts":"2026-02-25T15:53:44.368550355Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:242","msg":"Serving webhook server","host":"","port":9443} {"level":"info","ts":"2026-02-25T15:53:44.368039059Z","caller":"manager/server.go:83","msg":"starting server","name":"health probe","addr":"[::]:8081"} {"level":"info","ts":"2026-02-25T15:53:44.368687314Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:136","msg":"Starting certificate poll+watcher","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key","interval":10} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.372585545Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.MultiKueueConfig","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.373057275Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.AdmissionCheck","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.373358959Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Job","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.373366805Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.MultiKueueCluster","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.374965877Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.LimitRange","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.37519478Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.ResourceFlavor","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.375658616Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.LocalQueue","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.376417194Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Node","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.377615608Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.Workload","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.402616147Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"info","ts":"2026-02-25T15:53:44.469051647Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:136","msg":"Starting certificate poll+watcher","cert":"/etc/kueue/metrics/certs/tls.crt","key":"/etc/kueue/metrics/certs/tls.key","interval":10} {"level":"info","ts":"2026-02-25T15:53:44.469189233Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0005323f0"} {"level":"info","ts":"2026-02-25T15:53:44.469314418Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.AdmissionCheck"} {"level":"info","ts":"2026-02-25T15:53:44.469208079Z","logger":"localqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0005324d0"} {"level":"info","ts":"2026-02-25T15:53:44.469385041Z","logger":"localqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.469405585Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.469356113Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc000532700"} {"level":"info","ts":"2026-02-25T15:53:44.469359686Z","logger":"workloadpriorityclass-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.WorkloadPriorityClass"} {"level":"info","ts":"2026-02-25T15:53:44.469386427Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Namespace"} {"level":"info","ts":"2026-02-25T15:53:44.469343721Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ResourceFlavor"} {"level":"info","ts":"2026-02-25T15:53:44.469528143Z","logger":"localqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.LocalQueue"} {"level":"info","ts":"2026-02-25T15:53:44.469312616Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc000532310"} {"level":"info","ts":"2026-02-25T15:53:44.469614645Z","logger":"cohort-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0005325b0"} {"level":"info","ts":"2026-02-25T15:53:44.469668395Z","logger":"cohort-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Cohort"} {"level":"info","ts":"2026-02-25T15:53:44.469515708Z","logger":"tas-topology-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Topology"} {"level":"info","ts":"2026-02-25T15:53:44.469707353Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0005327e0"} {"level":"info","ts":"2026-02-25T15:53:44.469744939Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.LimitRange"} {"level":"info","ts":"2026-02-25T15:53:44.469791414Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.RuntimeClass"} {"level":"info","ts":"2026-02-25T15:53:44.469841502Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.469884951Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.LocalQueue"} {"level":"info","ts":"2026-02-25T15:53:44.469591047Z","logger":"tas-topology-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ResourceFlavor"} {"level":"info","ts":"2026-02-25T15:53:44.469945773Z","logger":"tas-topology-ungater","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-02-25T15:53:44.469919433Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:53:44.469997455Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Node"} {"level":"info","ts":"2026-02-25T15:53:44.469977487Z","logger":"tas-non-tas-usage-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"standalone","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-02-25T15:53:44.470091876Z","logger":"tas-node-failure-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Node"} {"level":"info","ts":"2026-02-25T15:53:44.469892497Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ResourceFlavor"} {"level":"info","ts":"2026-02-25T15:53:44.46980967Z","logger":"tas-topology-ungater","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} I0225 15:53:44.470329 1 leaderelection.go:258] "Attempting to acquire leader lease..." lock="openshift-kueue-operator/c1f6bfd2.kueue.x-k8s.io" {"level":"Level(-2)","ts":"2026-02-25T15:53:44.470399883Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:141","msg":"ResourceFlavor create event","replica-role":"follower","resourceFlavor":{"name":"default"}} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.470488034Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:215","msg":"LocalQueue create event","replica-role":"follower","localQueue":{"name":"test-queue","namespace":"kueue-managed-test"}} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.470636107Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:141","msg":"ResourceFlavor create event","replica-role":"follower","resourceFlavor":{"name":"default-flavor"}} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.470689046Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:141","msg":"ResourceFlavor create event","replica-role":"follower","resourceFlavor":{"name":"default-flavor2"}} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.475644953Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.ClusterQueue","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.475967039Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.RuntimeClass","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.476247521Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.WorkloadPriorityClass","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.4764467Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.Cohort","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.477298878Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.Topology","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.480087522Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Namespace","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} I0225 15:53:44.568784 1 plugins.go:157] Loaded 2 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,MutatingAdmissionPolicy. {"level":"info","ts":"2026-02-25T15:53:44.56993691Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.570001145Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.570206671Z","logger":"cohort-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.570244158Z","logger":"cohort-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.570328551Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:313","msg":"ClusterQueue create event","replica-role":"follower","clusterQueue":{"name":"cluster-queue"}} {"level":"info","ts":"2026-02-25T15:53:44.570435838Z","logger":"localqueue-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.570454802Z","logger":"localqueue-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.570494443Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:313","msg":"ClusterQueue create event","replica-role":"follower","clusterQueue":{"name":"cluster-queue2"}} {"level":"info","ts":"2026-02-25T15:53:44.570559442Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.570575526Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.57064351Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:313","msg":"ClusterQueue create event","replica-role":"follower","clusterQueue":{"name":"test-clusterqueue"}} {"level":"info","ts":"2026-02-25T15:53:44.570676238Z","logger":"tas-non-tas-usage-controller","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"standalone"} {"level":"info","ts":"2026-02-25T15:53:44.570690809Z","logger":"tas-non-tas-usage-controller","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"standalone","worker count":5} {"level":"info","ts":"2026-02-25T15:53:44.570821375Z","logger":"workloadpriorityclass-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.570851465Z","logger":"workloadpriorityclass-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.571494162Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.571551869Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.573127464Z","logger":"tas-node-failure-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.573192531Z","logger":"tas-node-failure-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.574116015Z","logger":"workload-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.574172258Z","logger":"workload-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":5} {"level":"info","ts":"2026-02-25T15:53:44.574296766Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.574326301Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} I0225 15:53:44.575096 1 handler.go:304] Adding GroupVersion visibility.kueue.x-k8s.io v1beta2 to ResourceManager I0225 15:53:44.575551 1 handler.go:304] Adding GroupVersion visibility.kueue.x-k8s.io v1beta1 to ResourceManager {"level":"info","ts":"2026-02-25T15:53:44.584440915Z","logger":"tas-topology-ungater","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.584552636Z","logger":"tas-topology-ungater","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":5} {"level":"info","ts":"2026-02-25T15:53:44.584778753Z","logger":"tas-topology-controller","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.58483679Z","logger":"tas-topology-controller","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} I0225 15:53:44.588925 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0225 15:53:44.588950 1 shared_informer.go:370] "Waiting for caches to sync" I0225 15:53:44.588968 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0225 15:53:44.588975 1 shared_informer.go:370] "Waiting for caches to sync" I0225 15:53:44.588986 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0225 15:53:44.588992 1 shared_informer.go:370] "Waiting for caches to sync" I0225 15:53:44.589364 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/visibility/tls.crt::/visibility/tls.key" I0225 15:53:44.589555 1 secure_serving.go:211] Serving securely on [::]:8082 I0225 15:53:44.589583 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" I0225 15:53:44.589632 1 apf_controller.go:377] Starting API Priority and Fairness config controller I0225 15:53:44.689891 1 apf_controller.go:382] Running API Priority and Fairness config worker I0225 15:53:44.689968 1 apf_controller.go:385] Running API Priority and Fairness periodic rebalancing process I0225 15:53:44.689922 1 shared_informer.go:377] "Caches are synced" I0225 15:53:44.689935 1 shared_informer.go:377] "Caches are synced" I0225 15:53:44.689938 1 shared_informer.go:377] "Caches are synced" {"level":"Level(-2)","ts":"2026-02-25T15:54:23.590111739Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"follower","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.590207371Z","logger":"workload-reconciler","caller":"core/workload_controller.go:831","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.608000117Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.608087502Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.19812975Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.198195966Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.212175285Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"follower","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.30798512Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"follower","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.308068183Z","logger":"workload-reconciler","caller":"core/workload_controller.go:831","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.317818492Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.317858668Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.534874458Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.534950465Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.548692336Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"follower","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.188895592Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"follower","workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"queue":"test-queue","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.197532922Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted","prevStatus":"pending","clusterQueue":"test-clusterqueue"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.733111904Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"follower","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.733173254Z","logger":"workload-reconciler","caller":"core/workload_controller.go:831","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.745537428Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.745598368Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:50.198748709Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:254","msg":"Queue update event","replica-role":"follower","localQueue":{"name":"test-queue","namespace":"kueue-managed-test"}} {"level":"Level(-2)","ts":"2026-02-25T15:54:50.199456463Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:346","msg":"ClusterQueue update event","replica-role":"follower","clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.084404451Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.08447968Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.094073562Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"follower","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.42723648Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted","clusterQueue":"test-clusterqueue"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.455541678Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"follower","workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.438076375Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:254","msg":"Queue update event","replica-role":"follower","localQueue":{"name":"test-queue","namespace":"kueue-managed-test"}} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.439229042Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:346","msg":"ClusterQueue update event","replica-role":"follower","clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.834404238Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"follower","workload":{"name":"job-test-job-8xmxz-b852d","namespace":"kueue-managed-test"},"queue":"test-queue","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.846591004Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-8xmxz-b852d","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted","prevStatus":"pending","clusterQueue":"test-clusterqueue"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.367637023Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"follower","workload":{"name":"job-test-job-c74bb-45a97","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.367693782Z","logger":"workload-reconciler","caller":"core/workload_controller.go:831","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-c74bb-45a97","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.376200517Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-c74bb-45a97","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.376256578Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-c74bb-45a97","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.844884492Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:254","msg":"Queue update event","replica-role":"follower","localQueue":{"name":"test-queue","namespace":"kueue-managed-test"}} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.845995225Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:346","msg":"ClusterQueue update event","replica-role":"follower","clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T16:00:52.70289662Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-c74bb-45a97","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T16:00:52.702967497Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-c74bb-45a97","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T16:00:52.714717848Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"follower","workload":{"name":"job-test-job-c74bb-45a97","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T16:00:53.043155422Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-8xmxz-b852d","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted","clusterQueue":"test-clusterqueue"} {"level":"Level(-2)","ts":"2026-02-25T16:00:53.070412922Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"follower","workload":{"name":"job-test-job-8xmxz-b852d","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted"} === Logs from pod kueue-controller-manager-76d4586765-w7lmz === --- Container: manager --- {"level":"info","ts":"2026-02-25T15:53:44.279621583Z","logger":"setup","caller":"kueue/main.go:579","msg":"Successfully loaded configuration","config":"apiVersion: config.kueue.x-k8s.io/v1beta2\nclientConnection:\n burst: 100\n qps: 50\ncontroller:\n groupKindConcurrency:\n ClusterQueue.kueue.x-k8s.io: 1\n Job.batch: 5\n LocalQueue.kueue.x-k8s.io: 1\n Pod: 5\n ResourceFlavor.kueue.x-k8s.io: 1\n Workload.kueue.x-k8s.io: 5\nhealth:\n healthProbeBindAddress: :8081\nintegrations:\n frameworks:\n - batch/job\n - pod\n - deployment\n - statefulset\n - leaderworkerset.x-k8s.io/leaderworkerset\n - jobset.x-k8s.io/jobset\ninternalCertManagement:\n enable: false\nkind: Configuration\nleaderElection:\n leaderElect: true\n leaseDuration: 2m17s\n renewDeadline: 1m47s\n resourceLock: leases\n resourceName: c1f6bfd2.kueue.x-k8s.io\n resourceNamespace: \"\"\n retryPeriod: 26s\nmanageJobsWithoutQueueName: true\nmanagedJobsNamespaceSelector:\n matchLabels:\n kueue.openshift.io/managed: \"true\"\nmetrics:\n bindAddress: :8443\n enableClusterQueueResources: true\nmultiKueue:\n dispatcherName: kueue.x-k8s.io/multikueue-dispatcher-all-at-once\n gcInterval: 1m0s\n origin: multikueue\n workerLostTimeout: 15m0s\nnamespace: openshift-kueue-operator\nwebhook:\n certDir: /tmp/k8s-webhook-server/serving-certs\n port: 9443\n"} {"level":"info","ts":"2026-02-25T15:53:44.279747887Z","logger":"setup","caller":"kueue/main.go:160","msg":"Initializing","gitVersion":"50b346f-dirty","gitCommit":"50b346fb172b592ffe564959eb2cb59527765417","buildDate":"2026-02-22T04:31:17Z"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.281674849Z","logger":"setup","caller":"features/kube_features.go:406","msg":"Loaded feature gates","featureGates":{"AdmissionFairSharing":true,"DynamicResourceAllocation":false,"ElasticJobsViaWorkloadSlices":false,"FailureRecoveryPolicy":false,"FlavorFungibility":true,"HierarchicalCohorts":true,"LendingLimit":true,"LocalQueueDefaulting":true,"LocalQueueMetrics":false,"ManagedJobsNamespaceSelectorAlwaysRespected":true,"MultiKueue":true,"MultiKueueAdaptersForCustomJobs":true,"MultiKueueAllowInsecureKubeconfigs":false,"MultiKueueBatchJobWithManagedBy":true,"MultiKueueClusterProfile":false,"MultiKueueRedoAdmissionOnEvictionInWorker":true,"MultiKueueWaitForWorkloadAdmitted":true,"ObjectRetentionPolicies":true,"PartialAdmission":true,"PrioritySortingWithinCohort":true,"PropagateBatchJobLabelsToWorkload":true,"ReclaimablePods":true,"RemoveFinalizersWithStrictPatch":true,"SanitizePodSets":true,"SkipFinalizersForPodsSuspendedByParent":true,"TASBalancedPlacement":false,"TASFailedNodeReplacement":true,"TASFailedNodeReplacementFailFast":true,"TASProfileLeastFreeCapacity":false,"TASProfileMixed":true,"TASReplaceNodeOnPodTermination":true,"TLSOptions":true,"TopologyAwareScheduling":true,"VisibilityOnDemand":true,"WorkloadRequestUseMergePatch":false}} {"level":"info","ts":"2026-02-25T15:53:44.281778129Z","logger":"setup","caller":"kueue/main.go:195","msg":"Initializing metrics certificate watcher using provided certificates","metrics-cert-path":"/etc/kueue/metrics/certs"} {"level":"info","ts":"2026-02-25T15:53:44.282148722Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:214","msg":"Updated current TLS certificate","cert":"/etc/kueue/metrics/certs/tls.crt","key":"/etc/kueue/metrics/certs/tls.key"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.282434386Z","logger":"setup","caller":"kueue/main.go:228","msg":"K8S Client","qps":50,"burst":100} {"level":"info","ts":"2026-02-25T15:53:44.30091856Z","logger":"setup","caller":"kueue/main.go:549","msg":"RoleTracker: leader election enabled"} {"level":"error","ts":"2026-02-25T15:53:44.303506867Z","logger":"setup","caller":"kueue/main.go:364","msg":"Skipping admission check controller setup: Provisioning Requests not supported (Possible cause: missing or unsupported cluster-autoscaler)","error":"no matches for kind \"ProvisioningRequest\" in version \"autoscaling.x-k8s.io/v1\"","stacktrace":"main.setupIndexes\n\t/workspace/upstream/kueue/src/cmd/kueue/main.go:364\nmain.main\n\t/workspace/upstream/kueue/src/cmd/kueue/main.go:298\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:285"} {"level":"info","ts":"2026-02-25T15:53:44.305857965Z","logger":"setup","caller":"kueue/main.go:506","msg":"Probe endpoints are configured on healthz and readyz"} {"level":"info","ts":"2026-02-25T15:53:44.307849898Z","logger":"setup","caller":"kueue/main.go:399","msg":"Skipping provisioning controller setup: Provisioning Requests not supported (Possible cause: missing or unsupported cluster-autoscaler)"} {"level":"info","ts":"2026-02-25T15:53:44.308011898Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"workload.codeflare.dev/v1beta2, Kind=AppWrapper","path":"/mutate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.308091229Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.308135679Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"workload.codeflare.dev/v1beta2, Kind=AppWrapper","path":"/validate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.308186638Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.308277059Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"apps/v1, Kind=Deployment","path":"/mutate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.308325654Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.308385596Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"apps/v1, Kind=Deployment","path":"/validate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.308431886Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.3085592Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"batch/v1, Kind=Job","path":"/mutate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.308602344Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.308621786Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"batch/v1, Kind=Job","path":"/validate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.308658489Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.308757891Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"jobset.x-k8s.io/v1alpha2, Kind=JobSet","path":"/mutate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.308797523Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.308815349Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"jobset.x-k8s.io/v1alpha2, Kind=JobSet","path":"/validate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.308863601Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.308934062Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=JAXJob","path":"/mutate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.308968472Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.308992373Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=JAXJob","path":"/validate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.309025871Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.309077677Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=PaddleJob","path":"/mutate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.309129509Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.309163402Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=PaddleJob","path":"/validate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.309200142Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.309264058Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=PyTorchJob","path":"/mutate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.309298958Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.309324593Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=PyTorchJob","path":"/validate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.30936498Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.309430139Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=TFJob","path":"/mutate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.309520135Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.309549091Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=TFJob","path":"/validate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.309603256Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.309682302Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=XGBoostJob","path":"/mutate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.309751505Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.309788546Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=XGBoostJob","path":"/validate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.309834512Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.30988365Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v2beta1, Kind=MPIJob","path":"/mutate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.309924823Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.309954839Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v2beta1, Kind=MPIJob","path":"/validate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.309995907Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.310079654Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"/v1, Kind=Pod","path":"/mutate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.310139549Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.310164284Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"/v1, Kind=Pod","path":"/validate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.310214291Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.310381361Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"leaderworkerset.x-k8s.io/v1, Kind=LeaderWorkerSet","path":"/mutate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.31051083Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.31056551Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"leaderworkerset.x-k8s.io/v1, Kind=LeaderWorkerSet","path":"/validate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.310626805Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.310751222Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"ray.io/v1, Kind=RayCluster","path":"/mutate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.310817583Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.310856509Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"ray.io/v1, Kind=RayCluster","path":"/validate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.310918893Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.310990165Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"ray.io/v1, Kind=RayJob","path":"/mutate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.311051318Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.311088179Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"ray.io/v1, Kind=RayJob","path":"/validate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.311499756Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.312242487Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"apps/v1, Kind=StatefulSet","path":"/mutate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.31249166Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.31253884Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"apps/v1, Kind=StatefulSet","path":"/validate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.312589856Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.312885251Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"trainer.kubeflow.org/v1alpha1, Kind=TrainJob","path":"/mutate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.312941616Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.312986347Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"trainer.kubeflow.org/v1alpha1, Kind=TrainJob","path":"/validate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.313049129Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.313214607Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=Workload","path":"/mutate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.313306305Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.313376373Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=Workload","path":"/validate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.313456132Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.313629481Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/convert"} {"level":"info","ts":"2026-02-25T15:53:44.313659096Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=Workload"} {"level":"info","ts":"2026-02-25T15:53:44.313710299Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ResourceFlavor","path":"/mutate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.313787755Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.313837229Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ResourceFlavor","path":"/validate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.313908388Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.314048405Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ClusterQueue","path":"/mutate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.314725043Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.314801746Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ClusterQueue","path":"/validate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.314860746Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.31494798Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.314990652Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=Cohort","path":"/validate-kueue-x-k8s-io-v1beta2-cohort"} {"level":"info","ts":"2026-02-25T15:53:44.315034476Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-cohort"} {"level":"info","ts":"2026-02-25T15:53:44.315147712Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=Cohort"} {"level":"info","ts":"2026-02-25T15:53:44.315236212Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=LocalQueue"} {"level":"info","ts":"2026-02-25T15:53:44.315258026Z","logger":"setup","caller":"kueue/main.go:349","msg":"Starting manager"} {"level":"info","ts":"2026-02-25T15:53:44.31533328Z","logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} {"level":"info","ts":"2026-02-25T15:53:44.315471423Z","logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":8443","secure":true} {"level":"info","ts":"2026-02-25T15:53:44.316260303Z","caller":"manager/server.go:83","msg":"starting server","name":"health probe","addr":"[::]:8081"} {"level":"info","ts":"2026-02-25T15:53:44.31633774Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:191","msg":"Starting webhook server"} {"level":"info","ts":"2026-02-25T15:53:44.31671217Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:214","msg":"Updated current TLS certificate","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key"} {"level":"info","ts":"2026-02-25T15:53:44.316798854Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:242","msg":"Serving webhook server","host":"","port":9443} {"level":"info","ts":"2026-02-25T15:53:44.317317806Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:136","msg":"Starting certificate poll+watcher","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key","interval":10} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.322565594Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.LimitRange","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.324181786Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Job","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.324418016Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.AdmissionCheck","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.334771556Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.LocalQueue","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.335081697Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.MultiKueueCluster","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.33950372Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.MultiKueueConfig","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.340560729Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Node","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.341150449Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.ResourceFlavor","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.354242079Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.Workload","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.392841688Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"info","ts":"2026-02-25T15:53:44.417945012Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:136","msg":"Starting certificate poll+watcher","cert":"/etc/kueue/metrics/certs/tls.crt","key":"/etc/kueue/metrics/certs/tls.key","interval":10} {"level":"info","ts":"2026-02-25T15:53:44.418063438Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002de850"} {"level":"info","ts":"2026-02-25T15:53:44.418190033Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002de700"} {"level":"info","ts":"2026-02-25T15:53:44.418248952Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ResourceFlavor"} {"level":"info","ts":"2026-02-25T15:53:44.418306847Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.418323341Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.418336783Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.AdmissionCheck"} {"level":"info","ts":"2026-02-25T15:53:44.418364917Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.418379128Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.418412681Z","logger":"localqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002de930"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.418457955Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:141","msg":"ResourceFlavor create event","replica-role":"follower","resourceFlavor":{"name":"default"}} {"level":"info","ts":"2026-02-25T15:53:44.418543799Z","logger":"workloadpriorityclass-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.WorkloadPriorityClass"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.418599616Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:141","msg":"ResourceFlavor create event","replica-role":"follower","resourceFlavor":{"name":"default-flavor"}} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.418627025Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:141","msg":"ResourceFlavor create event","replica-role":"follower","resourceFlavor":{"name":"default-flavor2"}} {"level":"info","ts":"2026-02-25T15:53:44.41866598Z","logger":"localqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.LocalQueue"} {"level":"info","ts":"2026-02-25T15:53:44.418793947Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002debd0"} {"level":"info","ts":"2026-02-25T15:53:44.418835076Z","logger":"localqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.418934927Z","logger":"cohort-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002dea10"} {"level":"info","ts":"2026-02-25T15:53:44.41902137Z","logger":"tas-topology-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Topology"} {"level":"info","ts":"2026-02-25T15:53:44.419069043Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Namespace"} {"level":"info","ts":"2026-02-25T15:53:44.419259296Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.419330123Z","logger":"cohort-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Cohort"} {"level":"info","ts":"2026-02-25T15:53:44.419517674Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002decb0"} {"level":"info","ts":"2026-02-25T15:53:44.419615479Z","logger":"tas-topology-ungater","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:53:44.419680116Z","logger":"tas-topology-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ResourceFlavor"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.419850793Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:215","msg":"LocalQueue create event","replica-role":"follower","localQueue":{"name":"test-queue","namespace":"kueue-managed-test"}} {"level":"info","ts":"2026-02-25T15:53:44.420079153Z","logger":"tas-topology-ungater","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-02-25T15:53:44.420087499Z","logger":"tas-non-tas-usage-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"standalone","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-02-25T15:53:44.420203472Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.LimitRange"} {"level":"info","ts":"2026-02-25T15:53:44.420264114Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.RuntimeClass"} {"level":"info","ts":"2026-02-25T15:53:44.420312116Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.42035359Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.LocalQueue"} {"level":"info","ts":"2026-02-25T15:53:44.420394273Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:53:44.420513225Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ResourceFlavor"} {"level":"info","ts":"2026-02-25T15:53:44.42061171Z","logger":"tas-node-failure-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Node"} {"level":"info","ts":"2026-02-25T15:53:44.420691072Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Node"} I0225 15:53:44.420838 1 leaderelection.go:258] "Attempting to acquire leader lease..." lock="openshift-kueue-operator/c1f6bfd2.kueue.x-k8s.io" {"level":"Level(-2)","ts":"2026-02-25T15:53:44.424441062Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.Topology","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.425208312Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.Cohort","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.425494622Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.WorkloadPriorityClass","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.425642886Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.RuntimeClass","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.425729818Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.ClusterQueue","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.429091863Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Namespace","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"info","ts":"2026-02-25T15:53:44.519303959Z","logger":"workloadpriorityclass-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.519420284Z","logger":"workloadpriorityclass-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.519413528Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:313","msg":"ClusterQueue create event","replica-role":"follower","clusterQueue":{"name":"cluster-queue"}} {"level":"info","ts":"2026-02-25T15:53:44.519598564Z","logger":"localqueue-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.519657428Z","logger":"localqueue-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.519611865Z","logger":"cohort-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.519742302Z","logger":"cohort-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.519623116Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:313","msg":"ClusterQueue create event","replica-role":"follower","clusterQueue":{"name":"cluster-queue2"}} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.519898159Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:313","msg":"ClusterQueue create event","replica-role":"follower","clusterQueue":{"name":"test-clusterqueue"}} {"level":"info","ts":"2026-02-25T15:53:44.520142016Z","logger":"tas-topology-controller","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.520162874Z","logger":"tas-topology-controller","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.520715763Z","logger":"tas-topology-ungater","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.520735396Z","logger":"tas-topology-ungater","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":5} {"level":"info","ts":"2026-02-25T15:53:44.52193053Z","logger":"workload-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.522003655Z","logger":"workload-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":5} {"level":"info","ts":"2026-02-25T15:53:44.522003161Z","logger":"tas-non-tas-usage-controller","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"standalone"} {"level":"info","ts":"2026-02-25T15:53:44.522044978Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.522092774Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.521979407Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.522012377Z","logger":"tas-node-failure-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.522128959Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.522134259Z","logger":"tas-node-failure-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.522052297Z","logger":"tas-non-tas-usage-controller","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"standalone","worker count":5} I0225 15:53:44.640704 1 plugins.go:157] Loaded 2 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,MutatingAdmissionPolicy. I0225 15:53:44.644255 1 handler.go:304] Adding GroupVersion visibility.kueue.x-k8s.io v1beta2 to ResourceManager I0225 15:53:44.644803 1 handler.go:304] Adding GroupVersion visibility.kueue.x-k8s.io v1beta1 to ResourceManager I0225 15:53:44.653678 1 secure_serving.go:211] Serving securely on [::]:8082 I0225 15:53:44.653813 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0225 15:53:44.653846 1 shared_informer.go:370] "Waiting for caches to sync" I0225 15:53:44.653881 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/visibility/tls.crt::/visibility/tls.key" I0225 15:53:44.653979 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" I0225 15:53:44.654449 1 apf_controller.go:377] Starting API Priority and Fairness config controller I0225 15:53:44.654499 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0225 15:53:44.654529 1 shared_informer.go:370] "Waiting for caches to sync" I0225 15:53:44.654578 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0225 15:53:44.654589 1 shared_informer.go:370] "Waiting for caches to sync" I0225 15:53:44.753987 1 shared_informer.go:377] "Caches are synced" I0225 15:53:44.755178 1 shared_informer.go:377] "Caches are synced" I0225 15:53:44.755183 1 shared_informer.go:377] "Caches are synced" I0225 15:53:44.755194 1 apf_controller.go:382] Running API Priority and Fairness config worker I0225 15:53:44.755303 1 apf_controller.go:385] Running API Priority and Fairness periodic rebalancing process I0225 15:54:23.478467 1 leaderelection.go:272] "Successfully acquired lease" lock="openshift-kueue-operator/c1f6bfd2.kueue.x-k8s.io" {"level":"debug","ts":"2026-02-25T15:54:23.47848655Z","logger":"events","caller":"recorder/recorder.go:104","msg":"kueue-controller-manager-76d4586765-w7lmz_e29d34fc-6f59-4925-af31-0047b63f4a24 became leader","type":"Normal","object":{"kind":"Lease","namespace":"openshift-kueue-operator","name":"c1f6bfd2.kueue.x-k8s.io","uid":"71c5dacc-f277-46a1-89d4-36b6039cf3f3","apiVersion":"coordination.k8s.io/v1","resourceVersion":"142117"},"reason":"LeaderElection"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.47863183Z","logger":"MultiKueueGC","caller":"multikueue/multikueuecluster.go:661","msg":"Starting Garbage Collector"} {"level":"info","ts":"2026-02-25T15:54:23.478798117Z","logger":"multikueue-admissioncheck","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.MultiKueueCluster"} {"level":"info","ts":"2026-02-25T15:54:23.47887902Z","logger":"multikueue-admissioncheck","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.AdmissionCheck"} {"level":"info","ts":"2026-02-25T15:54:23.478911067Z","logger":"multikueue-admissioncheck","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.MultiKueueConfig"} {"level":"info","ts":"2026-02-25T15:54:23.478876419Z","logger":"job","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:54:23.478937512Z","logger":"job","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Job"} {"level":"info","ts":"2026-02-25T15:54:23.47899078Z","logger":"job","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:54:23.479009867Z","logger":"leaderworkerset","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.LeaderWorkerSet"} {"level":"info","ts":"2026-02-25T15:54:23.479034679Z","logger":"job","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:54:23.479048968Z","logger":"job","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":5} {"level":"info","ts":"2026-02-25T15:54:23.479065076Z","logger":"multikueue-workload","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:54:23.479098312Z","logger":"multikueue-workload","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.MultiKueueConfig"} {"level":"info","ts":"2026-02-25T15:54:23.478935795Z","logger":"jobset","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:54:23.479202659Z","logger":"v1_pod","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:54:23.479218633Z","logger":"statefulset","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-02-25T15:54:23.479269984Z","logger":"multikueue-cluster","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002df0a0"} {"level":"info","ts":"2026-02-25T15:54:23.479312024Z","logger":"jobset","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1alpha2.JobSet"} {"level":"info","ts":"2026-02-25T15:54:23.478990311Z","logger":"multikueue-workload","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002df110"} {"level":"info","ts":"2026-02-25T15:54:23.479035148Z","logger":"leaderworkerset_pod","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-02-25T15:54:23.479407314Z","logger":"setup","caller":"roletracker/tracker.go:59","msg":"RoleTracker: transitioned to leader"} {"level":"info","ts":"2026-02-25T15:54:23.479279008Z","logger":"v1_pod","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-02-25T15:54:23.479430208Z","logger":"statefulset","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.StatefulSet"} {"level":"info","ts":"2026-02-25T15:54:23.479441315Z","logger":"multikueue-cluster","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.MultiKueueCluster"} {"level":"info","ts":"2026-02-25T15:54:23.479463892Z","logger":"multikueue-cluster","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Secret"} {"level":"info","ts":"2026-02-25T15:54:23.479584497Z","logger":"multikueue-cluster","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002df030"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.479790735Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"follower","namespace":"openshift-marketplace","name":"46657514f1e649ebaa51da741ac70392e61d69e281cd3964becce7ed5ab42c6","reconcileID":"4b63c326-7e10-4ff5-b4f6-8b586cbbe217","job":"openshift-marketplace/46657514f1e649ebaa51da741ac70392e61d69e281cd3964becce7ed5ab42c6","gvk":"batch/v1, Kind=Job","namespace":"openshift-marketplace"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.479805015Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-marketplace","name":"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76ac9","reconcileID":"a10ce01f-3c78-4c59-be98-b1c835b22d11","job":"openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76ac9","gvk":"batch/v1, Kind=Job","namespace":"openshift-marketplace"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.479848208Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-marketplace","name":"925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e56eb0c","reconcileID":"9263f448-2761-4f65-b891-1a83beb3b10f","job":"openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e56eb0c","gvk":"batch/v1, Kind=Job","namespace":"openshift-marketplace"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.480032346Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-marketplace","name":"c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78e8d070","reconcileID":"d2a96b26-2e64-441e-94de-692fef5cec4d","job":"openshift-marketplace/c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78e8d070","gvk":"batch/v1, Kind=Job","namespace":"openshift-marketplace"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.480050993Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-mwprf","reconcileID":"d4748d7e-84a5-4d4e-ab24-bf13177d201c","job":"kueue-managed-test/test-job-mwprf","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.492986141Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.JobSet","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.494641998Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.LeaderWorkerSet","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.495230317Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Secret","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.495339101Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.PriorityClass","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.495920814Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.StatefulSet","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"info","ts":"2026-02-25T15:54:23.580006576Z","logger":"jobset","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.58004821Z","logger":"jobset","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":1} {"level":"info","ts":"2026-02-25T15:54:23.580006578Z","logger":"multikueue-admissioncheck","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.580069816Z","logger":"multikueue-admissioncheck","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":1} {"level":"info","ts":"2026-02-25T15:54:23.580006225Z","logger":"leaderworkerset","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.580080288Z","logger":"leaderworkerset","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":1} {"level":"info","ts":"2026-02-25T15:54:23.580037828Z","logger":"statefulset","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.580091225Z","logger":"statefulset","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":1} {"level":"info","ts":"2026-02-25T15:54:23.580130816Z","logger":"multikueue-cluster","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.580138534Z","logger":"multikueue-cluster","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":1} {"level":"info","ts":"2026-02-25T15:54:23.580162296Z","logger":"leaderworkerset_pod","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.580169185Z","logger":"leaderworkerset_pod","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":5} {"level":"info","ts":"2026-02-25T15:54:23.580207148Z","logger":"multikueue-workload","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.580217794Z","logger":"multikueue-workload","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":5} {"level":"info","ts":"2026-02-25T15:54:23.581448472Z","logger":"v1_pod","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.581486301Z","logger":"v1_pod","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":5} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.595909321Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"14d0250c-88b6-4e65-8ab0-b09c6532a655"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.596063088Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"14d0250c-88b6-4e65-8ab0-b09c6532a655","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.59727002Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"leader","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.59739504Z","logger":"workload-reconciler","caller":"core/workload_controller.go:831","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.597499396Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"63e8c91b-221e-43b4-a3b7-c067d831381b"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.597516723Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-mwprf","reconcileID":"c2b98f22-2720-41d0-a6da-ab2b3398c53a","job":"kueue-managed-test/test-job-mwprf","gvk":"batch/v1, Kind=Job"} {"level":"debug","ts":"2026-02-25T15:54:23.59753245Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Created Workload: kueue-managed-test/job-test-job-mwprf-48c0e","type":"Normal","object":{"kind":"Job","namespace":"kueue-managed-test","name":"test-job-mwprf","uid":"eae93dbb-3e05-49df-b9c2-902ef552dd23","apiVersion":"batch/v1","resourceVersion":"142003"},"reason":"CreatedWorkload"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.608007747Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"fbdee5b1-6dd2-4c5f-8f86-b1fcb5c100eb"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.608167464Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"fbdee5b1-6dd2-4c5f-8f86-b1fcb5c100eb","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.6079943Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.608220862Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-mwprf","reconcileID":"531b9311-4487-473b-8594-708f2b9f6b60","job":"kueue-managed-test/test-job-mwprf","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.608250847Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.608316107Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"c8228417-97d6-4201-bee4-42f747ed20a0"} {"level":"Level(-2)","ts":"2026-02-25T15:54:26.584906078Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"kueue-unmanaged-test","name":"test-job-zwrrm","reconcileID":"abde3b87-0f40-4f82-84e9-bd573936408d","job":"kueue-unmanaged-test/test-job-zwrrm","gvk":"batch/v1, Kind=Job","namespace":"kueue-unmanaged-test"} {"level":"Level(-2)","ts":"2026-02-25T15:54:26.609606098Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"kueue-unmanaged-test","name":"test-job-zwrrm","reconcileID":"0b006413-f499-46bf-bf78-153b541a9638","job":"kueue-unmanaged-test/test-job-zwrrm","gvk":"batch/v1, Kind=Job","namespace":"kueue-unmanaged-test"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.198310364Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"551aeca3-1383-4804-9c23-1954904a284a"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.198408741Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.198536738Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.198653572Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"956a7f34-83e7-4968-8fcd-b4ef5b1e4b04"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.198443892Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"551aeca3-1383-4804-9c23-1954904a284a","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.212995169Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"leader","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.213453876Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"48722856-20b6-47fd-8095-8e8615d6c8eb"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.213543812Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"48722856-20b6-47fd-8095-8e8615d6c8eb","isDeleted":true} {"level":"Level(-2)","ts":"2026-02-25T15:54:30.835235027Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-dzl7v","reconcileID":"e752aa62-1aad-4717-bdfd-80dad67f456b","job":"openshift-kueue-operator/webhook-test-dzl7v","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:30.858796259Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-dzl7v","reconcileID":"15ac08f1-aaf6-4f72-b41b-c08fcc605878","job":"openshift-kueue-operator/webhook-test-dzl7v","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:34.393557978Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-dv8fw","reconcileID":"075ec704-995d-4370-8186-200f9c5542f1","job":"openshift-kueue-operator/webhook-test-dv8fw","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:34.413218051Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-dv8fw","reconcileID":"a303ebb0-0572-4258-9f37-d1a898c7f9e0","job":"openshift-kueue-operator/webhook-test-dv8fw","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:37.933881218Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-lrf58","reconcileID":"e4bbcdc9-5a76-4728-9eb7-b1b9959d445b","job":"openshift-kueue-operator/webhook-test-lrf58","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:37.955948082Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-lrf58","reconcileID":"acc8109f-e6b5-44d7-908e-031481504976","job":"openshift-kueue-operator/webhook-test-lrf58","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.297363222Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-ptq5g","reconcileID":"9bbf8981-4531-4e68-a567-e90da6cc7407","job":"kueue-managed-test/test-job-ptq5g","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.308004411Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"leader","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.308172898Z","logger":"workload-reconciler","caller":"core/workload_controller.go:831","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.308183112Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"9016c0e9-fbfa-47ff-b7f8-92657a9ede3f"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.308247305Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"9016c0e9-fbfa-47ff-b7f8-92657a9ede3f","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.308293487Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"072e3bcc-5d53-4d32-814a-17a5d1d0dbf5"} {"level":"debug","ts":"2026-02-25T15:54:38.308317696Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Created Workload: kueue-managed-test/job-test-job-ptq5g-d05b9","type":"Normal","object":{"kind":"Job","namespace":"kueue-managed-test","name":"test-job-ptq5g","uid":"5f6fcec3-8336-41b4-9a05-df83c6dbc4b4","apiVersion":"batch/v1","resourceVersion":"142270"},"reason":"CreatedWorkload"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.308350501Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-ptq5g","reconcileID":"c75832e9-ffba-4b76-b76b-d3320a212e9f","job":"kueue-managed-test/test-job-ptq5g","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.317758394Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.317828202Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.317874821Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"11048109-02d0-41d9-b41e-5dd4b02b6273"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.317777881Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"6bd89d4c-62cb-4d68-b4de-abdaa513a90f"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.317917803Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-ptq5g","reconcileID":"822f6940-afd7-45ed-9048-e8e8cded7f82","job":"kueue-managed-test/test-job-ptq5g","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.317940658Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"6bd89d4c-62cb-4d68-b4de-abdaa513a90f","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.822975545Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"kueue-unmanaged-test","name":"test-job-c7mms","reconcileID":"094cfa3b-6221-4875-9fe7-dc32f4260835","job":"kueue-unmanaged-test/test-job-c7mms","gvk":"batch/v1, Kind=Job","namespace":"kueue-unmanaged-test"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.843891064Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"kueue-unmanaged-test","name":"test-job-c7mms","reconcileID":"ce6e50c3-3bec-4ff9-9d18-16b83d9da8c1","job":"kueue-unmanaged-test/test-job-c7mms","gvk":"batch/v1, Kind=Job","namespace":"kueue-unmanaged-test"} {"level":"Level(-2)","ts":"2026-02-25T15:54:40.5800551Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-g9lg6","reconcileID":"2b85fe17-9b4a-46b0-87f1-f6b9f5e62648","job":"openshift-kueue-operator/webhook-test-g9lg6","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:40.598508257Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-g9lg6","reconcileID":"28c77f18-4284-4185-9586-7e5067bd0b15","job":"openshift-kueue-operator/webhook-test-g9lg6","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:41.66072042Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"kueue-unmanaged-test","name":"test-job-c7mms","reconcileID":"0901a6b3-b4cb-4dfd-879c-eca4b34ebd95","job":"kueue-unmanaged-test/test-job-c7mms","gvk":"batch/v1, Kind=Job","namespace":"kueue-unmanaged-test"} {"level":"Level(-2)","ts":"2026-02-25T15:54:44.121870782Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-tw6tr","reconcileID":"d9e4ee13-f09a-4de4-87a9-6a96f90670c4","job":"openshift-kueue-operator/webhook-test-tw6tr","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:44.142095921Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-tw6tr","reconcileID":"da121fbe-c306-4c8c-9061-f5882cffb292","job":"openshift-kueue-operator/webhook-test-tw6tr","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:47.636945527Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-2h4xr","reconcileID":"77e27f5b-a7c9-4a91-a417-57d9553a1829","job":"openshift-kueue-operator/webhook-test-2h4xr","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:47.656685073Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-2h4xr","reconcileID":"e0faeb79-5f5d-40ac-9f06-8ec791e4376d","job":"openshift-kueue-operator/webhook-test-2h4xr","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.535152469Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"0890d2dc-380f-45db-a161-111c7327b96d"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.535166412Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.535303136Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.535252072Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"0890d2dc-380f-45db-a161-111c7327b96d","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.535381625Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"bdc53d6e-4500-406a-9a17-66877cddd1c1"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.548638338Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"leader","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.548650975Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"3f429d71-7e77-48a7-8711-3a30a7c896e7"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.54868315Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"3f429d71-7e77-48a7-8711-3a30a7c896e7","isDeleted":true} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.179007304Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-b7dnm","reconcileID":"b1f8c609-cc34-4e4a-b463-2b43d2d2781f","job":"kueue-managed-test/test-job-b7dnm","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.187570212Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"leader","workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"queue":"test-queue","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.187579308Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"3308fff7-811d-44f4-a8c9-d2e5444b7cde"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.187672048Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"3308fff7-811d-44f4-a8c9-d2e5444b7cde","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.187736613Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"5af4000f-f01e-4e5f-9387-5efa7824087e"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.187875963Z","logger":"scheduler","caller":"scheduler/scheduler.go:248","msg":"Attempting to schedule workload","replica-role":"follower","schedulingCycle":1,"workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.187927084Z","logger":"scheduler","caller":"scheduler/scheduler.go:697","msg":"Workload assumed in the cache","replica-role":"follower","schedulingCycle":1,"workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.188975493Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-b7dnm","reconcileID":"f311f793-d174-4ee7-83c4-05f8a87d6344","job":"kueue-managed-test/test-job-b7dnm","gvk":"batch/v1, Kind=Job"} {"level":"debug","ts":"2026-02-25T15:54:49.188987061Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Created Workload: kueue-managed-test/job-test-job-b7dnm-99f6f","type":"Normal","object":{"kind":"Job","namespace":"kueue-managed-test","name":"test-job-b7dnm","uid":"56c63b2f-0462-44a1-8757-92af7ef51831","apiVersion":"batch/v1","resourceVersion":"142431"},"reason":"CreatedWorkload"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.197613757Z","logger":"scheduler","caller":"scheduler/scheduler.go:660","msg":"Workload successfully admitted and assigned flavors","replica-role":"follower","schedulingCycle":1,"workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"clusterQueue":{"name":"test-clusterqueue"},"assignments":[{"name":"main","flavors":{"cpu":"default","memory":"default"},"resourceUsage":{"cpu":"200m","memory":"512Mi"},"count":1}]} {"level":"debug","ts":"2026-02-25T15:54:49.197729126Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Quota reserved in ClusterQueue test-clusterqueue, wait time since queued was 0s","type":"Normal","object":{"kind":"Workload","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","uid":"d4ed6467-b308-417e-a3db-ba7c4d554b0c","apiVersion":"kueue.x-k8s.io/v1beta2","resourceVersion":"142436"},"reason":"QuotaReserved"} {"level":"debug","ts":"2026-02-25T15:54:49.197950237Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Admitted by ClusterQueue test-clusterqueue, wait time since reservation was 0s","type":"Normal","object":{"kind":"Workload","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","uid":"d4ed6467-b308-417e-a3db-ba7c4d554b0c","apiVersion":"kueue.x-k8s.io/v1beta2","resourceVersion":"142436"},"reason":"Admitted"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.197982053Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"aba0751d-4afb-4f88-bdea-0bd660268a40"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.198033929Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"aba0751d-4afb-4f88-bdea-0bd660268a40","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.198049313Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted","prevStatus":"pending","clusterQueue":"test-clusterqueue"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.19816536Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-b7dnm","reconcileID":"85c41867-e185-49c3-9fef-2d92cfbcbde3","job":"kueue-managed-test/test-job-b7dnm","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.198272806Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"29d8e8b0-050a-4299-b3d0-d239db21fa9b"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.198263743Z","logger":"job","caller":"jobframework/reconciler.go:598","msg":"Job admitted, unsuspending","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-b7dnm","reconcileID":"85c41867-e185-49c3-9fef-2d92cfbcbde3","job":"kueue-managed-test/test-job-b7dnm","gvk":"batch/v1, Kind=Job"} {"level":"debug","ts":"2026-02-25T15:54:49.206567237Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Admitted by clusterQueue test-clusterqueue","type":"Normal","object":{"kind":"Job","namespace":"kueue-managed-test","name":"test-job-b7dnm","uid":"56c63b2f-0462-44a1-8757-92af7ef51831","apiVersion":"batch/v1","resourceVersion":"142438"},"reason":"Started"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.206971061Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-b7dnm","reconcileID":"4776d43d-087e-404c-8214-2893aa1127f0","job":"kueue-managed-test/test-job-b7dnm","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.243256359Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-b7dnm","reconcileID":"ba747b46-40ac-48c4-833b-6ad7ef303167","job":"kueue-managed-test/test-job-b7dnm","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.722378742Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-fw8gz","reconcileID":"e8634250-df79-458c-b6e0-20d6818064e0","job":"kueue-managed-test/test-job-fw8gz","gvk":"batch/v1, Kind=Job"} {"level":"debug","ts":"2026-02-25T15:54:49.733313598Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Created Workload: kueue-managed-test/job-test-job-fw8gz-cb598","type":"Normal","object":{"kind":"Job","namespace":"kueue-managed-test","name":"test-job-fw8gz","uid":"2143be44-4cc1-4a53-bd4c-846481cb3829","apiVersion":"batch/v1","resourceVersion":"142456"},"reason":"CreatedWorkload"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.733306929Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-fw8gz","reconcileID":"8c098a2d-c793-4810-aa1d-7c6e04d826f0","job":"kueue-managed-test/test-job-fw8gz","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.7340941Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"leader","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.734141311Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"a1c60a28-4e53-4025-bfae-4fb88cf2e30a"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.734163011Z","logger":"workload-reconciler","caller":"core/workload_controller.go:831","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.734165496Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"a1c60a28-4e53-4025-bfae-4fb88cf2e30a","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.734221126Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"2c057638-7caf-4779-87f5-938e00920b62"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.743795981Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"34ff050d-8d8b-47c4-99b4-abd07d22b62d"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.743842066Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"34ff050d-8d8b-47c4-99b4-abd07d22b62d","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.743799222Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.743886353Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.743936362Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"d4251cfc-7f58-423b-9a64-c877a7c8dc6b"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.745699197Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-fw8gz","reconcileID":"50ab82f2-fadb-4c3f-ba24-4d75c683c672","job":"kueue-managed-test/test-job-fw8gz","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.745805592Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-fw8gz","reconcileID":"ed961e85-a380-4254-bec8-dc1bb48d1505","job":"kueue-managed-test/test-job-fw8gz","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:50.188819001Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:167","msg":"Reconcile LocalQueue","replica-role":"leader","namespace":"kueue-managed-test","name":"test-queue","reconcileID":"df47b88f-0e0b-47d5-abd8-3493e771286c"} {"level":"Level(-2)","ts":"2026-02-25T15:54:50.188849945Z","logger":"clusterqueue-reconciler","caller":"core/clusterqueue_controller.go:161","msg":"Reconcile ClusterQueue","replica-role":"leader","name":"test-clusterqueue","reconcileID":"465c365e-f43c-40c8-889b-de3874d2b4e0"} {"level":"Level(-2)","ts":"2026-02-25T15:54:50.199373861Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:254","msg":"Queue update event","replica-role":"leader","localQueue":{"name":"test-queue","namespace":"kueue-managed-test"}} {"level":"Level(-2)","ts":"2026-02-25T15:54:50.199608309Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:346","msg":"ClusterQueue update event","replica-role":"leader","clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T15:54:50.199806984Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:167","msg":"Reconcile LocalQueue","replica-role":"leader","namespace":"kueue-managed-test","name":"test-queue","reconcileID":"7eecdd52-f7b3-4c0c-b099-f38a83c402fc"} {"level":"Level(-2)","ts":"2026-02-25T15:54:50.19990456Z","logger":"clusterqueue-reconciler","caller":"core/clusterqueue_controller.go:161","msg":"Reconcile ClusterQueue","replica-role":"leader","name":"test-clusterqueue","reconcileID":"ba17753b-8293-4318-9d43-63f0760709e9"} {"level":"Level(-2)","ts":"2026-02-25T15:54:55.925348272Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-b7dnm","reconcileID":"12851701-68ac-4b69-a60b-d44b853b1ec7","job":"kueue-managed-test/test-job-b7dnm","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.418695762Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:95","msg":"Reconcile ResourceFlavor","replica-role":"leader","name":"default","reconcileID":"17974fed-cd8e-4ee6-9120-c14ddc74aa84"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.419682372Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:95","msg":"Reconcile ResourceFlavor","replica-role":"leader","name":"default-flavor","reconcileID":"be068d1b-f680-44e6-8993-bf0484560a81"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.419727144Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:95","msg":"Reconcile ResourceFlavor","replica-role":"leader","name":"default-flavor2","reconcileID":"41129b42-6671-4fc3-a591-b2983da17939"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.52325002Z","logger":"clusterqueue-reconciler","caller":"core/clusterqueue_controller.go:161","msg":"Reconcile ClusterQueue","replica-role":"leader","name":"cluster-queue","reconcileID":"3ca0422d-b657-4ea8-94fc-f04c1f08b7a1"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.523220996Z","logger":"tas-node-failure-reconciler","caller":"tas/node_failure_controller.go:73","msg":"Reconcile Node","replica-role":"leader","name":"ip-10-0-10-243.us-east-2.compute.internal","reconcileID":"16cc0c68-b045-48dd-83b4-62516fa7cb6f"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.523494845Z","logger":"tas-node-failure-reconciler","caller":"tas/node_failure_controller.go:73","msg":"Reconcile Node","replica-role":"leader","name":"ip-10-0-30-56.us-east-2.compute.internal","reconcileID":"2a94d5c1-d2fa-4f75-bace-53ab74350824"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.523559524Z","logger":"tas-node-failure-reconciler","caller":"tas/node_failure_controller.go:73","msg":"Reconcile Node","replica-role":"leader","name":"ip-10-0-32-68.us-east-2.compute.internal","reconcileID":"39c76157-b61d-486e-8657-2e4facf4296e"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.523615638Z","logger":"tas-node-failure-reconciler","caller":"tas/node_failure_controller.go:73","msg":"Reconcile Node","replica-role":"leader","name":"ip-10-0-48-252.us-east-2.compute.internal","reconcileID":"4f2f77b6-b983-4e85-ab88-caed448a0336"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.52367458Z","logger":"tas-node-failure-reconciler","caller":"tas/node_failure_controller.go:73","msg":"Reconcile Node","replica-role":"leader","name":"ip-10-0-66-156.us-east-2.compute.internal","reconcileID":"306dac96-1460-4da1-ab7f-1d03c587257e"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.523719741Z","logger":"tas-node-failure-reconciler","caller":"tas/node_failure_controller.go:73","msg":"Reconcile Node","replica-role":"leader","name":"ip-10-0-94-129.us-east-2.compute.internal","reconcileID":"8a0ff90c-f2e8-406a-a064-fe9991064b70"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.533833314Z","logger":"clusterqueue-reconciler","caller":"core/clusterqueue_controller.go:161","msg":"Reconcile ClusterQueue","replica-role":"leader","name":"cluster-queue2","reconcileID":"66f989ac-1097-448b-8d87-d2e45c147959"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.084278552Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.084439381Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.084338738Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"3c463d5e-0853-4834-a033-7c5c4c2e90a4"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.084537368Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"3c463d5e-0853-4834-a033-7c5c4c2e90a4","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.084503955Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"4d2a674b-ccdc-41dc-8052-98cfc109865d"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.094103263Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"leader","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.094103197Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"6dba1531-6475-4ed2-b7fd-e67f42243334"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.094256343Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"6dba1531-6475-4ed2-b7fd-e67f42243334","isDeleted":true} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.427221366Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"36711661-2013-44f7-bba5-60d02a8e509f"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.427295639Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"36711661-2013-44f7-bba5-60d02a8e509f","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.427238968Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted","clusterQueue":"test-clusterqueue"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.427470842Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"e9cb182b-cec6-4e9b-b8fb-116e861cdd2e"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.455526204Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"3498d9c9-0061-4439-83b5-cb57d068d9a8"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.455565004Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"leader","workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.455594574Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"3498d9c9-0061-4439-83b5-cb57d068d9a8","isDeleted":true} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.427979086Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:167","msg":"Reconcile LocalQueue","replica-role":"leader","namespace":"kueue-managed-test","name":"test-queue","reconcileID":"0d7c7c53-f69c-44be-9ad0-681111dade23"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.427981178Z","logger":"clusterqueue-reconciler","caller":"core/clusterqueue_controller.go:161","msg":"Reconcile ClusterQueue","replica-role":"leader","name":"test-clusterqueue","reconcileID":"eccc175e-6e4c-4c82-94f3-29f6fc7b6cbb"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.438916396Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:254","msg":"Queue update event","replica-role":"leader","localQueue":{"name":"test-queue","namespace":"kueue-managed-test"}} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.439022964Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:167","msg":"Reconcile LocalQueue","replica-role":"leader","namespace":"kueue-managed-test","name":"test-queue","reconcileID":"7cc60293-6b0a-4698-90e5-e74ed5738bff"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.440744364Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:346","msg":"ClusterQueue update event","replica-role":"leader","clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.440900864Z","logger":"clusterqueue-reconciler","caller":"core/clusterqueue_controller.go:161","msg":"Reconcile ClusterQueue","replica-role":"leader","name":"test-clusterqueue","reconcileID":"23e92830-d39a-475c-a416-bdad48d43d60"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.824456514Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-8xmxz","reconcileID":"e0336057-e71f-417e-b62b-1cfbb68d6a48","job":"kueue-managed-test/test-job-8xmxz","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.834447588Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-8xmxz","reconcileID":"c4804e44-0061-4be2-b20b-d3eb82d21295","job":"kueue-managed-test/test-job-8xmxz","gvk":"batch/v1, Kind=Job"} {"level":"debug","ts":"2026-02-25T15:57:51.834555519Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Created Workload: kueue-managed-test/job-test-job-8xmxz-b852d","type":"Normal","object":{"kind":"Job","namespace":"kueue-managed-test","name":"test-job-8xmxz","uid":"e677d513-5e59-49b3-a4a6-8031f1657633","apiVersion":"batch/v1","resourceVersion":"143859"},"reason":"CreatedWorkload"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.834772837Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-8xmxz-b852d","reconcileID":"b32b985c-1328-44a5-b1a3-7b7b969ca2f3"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.834874736Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-8xmxz-b852d","reconcileID":"b32b985c-1328-44a5-b1a3-7b7b969ca2f3","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.834809681Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"leader","workload":{"name":"job-test-job-8xmxz-b852d","namespace":"kueue-managed-test"},"queue":"test-queue","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.83510326Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-8xmxz-b852d","reconcileID":"1bae949e-5f18-4ed3-8ab2-afafa226b005"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.835210722Z","logger":"scheduler","caller":"scheduler/scheduler.go:248","msg":"Attempting to schedule workload","replica-role":"follower","schedulingCycle":2,"workload":{"name":"job-test-job-8xmxz-b852d","namespace":"kueue-managed-test"},"clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.835276282Z","logger":"scheduler","caller":"scheduler/scheduler.go:697","msg":"Workload assumed in the cache","replica-role":"follower","schedulingCycle":2,"workload":{"name":"job-test-job-8xmxz-b852d","namespace":"kueue-managed-test"},"clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.846643224Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-8xmxz","reconcileID":"2c7eacb3-76ac-4f46-abcb-1c61a8b20b79","job":"kueue-managed-test/test-job-8xmxz","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.846748063Z","logger":"scheduler","caller":"scheduler/scheduler.go:660","msg":"Workload successfully admitted and assigned flavors","replica-role":"follower","schedulingCycle":2,"workload":{"name":"job-test-job-8xmxz-b852d","namespace":"kueue-managed-test"},"clusterQueue":{"name":"test-clusterqueue"},"assignments":[{"name":"main","flavors":{"cpu":"default","memory":"default"},"resourceUsage":{"cpu":"200m","memory":"512Mi"},"count":1}]} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.846778969Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-8xmxz-b852d","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted","prevStatus":"pending","clusterQueue":"test-clusterqueue"} {"level":"debug","ts":"2026-02-25T15:57:51.84683832Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Quota reserved in ClusterQueue test-clusterqueue, wait time since queued was 1s","type":"Normal","object":{"kind":"Workload","namespace":"kueue-managed-test","name":"job-test-job-8xmxz-b852d","uid":"be76319c-1657-4941-bd8c-b056e1db83e8","apiVersion":"kueue.x-k8s.io/v1beta2","resourceVersion":"143864"},"reason":"QuotaReserved"} {"level":"debug","ts":"2026-02-25T15:57:51.846897916Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Admitted by ClusterQueue test-clusterqueue, wait time since reservation was 0s","type":"Normal","object":{"kind":"Workload","namespace":"kueue-managed-test","name":"job-test-job-8xmxz-b852d","uid":"be76319c-1657-4941-bd8c-b056e1db83e8","apiVersion":"kueue.x-k8s.io/v1beta2","resourceVersion":"143864"},"reason":"Admitted"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.846850889Z","logger":"job","caller":"jobframework/reconciler.go:598","msg":"Job admitted, unsuspending","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-8xmxz","reconcileID":"2c7eacb3-76ac-4f46-abcb-1c61a8b20b79","job":"kueue-managed-test/test-job-8xmxz","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.846930825Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-8xmxz-b852d","reconcileID":"4481cf24-7ba1-4312-82f5-f24e04c5e735"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.846951987Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-8xmxz-b852d","reconcileID":"40546c1b-1006-4f87-8fc6-ae9188d38633"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.846999705Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-8xmxz-b852d","reconcileID":"40546c1b-1006-4f87-8fc6-ae9188d38633","isDeleted":false} {"level":"debug","ts":"2026-02-25T15:57:51.85601836Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Admitted by clusterQueue test-clusterqueue","type":"Normal","object":{"kind":"Job","namespace":"kueue-managed-test","name":"test-job-8xmxz","uid":"e677d513-5e59-49b3-a4a6-8031f1657633","apiVersion":"batch/v1","resourceVersion":"143866"},"reason":"Started"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.856069032Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-8xmxz","reconcileID":"36d011db-870f-4b23-a723-9589dc7463f8","job":"kueue-managed-test/test-job-8xmxz","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.856231329Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-8xmxz","reconcileID":"f49c770d-58da-48d5-b7df-5c7c7226b148","job":"kueue-managed-test/test-job-8xmxz","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.890242775Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-8xmxz","reconcileID":"55dd61b6-4eda-4776-b9e6-e89791000d5c","job":"kueue-managed-test/test-job-8xmxz","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.358416465Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-c74bb","reconcileID":"090ae3bd-e0b6-47c9-8e2a-f2dd982ead80","job":"kueue-managed-test/test-job-c74bb","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.366729836Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-c74bb-45a97","reconcileID":"e085382a-d043-4b1d-9d4a-748bad2bdaaf"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.366819823Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-c74bb-45a97","reconcileID":"e085382a-d043-4b1d-9d4a-748bad2bdaaf","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.36673599Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"leader","workload":{"name":"job-test-job-c74bb-45a97","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.36688869Z","logger":"workload-reconciler","caller":"core/workload_controller.go:831","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-c74bb-45a97","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.366938364Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-c74bb-45a97","reconcileID":"30cb6a9d-3060-49c2-af53-a1facce8bb90"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.367964129Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-c74bb","reconcileID":"1b1e76e3-10a7-44a2-93ae-a04dfcf0a31b","job":"kueue-managed-test/test-job-c74bb","gvk":"batch/v1, Kind=Job"} {"level":"debug","ts":"2026-02-25T15:57:52.367974739Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Created Workload: kueue-managed-test/job-test-job-c74bb-45a97","type":"Normal","object":{"kind":"Job","namespace":"kueue-managed-test","name":"test-job-c74bb","uid":"80ee4f54-69a7-479d-8010-36d76047a2cb","apiVersion":"batch/v1","resourceVersion":"143882"},"reason":"CreatedWorkload"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.37620912Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-c74bb-45a97","reconcileID":"a38ea54f-af34-4775-877b-16ca7693e7d0"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.376252802Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-c74bb-45a97","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.376267207Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-c74bb-45a97","reconcileID":"a38ea54f-af34-4775-877b-16ca7693e7d0","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.376306638Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-c74bb-45a97","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.376360532Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-c74bb-45a97","reconcileID":"b9177d6d-9084-4e8a-8ee3-73ef18635a31"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.37635675Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-c74bb","reconcileID":"781576ad-9c0e-4eda-ad5f-cf074b7e8882","job":"kueue-managed-test/test-job-c74bb","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.835840643Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:167","msg":"Reconcile LocalQueue","replica-role":"leader","namespace":"kueue-managed-test","name":"test-queue","reconcileID":"1b318c95-922c-4b7a-9ebf-7cf38ba278f9"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.835842794Z","logger":"clusterqueue-reconciler","caller":"core/clusterqueue_controller.go:161","msg":"Reconcile ClusterQueue","replica-role":"leader","name":"test-clusterqueue","reconcileID":"c029edd9-d514-4b81-84ad-e0e7d53d535a"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.845241842Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:254","msg":"Queue update event","replica-role":"leader","localQueue":{"name":"test-queue","namespace":"kueue-managed-test"}} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.845380277Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:167","msg":"Reconcile LocalQueue","replica-role":"leader","namespace":"kueue-managed-test","name":"test-queue","reconcileID":"eb2b3439-1685-4aa9-87ea-d64f9839f552"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.846865127Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:346","msg":"ClusterQueue update event","replica-role":"leader","clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.847035515Z","logger":"clusterqueue-reconciler","caller":"core/clusterqueue_controller.go:161","msg":"Reconcile ClusterQueue","replica-role":"leader","name":"test-clusterqueue","reconcileID":"7f24134e-1a0a-4747-87ec-b97080828282"} {"level":"Level(-2)","ts":"2026-02-25T15:57:54.483219685Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-8xmxz","reconcileID":"209edf6a-e8c5-4ec9-afc0-93432d9c7631","job":"kueue-managed-test/test-job-8xmxz","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T16:00:52.702921597Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-c74bb-45a97","reconcileID":"56c32a69-0807-4839-b190-a3e56bc5d26a"} {"level":"Level(-2)","ts":"2026-02-25T16:00:52.703064735Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-c74bb-45a97","reconcileID":"56c32a69-0807-4839-b190-a3e56bc5d26a","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T16:00:52.702994165Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-c74bb-45a97","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T16:00:52.703156108Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-c74bb-45a97","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T16:00:52.703225822Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-c74bb-45a97","reconcileID":"9bc7a4ee-6c2d-429e-bacb-861ad02a9cb5"} {"level":"Level(-2)","ts":"2026-02-25T16:00:52.714842149Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"leader","workload":{"name":"job-test-job-c74bb-45a97","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T16:00:52.714838594Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-c74bb-45a97","reconcileID":"879472a5-6ce5-440f-bd31-f4ed7ef6d4c8"} {"level":"Level(-2)","ts":"2026-02-25T16:00:52.714961967Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-c74bb-45a97","reconcileID":"879472a5-6ce5-440f-bd31-f4ed7ef6d4c8","isDeleted":true} {"level":"Level(-2)","ts":"2026-02-25T16:00:53.042959562Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-8xmxz-b852d","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted","clusterQueue":"test-clusterqueue"} {"level":"Level(-2)","ts":"2026-02-25T16:00:53.043024628Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-8xmxz-b852d","reconcileID":"60e5ee9a-1d47-430f-aed1-aecbcaf005eb"} {"level":"Level(-2)","ts":"2026-02-25T16:00:53.043090876Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-8xmxz-b852d","reconcileID":"60e5ee9a-1d47-430f-aed1-aecbcaf005eb","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T16:00:53.043204166Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-8xmxz-b852d","reconcileID":"a4847ba6-7196-42e0-ac48-11522922d227"} {"level":"Level(-2)","ts":"2026-02-25T16:00:53.070454093Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-8xmxz-b852d","reconcileID":"1a216286-193d-4f40-8279-dd73d0d47b0f"} {"level":"Level(-2)","ts":"2026-02-25T16:00:53.070568122Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-8xmxz-b852d","reconcileID":"1a216286-193d-4f40-8279-dd73d0d47b0f","isDeleted":true} {"level":"Level(-2)","ts":"2026-02-25T16:00:53.070501725Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"leader","workload":{"name":"job-test-job-8xmxz-b852d","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted"} Attempt #2 Failed. Retrying ↺ @ 02/25/26 13:00:54.251 STEP: creating job in labeled namespace @ 02/25/26 13:00:54.577 STEP: verifying workload created in namespace kueue-managed-test and uid 4a2dca97-8326-41ed-9c11-905797eefd97 @ 02/25/26 13:00:54.755 STEP: creating job in labeled namespace not managed by Kueue @ 02/25/26 13:00:55.084 STEP: Destroying job test-job-2bkxc @ 02/25/26 13:03:55.248 STEP: Destroying job test-job-869lt @ 02/25/26 13:03:55.597 [FAILED] in [It] - /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:559 @ 02/25/26 13:03:55.959 STEP: Test failed - dumping kueue-controller-manager logs @ 02/25/26 13:03:55.96 === Logs from pod kueue-controller-manager-76d4586765-gt7g5 === --- Container: manager --- {"level":"info","ts":"2026-02-25T15:53:44.340461016Z","logger":"setup","caller":"kueue/main.go:579","msg":"Successfully loaded configuration","config":"apiVersion: config.kueue.x-k8s.io/v1beta2\nclientConnection:\n burst: 100\n qps: 50\ncontroller:\n groupKindConcurrency:\n ClusterQueue.kueue.x-k8s.io: 1\n Job.batch: 5\n LocalQueue.kueue.x-k8s.io: 1\n Pod: 5\n ResourceFlavor.kueue.x-k8s.io: 1\n Workload.kueue.x-k8s.io: 5\nhealth:\n healthProbeBindAddress: :8081\nintegrations:\n frameworks:\n - batch/job\n - pod\n - deployment\n - statefulset\n - leaderworkerset.x-k8s.io/leaderworkerset\n - jobset.x-k8s.io/jobset\ninternalCertManagement:\n enable: false\nkind: Configuration\nleaderElection:\n leaderElect: true\n leaseDuration: 2m17s\n renewDeadline: 1m47s\n resourceLock: leases\n resourceName: c1f6bfd2.kueue.x-k8s.io\n resourceNamespace: \"\"\n retryPeriod: 26s\nmanageJobsWithoutQueueName: true\nmanagedJobsNamespaceSelector:\n matchLabels:\n kueue.openshift.io/managed: \"true\"\nmetrics:\n bindAddress: :8443\n enableClusterQueueResources: true\nmultiKueue:\n dispatcherName: kueue.x-k8s.io/multikueue-dispatcher-all-at-once\n gcInterval: 1m0s\n origin: multikueue\n workerLostTimeout: 15m0s\nnamespace: openshift-kueue-operator\nwebhook:\n certDir: /tmp/k8s-webhook-server/serving-certs\n port: 9443\n"} {"level":"info","ts":"2026-02-25T15:53:44.340957399Z","logger":"setup","caller":"kueue/main.go:160","msg":"Initializing","gitVersion":"50b346f-dirty","gitCommit":"50b346fb172b592ffe564959eb2cb59527765417","buildDate":"2026-02-22T04:31:17Z"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.341086985Z","logger":"setup","caller":"features/kube_features.go:406","msg":"Loaded feature gates","featureGates":{"AdmissionFairSharing":true,"DynamicResourceAllocation":false,"ElasticJobsViaWorkloadSlices":false,"FailureRecoveryPolicy":false,"FlavorFungibility":true,"HierarchicalCohorts":true,"LendingLimit":true,"LocalQueueDefaulting":true,"LocalQueueMetrics":false,"ManagedJobsNamespaceSelectorAlwaysRespected":true,"MultiKueue":true,"MultiKueueAdaptersForCustomJobs":true,"MultiKueueAllowInsecureKubeconfigs":false,"MultiKueueBatchJobWithManagedBy":true,"MultiKueueClusterProfile":false,"MultiKueueRedoAdmissionOnEvictionInWorker":true,"MultiKueueWaitForWorkloadAdmitted":true,"ObjectRetentionPolicies":true,"PartialAdmission":true,"PrioritySortingWithinCohort":true,"PropagateBatchJobLabelsToWorkload":true,"ReclaimablePods":true,"RemoveFinalizersWithStrictPatch":true,"SanitizePodSets":true,"SkipFinalizersForPodsSuspendedByParent":true,"TASBalancedPlacement":false,"TASFailedNodeReplacement":true,"TASFailedNodeReplacementFailFast":true,"TASProfileLeastFreeCapacity":false,"TASProfileMixed":true,"TASReplaceNodeOnPodTermination":true,"TLSOptions":true,"TopologyAwareScheduling":true,"VisibilityOnDemand":true,"WorkloadRequestUseMergePatch":false}} {"level":"info","ts":"2026-02-25T15:53:44.341168181Z","logger":"setup","caller":"kueue/main.go:195","msg":"Initializing metrics certificate watcher using provided certificates","metrics-cert-path":"/etc/kueue/metrics/certs"} {"level":"info","ts":"2026-02-25T15:53:44.341395682Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:214","msg":"Updated current TLS certificate","cert":"/etc/kueue/metrics/certs/tls.crt","key":"/etc/kueue/metrics/certs/tls.key"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.341595861Z","logger":"setup","caller":"kueue/main.go:228","msg":"K8S Client","qps":50,"burst":100} {"level":"info","ts":"2026-02-25T15:53:44.356912395Z","logger":"setup","caller":"kueue/main.go:549","msg":"RoleTracker: leader election enabled"} {"level":"error","ts":"2026-02-25T15:53:44.358927783Z","logger":"setup","caller":"kueue/main.go:364","msg":"Skipping admission check controller setup: Provisioning Requests not supported (Possible cause: missing or unsupported cluster-autoscaler)","error":"no matches for kind \"ProvisioningRequest\" in version \"autoscaling.x-k8s.io/v1\"","stacktrace":"main.setupIndexes\n\t/workspace/upstream/kueue/src/cmd/kueue/main.go:364\nmain.main\n\t/workspace/upstream/kueue/src/cmd/kueue/main.go:298\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:285"} {"level":"info","ts":"2026-02-25T15:53:44.36036001Z","logger":"setup","caller":"kueue/main.go:506","msg":"Probe endpoints are configured on healthz and readyz"} {"level":"info","ts":"2026-02-25T15:53:44.363733741Z","logger":"setup","caller":"kueue/main.go:399","msg":"Skipping provisioning controller setup: Provisioning Requests not supported (Possible cause: missing or unsupported cluster-autoscaler)"} {"level":"info","ts":"2026-02-25T15:53:44.363993494Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"workload.codeflare.dev/v1beta2, Kind=AppWrapper","path":"/mutate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.364200276Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.364298679Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"workload.codeflare.dev/v1beta2, Kind=AppWrapper","path":"/validate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.364489979Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.364596082Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"apps/v1, Kind=Deployment","path":"/mutate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.364648156Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.364674283Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"apps/v1, Kind=Deployment","path":"/validate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.364711319Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.364825818Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"batch/v1, Kind=Job","path":"/mutate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.364868354Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.364892466Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"batch/v1, Kind=Job","path":"/validate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.364927808Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.36505134Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"jobset.x-k8s.io/v1alpha2, Kind=JobSet","path":"/mutate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.365097013Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.36512396Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"jobset.x-k8s.io/v1alpha2, Kind=JobSet","path":"/validate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.365163907Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.365236227Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=JAXJob","path":"/mutate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.365279072Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.365304314Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=JAXJob","path":"/validate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.365342609Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.365384023Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=PaddleJob","path":"/mutate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.365420423Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.365443718Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=PaddleJob","path":"/validate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.365478474Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.365522616Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=PyTorchJob","path":"/mutate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.365560908Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.365586148Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=PyTorchJob","path":"/validate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.365621753Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.365667208Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=TFJob","path":"/mutate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.365709175Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.365732952Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=TFJob","path":"/validate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.365768288Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.365807752Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=XGBoostJob","path":"/mutate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.365843187Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.365867943Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=XGBoostJob","path":"/validate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.36590428Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.365944172Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v2beta1, Kind=MPIJob","path":"/mutate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.365981326Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.36600501Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v2beta1, Kind=MPIJob","path":"/validate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.366058861Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.366129671Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"/v1, Kind=Pod","path":"/mutate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.366168454Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.366192602Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"/v1, Kind=Pod","path":"/validate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.366227307Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.366335588Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"leaderworkerset.x-k8s.io/v1, Kind=LeaderWorkerSet","path":"/mutate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.366376357Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.366401096Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"leaderworkerset.x-k8s.io/v1, Kind=LeaderWorkerSet","path":"/validate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.36643766Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.366508072Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"ray.io/v1, Kind=RayCluster","path":"/mutate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.366544727Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.366569756Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"ray.io/v1, Kind=RayCluster","path":"/validate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.366605781Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.366650474Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"ray.io/v1, Kind=RayJob","path":"/mutate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.36668701Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.366708993Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"ray.io/v1, Kind=RayJob","path":"/validate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.366746876Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.366874134Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"apps/v1, Kind=StatefulSet","path":"/mutate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.366947915Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.366974333Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"apps/v1, Kind=StatefulSet","path":"/validate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.367012744Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.367111915Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"trainer.kubeflow.org/v1alpha1, Kind=TrainJob","path":"/mutate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.367151586Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.367175979Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"trainer.kubeflow.org/v1alpha1, Kind=TrainJob","path":"/validate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.367211817Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.367251848Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=Workload","path":"/mutate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.367288031Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.36731295Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=Workload","path":"/validate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.367349904Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.367412599Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/convert"} {"level":"info","ts":"2026-02-25T15:53:44.367420421Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=Workload"} {"level":"info","ts":"2026-02-25T15:53:44.367442335Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ResourceFlavor","path":"/mutate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.367478578Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.367502036Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ResourceFlavor","path":"/validate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.36753712Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.367585005Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ClusterQueue","path":"/mutate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.367621525Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.367645385Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ClusterQueue","path":"/validate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.367681207Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.367711727Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.367737372Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=Cohort","path":"/validate-kueue-x-k8s-io-v1beta2-cohort"} {"level":"info","ts":"2026-02-25T15:53:44.367772249Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-cohort"} {"level":"info","ts":"2026-02-25T15:53:44.367801074Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=Cohort"} {"level":"info","ts":"2026-02-25T15:53:44.367830265Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=LocalQueue"} {"level":"info","ts":"2026-02-25T15:53:44.36785223Z","logger":"setup","caller":"kueue/main.go:349","msg":"Starting manager"} {"level":"info","ts":"2026-02-25T15:53:44.367907557Z","logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} {"level":"info","ts":"2026-02-25T15:53:44.368055438Z","logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":8443","secure":true} {"level":"info","ts":"2026-02-25T15:53:44.368056519Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:191","msg":"Starting webhook server"} {"level":"info","ts":"2026-02-25T15:53:44.368433564Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:214","msg":"Updated current TLS certificate","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key"} {"level":"info","ts":"2026-02-25T15:53:44.368550355Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:242","msg":"Serving webhook server","host":"","port":9443} {"level":"info","ts":"2026-02-25T15:53:44.368039059Z","caller":"manager/server.go:83","msg":"starting server","name":"health probe","addr":"[::]:8081"} {"level":"info","ts":"2026-02-25T15:53:44.368687314Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:136","msg":"Starting certificate poll+watcher","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key","interval":10} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.372585545Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.MultiKueueConfig","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.373057275Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.AdmissionCheck","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.373358959Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Job","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.373366805Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.MultiKueueCluster","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.374965877Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.LimitRange","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.37519478Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.ResourceFlavor","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.375658616Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.LocalQueue","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.376417194Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Node","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.377615608Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.Workload","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.402616147Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"info","ts":"2026-02-25T15:53:44.469051647Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:136","msg":"Starting certificate poll+watcher","cert":"/etc/kueue/metrics/certs/tls.crt","key":"/etc/kueue/metrics/certs/tls.key","interval":10} {"level":"info","ts":"2026-02-25T15:53:44.469189233Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0005323f0"} {"level":"info","ts":"2026-02-25T15:53:44.469314418Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.AdmissionCheck"} {"level":"info","ts":"2026-02-25T15:53:44.469208079Z","logger":"localqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0005324d0"} {"level":"info","ts":"2026-02-25T15:53:44.469385041Z","logger":"localqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.469405585Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.469356113Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc000532700"} {"level":"info","ts":"2026-02-25T15:53:44.469359686Z","logger":"workloadpriorityclass-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.WorkloadPriorityClass"} {"level":"info","ts":"2026-02-25T15:53:44.469386427Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Namespace"} {"level":"info","ts":"2026-02-25T15:53:44.469343721Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ResourceFlavor"} {"level":"info","ts":"2026-02-25T15:53:44.469528143Z","logger":"localqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.LocalQueue"} {"level":"info","ts":"2026-02-25T15:53:44.469312616Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc000532310"} {"level":"info","ts":"2026-02-25T15:53:44.469614645Z","logger":"cohort-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0005325b0"} {"level":"info","ts":"2026-02-25T15:53:44.469668395Z","logger":"cohort-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Cohort"} {"level":"info","ts":"2026-02-25T15:53:44.469515708Z","logger":"tas-topology-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Topology"} {"level":"info","ts":"2026-02-25T15:53:44.469707353Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0005327e0"} {"level":"info","ts":"2026-02-25T15:53:44.469744939Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.LimitRange"} {"level":"info","ts":"2026-02-25T15:53:44.469791414Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.RuntimeClass"} {"level":"info","ts":"2026-02-25T15:53:44.469841502Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.469884951Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.LocalQueue"} {"level":"info","ts":"2026-02-25T15:53:44.469591047Z","logger":"tas-topology-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ResourceFlavor"} {"level":"info","ts":"2026-02-25T15:53:44.469945773Z","logger":"tas-topology-ungater","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-02-25T15:53:44.469919433Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:53:44.469997455Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Node"} {"level":"info","ts":"2026-02-25T15:53:44.469977487Z","logger":"tas-non-tas-usage-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"standalone","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-02-25T15:53:44.470091876Z","logger":"tas-node-failure-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Node"} {"level":"info","ts":"2026-02-25T15:53:44.469892497Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ResourceFlavor"} {"level":"info","ts":"2026-02-25T15:53:44.46980967Z","logger":"tas-topology-ungater","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} I0225 15:53:44.470329 1 leaderelection.go:258] "Attempting to acquire leader lease..." lock="openshift-kueue-operator/c1f6bfd2.kueue.x-k8s.io" {"level":"Level(-2)","ts":"2026-02-25T15:53:44.470399883Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:141","msg":"ResourceFlavor create event","replica-role":"follower","resourceFlavor":{"name":"default"}} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.470488034Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:215","msg":"LocalQueue create event","replica-role":"follower","localQueue":{"name":"test-queue","namespace":"kueue-managed-test"}} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.470636107Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:141","msg":"ResourceFlavor create event","replica-role":"follower","resourceFlavor":{"name":"default-flavor"}} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.470689046Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:141","msg":"ResourceFlavor create event","replica-role":"follower","resourceFlavor":{"name":"default-flavor2"}} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.475644953Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.ClusterQueue","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.475967039Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.RuntimeClass","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.476247521Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.WorkloadPriorityClass","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.4764467Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.Cohort","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.477298878Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.Topology","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.480087522Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Namespace","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} I0225 15:53:44.568784 1 plugins.go:157] Loaded 2 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,MutatingAdmissionPolicy. {"level":"info","ts":"2026-02-25T15:53:44.56993691Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.570001145Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.570206671Z","logger":"cohort-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.570244158Z","logger":"cohort-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.570328551Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:313","msg":"ClusterQueue create event","replica-role":"follower","clusterQueue":{"name":"cluster-queue"}} {"level":"info","ts":"2026-02-25T15:53:44.570435838Z","logger":"localqueue-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.570454802Z","logger":"localqueue-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.570494443Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:313","msg":"ClusterQueue create event","replica-role":"follower","clusterQueue":{"name":"cluster-queue2"}} {"level":"info","ts":"2026-02-25T15:53:44.570559442Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.570575526Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.57064351Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:313","msg":"ClusterQueue create event","replica-role":"follower","clusterQueue":{"name":"test-clusterqueue"}} {"level":"info","ts":"2026-02-25T15:53:44.570676238Z","logger":"tas-non-tas-usage-controller","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"standalone"} {"level":"info","ts":"2026-02-25T15:53:44.570690809Z","logger":"tas-non-tas-usage-controller","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"standalone","worker count":5} {"level":"info","ts":"2026-02-25T15:53:44.570821375Z","logger":"workloadpriorityclass-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.570851465Z","logger":"workloadpriorityclass-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.571494162Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.571551869Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.573127464Z","logger":"tas-node-failure-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.573192531Z","logger":"tas-node-failure-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.574116015Z","logger":"workload-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.574172258Z","logger":"workload-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":5} {"level":"info","ts":"2026-02-25T15:53:44.574296766Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.574326301Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} I0225 15:53:44.575096 1 handler.go:304] Adding GroupVersion visibility.kueue.x-k8s.io v1beta2 to ResourceManager I0225 15:53:44.575551 1 handler.go:304] Adding GroupVersion visibility.kueue.x-k8s.io v1beta1 to ResourceManager {"level":"info","ts":"2026-02-25T15:53:44.584440915Z","logger":"tas-topology-ungater","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.584552636Z","logger":"tas-topology-ungater","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":5} {"level":"info","ts":"2026-02-25T15:53:44.584778753Z","logger":"tas-topology-controller","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.58483679Z","logger":"tas-topology-controller","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} I0225 15:53:44.588925 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0225 15:53:44.588950 1 shared_informer.go:370] "Waiting for caches to sync" I0225 15:53:44.588968 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0225 15:53:44.588975 1 shared_informer.go:370] "Waiting for caches to sync" I0225 15:53:44.588986 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0225 15:53:44.588992 1 shared_informer.go:370] "Waiting for caches to sync" I0225 15:53:44.589364 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/visibility/tls.crt::/visibility/tls.key" I0225 15:53:44.589555 1 secure_serving.go:211] Serving securely on [::]:8082 I0225 15:53:44.589583 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" I0225 15:53:44.589632 1 apf_controller.go:377] Starting API Priority and Fairness config controller I0225 15:53:44.689891 1 apf_controller.go:382] Running API Priority and Fairness config worker I0225 15:53:44.689968 1 apf_controller.go:385] Running API Priority and Fairness periodic rebalancing process I0225 15:53:44.689922 1 shared_informer.go:377] "Caches are synced" I0225 15:53:44.689935 1 shared_informer.go:377] "Caches are synced" I0225 15:53:44.689938 1 shared_informer.go:377] "Caches are synced" {"level":"Level(-2)","ts":"2026-02-25T15:54:23.590111739Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"follower","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.590207371Z","logger":"workload-reconciler","caller":"core/workload_controller.go:831","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.608000117Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.608087502Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.19812975Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.198195966Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.212175285Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"follower","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.30798512Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"follower","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.308068183Z","logger":"workload-reconciler","caller":"core/workload_controller.go:831","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.317818492Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.317858668Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.534874458Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.534950465Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.548692336Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"follower","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.188895592Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"follower","workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"queue":"test-queue","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.197532922Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted","prevStatus":"pending","clusterQueue":"test-clusterqueue"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.733111904Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"follower","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.733173254Z","logger":"workload-reconciler","caller":"core/workload_controller.go:831","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.745537428Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.745598368Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:50.198748709Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:254","msg":"Queue update event","replica-role":"follower","localQueue":{"name":"test-queue","namespace":"kueue-managed-test"}} {"level":"Level(-2)","ts":"2026-02-25T15:54:50.199456463Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:346","msg":"ClusterQueue update event","replica-role":"follower","clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.084404451Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.08447968Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.094073562Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"follower","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.42723648Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted","clusterQueue":"test-clusterqueue"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.455541678Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"follower","workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.438076375Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:254","msg":"Queue update event","replica-role":"follower","localQueue":{"name":"test-queue","namespace":"kueue-managed-test"}} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.439229042Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:346","msg":"ClusterQueue update event","replica-role":"follower","clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.834404238Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"follower","workload":{"name":"job-test-job-8xmxz-b852d","namespace":"kueue-managed-test"},"queue":"test-queue","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.846591004Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-8xmxz-b852d","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted","prevStatus":"pending","clusterQueue":"test-clusterqueue"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.367637023Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"follower","workload":{"name":"job-test-job-c74bb-45a97","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.367693782Z","logger":"workload-reconciler","caller":"core/workload_controller.go:831","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-c74bb-45a97","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.376200517Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-c74bb-45a97","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.376256578Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-c74bb-45a97","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.844884492Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:254","msg":"Queue update event","replica-role":"follower","localQueue":{"name":"test-queue","namespace":"kueue-managed-test"}} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.845995225Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:346","msg":"ClusterQueue update event","replica-role":"follower","clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T16:00:52.70289662Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-c74bb-45a97","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T16:00:52.702967497Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-c74bb-45a97","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T16:00:52.714717848Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"follower","workload":{"name":"job-test-job-c74bb-45a97","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T16:00:53.043155422Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-8xmxz-b852d","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted","clusterQueue":"test-clusterqueue"} {"level":"Level(-2)","ts":"2026-02-25T16:00:53.070412922Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"follower","workload":{"name":"job-test-job-8xmxz-b852d","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted"} {"level":"Level(-2)","ts":"2026-02-25T16:00:54.052747159Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:254","msg":"Queue update event","replica-role":"follower","localQueue":{"name":"test-queue","namespace":"kueue-managed-test"}} {"level":"Level(-2)","ts":"2026-02-25T16:00:54.054153261Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:346","msg":"ClusterQueue update event","replica-role":"follower","clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T16:00:54.71234637Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"follower","workload":{"name":"job-test-job-869lt-9a3b7","namespace":"kueue-managed-test"},"queue":"test-queue","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T16:00:54.723382851Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-869lt-9a3b7","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted","prevStatus":"pending","clusterQueue":"test-clusterqueue"} {"level":"Level(-2)","ts":"2026-02-25T16:00:55.213121219Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"follower","workload":{"name":"job-test-job-2bkxc-20dff","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T16:00:55.213193967Z","logger":"workload-reconciler","caller":"core/workload_controller.go:831","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-2bkxc-20dff","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T16:00:55.222166808Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-2bkxc-20dff","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T16:00:55.22221245Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-2bkxc-20dff","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T16:00:55.723715984Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:254","msg":"Queue update event","replica-role":"follower","localQueue":{"name":"test-queue","namespace":"kueue-managed-test"}} {"level":"Level(-2)","ts":"2026-02-25T16:00:55.724856486Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:346","msg":"ClusterQueue update event","replica-role":"follower","clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T16:03:55.554819687Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-2bkxc-20dff","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T16:03:55.554892099Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"follower","workload":{"name":"job-test-job-2bkxc-20dff","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T16:03:55.566917762Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"follower","workload":{"name":"job-test-job-2bkxc-20dff","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T16:03:55.920208403Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"follower","workload":{"name":"job-test-job-869lt-9a3b7","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted","clusterQueue":"test-clusterqueue"} {"level":"Level(-2)","ts":"2026-02-25T16:03:55.9577366Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"follower","workload":{"name":"job-test-job-869lt-9a3b7","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted"} === Logs from pod kueue-controller-manager-76d4586765-w7lmz === --- Container: manager --- {"level":"error","ts":"2026-02-25T15:53:44.303506867Z","logger":"setup","caller":"kueue/main.go:364","msg":"Skipping admission check controller setup: Provisioning Requests not supported (Possible cause: missing or unsupported cluster-autoscaler)","error":"no matches for kind \"ProvisioningRequest\" in version \"autoscaling.x-k8s.io/v1\"","stacktrace":"main.setupIndexes\n\t/workspace/upstream/kueue/src/cmd/kueue/main.go:364\nmain.main\n\t/workspace/upstream/kueue/src/cmd/kueue/main.go:298\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:285"} {"level":"info","ts":"2026-02-25T15:53:44.305857965Z","logger":"setup","caller":"kueue/main.go:506","msg":"Probe endpoints are configured on healthz and readyz"} {"level":"info","ts":"2026-02-25T15:53:44.307849898Z","logger":"setup","caller":"kueue/main.go:399","msg":"Skipping provisioning controller setup: Provisioning Requests not supported (Possible cause: missing or unsupported cluster-autoscaler)"} {"level":"info","ts":"2026-02-25T15:53:44.308011898Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"workload.codeflare.dev/v1beta2, Kind=AppWrapper","path":"/mutate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.308091229Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.308135679Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"workload.codeflare.dev/v1beta2, Kind=AppWrapper","path":"/validate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.308186638Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-workload-codeflare-dev-v1beta2-appwrapper"} {"level":"info","ts":"2026-02-25T15:53:44.308277059Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"apps/v1, Kind=Deployment","path":"/mutate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.308325654Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.308385596Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"apps/v1, Kind=Deployment","path":"/validate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.308431886Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-apps-v1-deployment"} {"level":"info","ts":"2026-02-25T15:53:44.3085592Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"batch/v1, Kind=Job","path":"/mutate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.308602344Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.308621786Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"batch/v1, Kind=Job","path":"/validate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.308658489Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-batch-v1-job"} {"level":"info","ts":"2026-02-25T15:53:44.308757891Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"jobset.x-k8s.io/v1alpha2, Kind=JobSet","path":"/mutate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.308797523Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.308815349Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"jobset.x-k8s.io/v1alpha2, Kind=JobSet","path":"/validate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.308863601Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-jobset-x-k8s-io-v1alpha2-jobset"} {"level":"info","ts":"2026-02-25T15:53:44.308934062Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=JAXJob","path":"/mutate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.308968472Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.308992373Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=JAXJob","path":"/validate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.309025871Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-jaxjob"} {"level":"info","ts":"2026-02-25T15:53:44.309077677Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=PaddleJob","path":"/mutate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.309129509Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.309163402Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=PaddleJob","path":"/validate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.309200142Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-paddlejob"} {"level":"info","ts":"2026-02-25T15:53:44.309264058Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=PyTorchJob","path":"/mutate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.309298958Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.309324593Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=PyTorchJob","path":"/validate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.30936498Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-pytorchjob"} {"level":"info","ts":"2026-02-25T15:53:44.309430139Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=TFJob","path":"/mutate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.309520135Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.309549091Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=TFJob","path":"/validate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.309603256Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-tfjob"} {"level":"info","ts":"2026-02-25T15:53:44.309682302Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v1, Kind=XGBoostJob","path":"/mutate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.309751505Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.309788546Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v1, Kind=XGBoostJob","path":"/validate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.309834512Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v1-xgboostjob"} {"level":"info","ts":"2026-02-25T15:53:44.30988365Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kubeflow.org/v2beta1, Kind=MPIJob","path":"/mutate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.309924823Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.309954839Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kubeflow.org/v2beta1, Kind=MPIJob","path":"/validate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.309995907Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kubeflow-org-v2beta1-mpijob"} {"level":"info","ts":"2026-02-25T15:53:44.310079654Z","caller":"webhook/builder.go:178","msg":"Registering a mutating webhook","GVK":"/v1, Kind=Pod","path":"/mutate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.310139549Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.310164284Z","caller":"webhook/builder.go:210","msg":"Registering a validating webhook","GVK":"/v1, Kind=Pod","path":"/validate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.310214291Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate--v1-pod"} {"level":"info","ts":"2026-02-25T15:53:44.310381361Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"leaderworkerset.x-k8s.io/v1, Kind=LeaderWorkerSet","path":"/mutate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.31051083Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.31056551Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"leaderworkerset.x-k8s.io/v1, Kind=LeaderWorkerSet","path":"/validate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.310626805Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-leaderworkerset-x-k8s-io-v1-leaderworkerset"} {"level":"info","ts":"2026-02-25T15:53:44.310751222Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"ray.io/v1, Kind=RayCluster","path":"/mutate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.310817583Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.310856509Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"ray.io/v1, Kind=RayCluster","path":"/validate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.310918893Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-ray-io-v1-raycluster"} {"level":"info","ts":"2026-02-25T15:53:44.310990165Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"ray.io/v1, Kind=RayJob","path":"/mutate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.311051318Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.311088179Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"ray.io/v1, Kind=RayJob","path":"/validate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.311499756Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-ray-io-v1-rayjob"} {"level":"info","ts":"2026-02-25T15:53:44.312242487Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"apps/v1, Kind=StatefulSet","path":"/mutate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.31249166Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.31253884Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"apps/v1, Kind=StatefulSet","path":"/validate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.312589856Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-apps-v1-statefulset"} {"level":"info","ts":"2026-02-25T15:53:44.312885251Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"trainer.kubeflow.org/v1alpha1, Kind=TrainJob","path":"/mutate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.312941616Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.312986347Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"trainer.kubeflow.org/v1alpha1, Kind=TrainJob","path":"/validate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.313049129Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-trainer-kubeflow-org-v1alpha1-trainjob"} {"level":"info","ts":"2026-02-25T15:53:44.313214607Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=Workload","path":"/mutate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.313306305Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.313376373Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=Workload","path":"/validate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.313456132Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-workload"} {"level":"info","ts":"2026-02-25T15:53:44.313629481Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/convert"} {"level":"info","ts":"2026-02-25T15:53:44.313659096Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=Workload"} {"level":"info","ts":"2026-02-25T15:53:44.313710299Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ResourceFlavor","path":"/mutate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.313787755Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.313837229Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ResourceFlavor","path":"/validate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.313908388Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-resourceflavor"} {"level":"info","ts":"2026-02-25T15:53:44.314048405Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:220","msg":"Registering a mutating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ClusterQueue","path":"/mutate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.314725043Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/mutate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.314801746Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=ClusterQueue","path":"/validate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.314860746Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-clusterqueue"} {"level":"info","ts":"2026-02-25T15:53:44.31494798Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.314990652Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:258","msg":"Registering a validating webhook","GVK":"kueue.x-k8s.io/v1beta2, Kind=Cohort","path":"/validate-kueue-x-k8s-io-v1beta2-cohort"} {"level":"info","ts":"2026-02-25T15:53:44.315034476Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","path":"/validate-kueue-x-k8s-io-v1beta2-cohort"} {"level":"info","ts":"2026-02-25T15:53:44.315147712Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=Cohort"} {"level":"info","ts":"2026-02-25T15:53:44.315236212Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:289","msg":"Conversion webhook enabled","GVK":"kueue.x-k8s.io/v1beta2, Kind=LocalQueue"} {"level":"info","ts":"2026-02-25T15:53:44.315258026Z","logger":"setup","caller":"kueue/main.go:349","msg":"Starting manager"} {"level":"info","ts":"2026-02-25T15:53:44.31533328Z","logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} {"level":"info","ts":"2026-02-25T15:53:44.315471423Z","logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":8443","secure":true} {"level":"info","ts":"2026-02-25T15:53:44.316260303Z","caller":"manager/server.go:83","msg":"starting server","name":"health probe","addr":"[::]:8081"} {"level":"info","ts":"2026-02-25T15:53:44.31633774Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:191","msg":"Starting webhook server"} {"level":"info","ts":"2026-02-25T15:53:44.31671217Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:214","msg":"Updated current TLS certificate","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key"} {"level":"info","ts":"2026-02-25T15:53:44.316798854Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:242","msg":"Serving webhook server","host":"","port":9443} {"level":"info","ts":"2026-02-25T15:53:44.317317806Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:136","msg":"Starting certificate poll+watcher","cert":"/tmp/k8s-webhook-server/serving-certs/tls.crt","key":"/tmp/k8s-webhook-server/serving-certs/tls.key","interval":10} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.322565594Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.LimitRange","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.324181786Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Job","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.324418016Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.AdmissionCheck","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.334771556Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.LocalQueue","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.335081697Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.MultiKueueCluster","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.33950372Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.MultiKueueConfig","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.340560729Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Node","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.341150449Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.ResourceFlavor","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.354242079Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.Workload","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.392841688Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"info","ts":"2026-02-25T15:53:44.417945012Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:136","msg":"Starting certificate poll+watcher","cert":"/etc/kueue/metrics/certs/tls.crt","key":"/etc/kueue/metrics/certs/tls.key","interval":10} {"level":"info","ts":"2026-02-25T15:53:44.418063438Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002de850"} {"level":"info","ts":"2026-02-25T15:53:44.418190033Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002de700"} {"level":"info","ts":"2026-02-25T15:53:44.418248952Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ResourceFlavor"} {"level":"info","ts":"2026-02-25T15:53:44.418306847Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.418323341Z","logger":"resourceflavor-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.418336783Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.AdmissionCheck"} {"level":"info","ts":"2026-02-25T15:53:44.418364917Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.418379128Z","logger":"admissioncheck-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.418412681Z","logger":"localqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002de930"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.418457955Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:141","msg":"ResourceFlavor create event","replica-role":"follower","resourceFlavor":{"name":"default"}} {"level":"info","ts":"2026-02-25T15:53:44.418543799Z","logger":"workloadpriorityclass-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.WorkloadPriorityClass"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.418599616Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:141","msg":"ResourceFlavor create event","replica-role":"follower","resourceFlavor":{"name":"default-flavor"}} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.418627025Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:141","msg":"ResourceFlavor create event","replica-role":"follower","resourceFlavor":{"name":"default-flavor2"}} {"level":"info","ts":"2026-02-25T15:53:44.41866598Z","logger":"localqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.LocalQueue"} {"level":"info","ts":"2026-02-25T15:53:44.418793947Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002debd0"} {"level":"info","ts":"2026-02-25T15:53:44.418835076Z","logger":"localqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.418934927Z","logger":"cohort-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002dea10"} {"level":"info","ts":"2026-02-25T15:53:44.41902137Z","logger":"tas-topology-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Topology"} {"level":"info","ts":"2026-02-25T15:53:44.419069043Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Namespace"} {"level":"info","ts":"2026-02-25T15:53:44.419259296Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.419330123Z","logger":"cohort-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Cohort"} {"level":"info","ts":"2026-02-25T15:53:44.419517674Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002decb0"} {"level":"info","ts":"2026-02-25T15:53:44.419615479Z","logger":"tas-topology-ungater","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:53:44.419680116Z","logger":"tas-topology-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ResourceFlavor"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.419850793Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:215","msg":"LocalQueue create event","replica-role":"follower","localQueue":{"name":"test-queue","namespace":"kueue-managed-test"}} {"level":"info","ts":"2026-02-25T15:53:44.420079153Z","logger":"tas-topology-ungater","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-02-25T15:53:44.420087499Z","logger":"tas-non-tas-usage-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"standalone","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-02-25T15:53:44.420203472Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.LimitRange"} {"level":"info","ts":"2026-02-25T15:53:44.420264114Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.RuntimeClass"} {"level":"info","ts":"2026-02-25T15:53:44.420312116Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ClusterQueue"} {"level":"info","ts":"2026-02-25T15:53:44.42035359Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.LocalQueue"} {"level":"info","ts":"2026-02-25T15:53:44.420394273Z","logger":"workload-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:53:44.420513225Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.ResourceFlavor"} {"level":"info","ts":"2026-02-25T15:53:44.42061171Z","logger":"tas-node-failure-reconciler","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Node"} {"level":"info","ts":"2026-02-25T15:53:44.420691072Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Node"} I0225 15:53:44.420838 1 leaderelection.go:258] "Attempting to acquire leader lease..." lock="openshift-kueue-operator/c1f6bfd2.kueue.x-k8s.io" {"level":"Level(-2)","ts":"2026-02-25T15:53:44.424441062Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.Topology","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.425208312Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.Cohort","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.425494622Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.WorkloadPriorityClass","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.425642886Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.RuntimeClass","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.425729818Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1beta2.ClusterQueue","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.429091863Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Namespace","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"info","ts":"2026-02-25T15:53:44.519303959Z","logger":"workloadpriorityclass-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.519420284Z","logger":"workloadpriorityclass-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.519413528Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:313","msg":"ClusterQueue create event","replica-role":"follower","clusterQueue":{"name":"cluster-queue"}} {"level":"info","ts":"2026-02-25T15:53:44.519598564Z","logger":"localqueue-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.519657428Z","logger":"localqueue-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.519611865Z","logger":"cohort-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.519742302Z","logger":"cohort-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.519623116Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:313","msg":"ClusterQueue create event","replica-role":"follower","clusterQueue":{"name":"cluster-queue2"}} {"level":"Level(-2)","ts":"2026-02-25T15:53:44.519898159Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:313","msg":"ClusterQueue create event","replica-role":"follower","clusterQueue":{"name":"test-clusterqueue"}} {"level":"info","ts":"2026-02-25T15:53:44.520142016Z","logger":"tas-topology-controller","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.520162874Z","logger":"tas-topology-controller","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.520715763Z","logger":"tas-topology-ungater","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.520735396Z","logger":"tas-topology-ungater","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":5} {"level":"info","ts":"2026-02-25T15:53:44.52193053Z","logger":"workload-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.522003655Z","logger":"workload-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":5} {"level":"info","ts":"2026-02-25T15:53:44.522003161Z","logger":"tas-non-tas-usage-controller","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"standalone"} {"level":"info","ts":"2026-02-25T15:53:44.522044978Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.522092774Z","logger":"tas-resource-flavor-controller","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.521979407Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.522012377Z","logger":"tas-node-failure-reconciler","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:53:44.522128959Z","logger":"clusterqueue-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.522134259Z","logger":"tas-node-failure-reconciler","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":1} {"level":"info","ts":"2026-02-25T15:53:44.522052297Z","logger":"tas-non-tas-usage-controller","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"standalone","worker count":5} I0225 15:53:44.640704 1 plugins.go:157] Loaded 2 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,MutatingAdmissionPolicy. I0225 15:53:44.644255 1 handler.go:304] Adding GroupVersion visibility.kueue.x-k8s.io v1beta2 to ResourceManager I0225 15:53:44.644803 1 handler.go:304] Adding GroupVersion visibility.kueue.x-k8s.io v1beta1 to ResourceManager I0225 15:53:44.653678 1 secure_serving.go:211] Serving securely on [::]:8082 I0225 15:53:44.653813 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0225 15:53:44.653846 1 shared_informer.go:370] "Waiting for caches to sync" I0225 15:53:44.653881 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/visibility/tls.crt::/visibility/tls.key" I0225 15:53:44.653979 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" I0225 15:53:44.654449 1 apf_controller.go:377] Starting API Priority and Fairness config controller I0225 15:53:44.654499 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0225 15:53:44.654529 1 shared_informer.go:370] "Waiting for caches to sync" I0225 15:53:44.654578 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0225 15:53:44.654589 1 shared_informer.go:370] "Waiting for caches to sync" I0225 15:53:44.753987 1 shared_informer.go:377] "Caches are synced" I0225 15:53:44.755178 1 shared_informer.go:377] "Caches are synced" I0225 15:53:44.755183 1 shared_informer.go:377] "Caches are synced" I0225 15:53:44.755194 1 apf_controller.go:382] Running API Priority and Fairness config worker I0225 15:53:44.755303 1 apf_controller.go:385] Running API Priority and Fairness periodic rebalancing process I0225 15:54:23.478467 1 leaderelection.go:272] "Successfully acquired lease" lock="openshift-kueue-operator/c1f6bfd2.kueue.x-k8s.io" {"level":"debug","ts":"2026-02-25T15:54:23.47848655Z","logger":"events","caller":"recorder/recorder.go:104","msg":"kueue-controller-manager-76d4586765-w7lmz_e29d34fc-6f59-4925-af31-0047b63f4a24 became leader","type":"Normal","object":{"kind":"Lease","namespace":"openshift-kueue-operator","name":"c1f6bfd2.kueue.x-k8s.io","uid":"71c5dacc-f277-46a1-89d4-36b6039cf3f3","apiVersion":"coordination.k8s.io/v1","resourceVersion":"142117"},"reason":"LeaderElection"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.47863183Z","logger":"MultiKueueGC","caller":"multikueue/multikueuecluster.go:661","msg":"Starting Garbage Collector"} {"level":"info","ts":"2026-02-25T15:54:23.478798117Z","logger":"multikueue-admissioncheck","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.MultiKueueCluster"} {"level":"info","ts":"2026-02-25T15:54:23.47887902Z","logger":"multikueue-admissioncheck","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.AdmissionCheck"} {"level":"info","ts":"2026-02-25T15:54:23.478911067Z","logger":"multikueue-admissioncheck","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.MultiKueueConfig"} {"level":"info","ts":"2026-02-25T15:54:23.478876419Z","logger":"job","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:54:23.478937512Z","logger":"job","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Job"} {"level":"info","ts":"2026-02-25T15:54:23.47899078Z","logger":"job","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:54:23.479009867Z","logger":"leaderworkerset","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.LeaderWorkerSet"} {"level":"info","ts":"2026-02-25T15:54:23.479034679Z","logger":"job","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"follower"} {"level":"info","ts":"2026-02-25T15:54:23.479048968Z","logger":"job","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"follower","worker count":5} {"level":"info","ts":"2026-02-25T15:54:23.479065076Z","logger":"multikueue-workload","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:54:23.479098312Z","logger":"multikueue-workload","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.MultiKueueConfig"} {"level":"info","ts":"2026-02-25T15:54:23.478935795Z","logger":"jobset","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:54:23.479202659Z","logger":"v1_pod","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.Workload"} {"level":"info","ts":"2026-02-25T15:54:23.479218633Z","logger":"statefulset","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-02-25T15:54:23.479269984Z","logger":"multikueue-cluster","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002df0a0"} {"level":"info","ts":"2026-02-25T15:54:23.479312024Z","logger":"jobset","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1alpha2.JobSet"} {"level":"info","ts":"2026-02-25T15:54:23.478990311Z","logger":"multikueue-workload","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002df110"} {"level":"info","ts":"2026-02-25T15:54:23.479035148Z","logger":"leaderworkerset_pod","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-02-25T15:54:23.479407314Z","logger":"setup","caller":"roletracker/tracker.go:59","msg":"RoleTracker: transitioned to leader"} {"level":"info","ts":"2026-02-25T15:54:23.479279008Z","logger":"v1_pod","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Pod"} {"level":"info","ts":"2026-02-25T15:54:23.479430208Z","logger":"statefulset","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.StatefulSet"} {"level":"info","ts":"2026-02-25T15:54:23.479441315Z","logger":"multikueue-cluster","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1beta2.MultiKueueCluster"} {"level":"info","ts":"2026-02-25T15:54:23.479463892Z","logger":"multikueue-cluster","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"kind source: *v1.Secret"} {"level":"info","ts":"2026-02-25T15:54:23.479584497Z","logger":"multikueue-cluster","caller":"controller/controller.go:353","msg":"Starting EventSource","replica-role":"follower","source":"channel source: 0xc0002df030"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.479790735Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"follower","namespace":"openshift-marketplace","name":"46657514f1e649ebaa51da741ac70392e61d69e281cd3964becce7ed5ab42c6","reconcileID":"4b63c326-7e10-4ff5-b4f6-8b586cbbe217","job":"openshift-marketplace/46657514f1e649ebaa51da741ac70392e61d69e281cd3964becce7ed5ab42c6","gvk":"batch/v1, Kind=Job","namespace":"openshift-marketplace"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.479805015Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-marketplace","name":"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76ac9","reconcileID":"a10ce01f-3c78-4c59-be98-b1c835b22d11","job":"openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76ac9","gvk":"batch/v1, Kind=Job","namespace":"openshift-marketplace"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.479848208Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-marketplace","name":"925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e56eb0c","reconcileID":"9263f448-2761-4f65-b891-1a83beb3b10f","job":"openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e56eb0c","gvk":"batch/v1, Kind=Job","namespace":"openshift-marketplace"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.480032346Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-marketplace","name":"c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78e8d070","reconcileID":"d2a96b26-2e64-441e-94de-692fef5cec4d","job":"openshift-marketplace/c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78e8d070","gvk":"batch/v1, Kind=Job","namespace":"openshift-marketplace"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.480050993Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-mwprf","reconcileID":"d4748d7e-84a5-4d4e-ab24-bf13177d201c","job":"kueue-managed-test/test-job-mwprf","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.492986141Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.JobSet","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.494641998Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.LeaderWorkerSet","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.495230317Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Secret","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.495339101Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.PriorityClass","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.495920814Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.StatefulSet","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"} {"level":"info","ts":"2026-02-25T15:54:23.580006576Z","logger":"jobset","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.58004821Z","logger":"jobset","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":1} {"level":"info","ts":"2026-02-25T15:54:23.580006578Z","logger":"multikueue-admissioncheck","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.580069816Z","logger":"multikueue-admissioncheck","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":1} {"level":"info","ts":"2026-02-25T15:54:23.580006225Z","logger":"leaderworkerset","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.580080288Z","logger":"leaderworkerset","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":1} {"level":"info","ts":"2026-02-25T15:54:23.580037828Z","logger":"statefulset","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.580091225Z","logger":"statefulset","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":1} {"level":"info","ts":"2026-02-25T15:54:23.580130816Z","logger":"multikueue-cluster","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.580138534Z","logger":"multikueue-cluster","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":1} {"level":"info","ts":"2026-02-25T15:54:23.580162296Z","logger":"leaderworkerset_pod","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.580169185Z","logger":"leaderworkerset_pod","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":5} {"level":"info","ts":"2026-02-25T15:54:23.580207148Z","logger":"multikueue-workload","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.580217794Z","logger":"multikueue-workload","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":5} {"level":"info","ts":"2026-02-25T15:54:23.581448472Z","logger":"v1_pod","caller":"controller/controller.go:286","msg":"Starting Controller","replica-role":"leader"} {"level":"info","ts":"2026-02-25T15:54:23.581486301Z","logger":"v1_pod","caller":"controller/controller.go:289","msg":"Starting workers","replica-role":"leader","worker count":5} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.595909321Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"14d0250c-88b6-4e65-8ab0-b09c6532a655"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.596063088Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"14d0250c-88b6-4e65-8ab0-b09c6532a655","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.59727002Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"leader","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.59739504Z","logger":"workload-reconciler","caller":"core/workload_controller.go:831","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.597499396Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"63e8c91b-221e-43b4-a3b7-c067d831381b"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.597516723Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-mwprf","reconcileID":"c2b98f22-2720-41d0-a6da-ab2b3398c53a","job":"kueue-managed-test/test-job-mwprf","gvk":"batch/v1, Kind=Job"} {"level":"debug","ts":"2026-02-25T15:54:23.59753245Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Created Workload: kueue-managed-test/job-test-job-mwprf-48c0e","type":"Normal","object":{"kind":"Job","namespace":"kueue-managed-test","name":"test-job-mwprf","uid":"eae93dbb-3e05-49df-b9c2-902ef552dd23","apiVersion":"batch/v1","resourceVersion":"142003"},"reason":"CreatedWorkload"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.608007747Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"fbdee5b1-6dd2-4c5f-8f86-b1fcb5c100eb"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.608167464Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"fbdee5b1-6dd2-4c5f-8f86-b1fcb5c100eb","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.6079943Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.608220862Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-mwprf","reconcileID":"531b9311-4487-473b-8594-708f2b9f6b60","job":"kueue-managed-test/test-job-mwprf","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.608250847Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:23.608316107Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"c8228417-97d6-4201-bee4-42f747ed20a0"} {"level":"Level(-2)","ts":"2026-02-25T15:54:26.584906078Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"kueue-unmanaged-test","name":"test-job-zwrrm","reconcileID":"abde3b87-0f40-4f82-84e9-bd573936408d","job":"kueue-unmanaged-test/test-job-zwrrm","gvk":"batch/v1, Kind=Job","namespace":"kueue-unmanaged-test"} {"level":"Level(-2)","ts":"2026-02-25T15:54:26.609606098Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"kueue-unmanaged-test","name":"test-job-zwrrm","reconcileID":"0b006413-f499-46bf-bf78-153b541a9638","job":"kueue-unmanaged-test/test-job-zwrrm","gvk":"batch/v1, Kind=Job","namespace":"kueue-unmanaged-test"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.198310364Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"551aeca3-1383-4804-9c23-1954904a284a"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.198408741Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.198536738Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.198653572Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"956a7f34-83e7-4968-8fcd-b4ef5b1e4b04"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.198443892Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"551aeca3-1383-4804-9c23-1954904a284a","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.212995169Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"leader","workload":{"name":"job-test-job-mwprf-48c0e","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.213453876Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"48722856-20b6-47fd-8095-8e8615d6c8eb"} {"level":"Level(-2)","ts":"2026-02-25T15:54:28.213543812Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-mwprf-48c0e","reconcileID":"48722856-20b6-47fd-8095-8e8615d6c8eb","isDeleted":true} {"level":"Level(-2)","ts":"2026-02-25T15:54:30.835235027Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-dzl7v","reconcileID":"e752aa62-1aad-4717-bdfd-80dad67f456b","job":"openshift-kueue-operator/webhook-test-dzl7v","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:30.858796259Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-dzl7v","reconcileID":"15ac08f1-aaf6-4f72-b41b-c08fcc605878","job":"openshift-kueue-operator/webhook-test-dzl7v","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:34.393557978Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-dv8fw","reconcileID":"075ec704-995d-4370-8186-200f9c5542f1","job":"openshift-kueue-operator/webhook-test-dv8fw","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:34.413218051Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-dv8fw","reconcileID":"a303ebb0-0572-4258-9f37-d1a898c7f9e0","job":"openshift-kueue-operator/webhook-test-dv8fw","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:37.933881218Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-lrf58","reconcileID":"e4bbcdc9-5a76-4728-9eb7-b1b9959d445b","job":"openshift-kueue-operator/webhook-test-lrf58","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:37.955948082Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-lrf58","reconcileID":"acc8109f-e6b5-44d7-908e-031481504976","job":"openshift-kueue-operator/webhook-test-lrf58","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.297363222Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-ptq5g","reconcileID":"9bbf8981-4531-4e68-a567-e90da6cc7407","job":"kueue-managed-test/test-job-ptq5g","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.308004411Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"leader","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.308172898Z","logger":"workload-reconciler","caller":"core/workload_controller.go:831","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.308183112Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"9016c0e9-fbfa-47ff-b7f8-92657a9ede3f"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.308247305Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"9016c0e9-fbfa-47ff-b7f8-92657a9ede3f","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.308293487Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"072e3bcc-5d53-4d32-814a-17a5d1d0dbf5"} {"level":"debug","ts":"2026-02-25T15:54:38.308317696Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Created Workload: kueue-managed-test/job-test-job-ptq5g-d05b9","type":"Normal","object":{"kind":"Job","namespace":"kueue-managed-test","name":"test-job-ptq5g","uid":"5f6fcec3-8336-41b4-9a05-df83c6dbc4b4","apiVersion":"batch/v1","resourceVersion":"142270"},"reason":"CreatedWorkload"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.308350501Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-ptq5g","reconcileID":"c75832e9-ffba-4b76-b76b-d3320a212e9f","job":"kueue-managed-test/test-job-ptq5g","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.317758394Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.317828202Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.317874821Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"11048109-02d0-41d9-b41e-5dd4b02b6273"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.317777881Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"6bd89d4c-62cb-4d68-b4de-abdaa513a90f"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.317917803Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-ptq5g","reconcileID":"822f6940-afd7-45ed-9048-e8e8cded7f82","job":"kueue-managed-test/test-job-ptq5g","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.317940658Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"6bd89d4c-62cb-4d68-b4de-abdaa513a90f","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.822975545Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"kueue-unmanaged-test","name":"test-job-c7mms","reconcileID":"094cfa3b-6221-4875-9fe7-dc32f4260835","job":"kueue-unmanaged-test/test-job-c7mms","gvk":"batch/v1, Kind=Job","namespace":"kueue-unmanaged-test"} {"level":"Level(-2)","ts":"2026-02-25T15:54:38.843891064Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"kueue-unmanaged-test","name":"test-job-c7mms","reconcileID":"ce6e50c3-3bec-4ff9-9d18-16b83d9da8c1","job":"kueue-unmanaged-test/test-job-c7mms","gvk":"batch/v1, Kind=Job","namespace":"kueue-unmanaged-test"} {"level":"Level(-2)","ts":"2026-02-25T15:54:40.5800551Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-g9lg6","reconcileID":"2b85fe17-9b4a-46b0-87f1-f6b9f5e62648","job":"openshift-kueue-operator/webhook-test-g9lg6","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:40.598508257Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-g9lg6","reconcileID":"28c77f18-4284-4185-9586-7e5067bd0b15","job":"openshift-kueue-operator/webhook-test-g9lg6","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:41.66072042Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"kueue-unmanaged-test","name":"test-job-c7mms","reconcileID":"0901a6b3-b4cb-4dfd-879c-eca4b34ebd95","job":"kueue-unmanaged-test/test-job-c7mms","gvk":"batch/v1, Kind=Job","namespace":"kueue-unmanaged-test"} {"level":"Level(-2)","ts":"2026-02-25T15:54:44.121870782Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-tw6tr","reconcileID":"d9e4ee13-f09a-4de4-87a9-6a96f90670c4","job":"openshift-kueue-operator/webhook-test-tw6tr","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:44.142095921Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-tw6tr","reconcileID":"da121fbe-c306-4c8c-9061-f5882cffb292","job":"openshift-kueue-operator/webhook-test-tw6tr","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:47.636945527Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-2h4xr","reconcileID":"77e27f5b-a7c9-4a91-a417-57d9553a1829","job":"openshift-kueue-operator/webhook-test-2h4xr","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:47.656685073Z","logger":"job","caller":"jobframework/reconciler.go:343","msg":"Namespace not opted in for Kueue management","replica-role":"leader","namespace":"openshift-kueue-operator","name":"webhook-test-2h4xr","reconcileID":"e0faeb79-5f5d-40ac-9f06-8ec791e4376d","job":"openshift-kueue-operator/webhook-test-2h4xr","gvk":"batch/v1, Kind=Job","namespace":"openshift-kueue-operator"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.535152469Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"0890d2dc-380f-45db-a161-111c7327b96d"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.535166412Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.535303136Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.535252072Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"0890d2dc-380f-45db-a161-111c7327b96d","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.535381625Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"bdc53d6e-4500-406a-9a17-66877cddd1c1"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.548638338Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"leader","workload":{"name":"job-test-job-ptq5g-d05b9","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.548650975Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"3f429d71-7e77-48a7-8711-3a30a7c896e7"} {"level":"Level(-2)","ts":"2026-02-25T15:54:48.54868315Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-ptq5g-d05b9","reconcileID":"3f429d71-7e77-48a7-8711-3a30a7c896e7","isDeleted":true} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.179007304Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-b7dnm","reconcileID":"b1f8c609-cc34-4e4a-b463-2b43d2d2781f","job":"kueue-managed-test/test-job-b7dnm","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.187570212Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"leader","workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"queue":"test-queue","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.187579308Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"3308fff7-811d-44f4-a8c9-d2e5444b7cde"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.187672048Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"3308fff7-811d-44f4-a8c9-d2e5444b7cde","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.187736613Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"5af4000f-f01e-4e5f-9387-5efa7824087e"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.187875963Z","logger":"scheduler","caller":"scheduler/scheduler.go:248","msg":"Attempting to schedule workload","replica-role":"follower","schedulingCycle":1,"workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.187927084Z","logger":"scheduler","caller":"scheduler/scheduler.go:697","msg":"Workload assumed in the cache","replica-role":"follower","schedulingCycle":1,"workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.188975493Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-b7dnm","reconcileID":"f311f793-d174-4ee7-83c4-05f8a87d6344","job":"kueue-managed-test/test-job-b7dnm","gvk":"batch/v1, Kind=Job"} {"level":"debug","ts":"2026-02-25T15:54:49.188987061Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Created Workload: kueue-managed-test/job-test-job-b7dnm-99f6f","type":"Normal","object":{"kind":"Job","namespace":"kueue-managed-test","name":"test-job-b7dnm","uid":"56c63b2f-0462-44a1-8757-92af7ef51831","apiVersion":"batch/v1","resourceVersion":"142431"},"reason":"CreatedWorkload"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.197613757Z","logger":"scheduler","caller":"scheduler/scheduler.go:660","msg":"Workload successfully admitted and assigned flavors","replica-role":"follower","schedulingCycle":1,"workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"clusterQueue":{"name":"test-clusterqueue"},"assignments":[{"name":"main","flavors":{"cpu":"default","memory":"default"},"resourceUsage":{"cpu":"200m","memory":"512Mi"},"count":1}]} {"level":"debug","ts":"2026-02-25T15:54:49.197729126Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Quota reserved in ClusterQueue test-clusterqueue, wait time since queued was 0s","type":"Normal","object":{"kind":"Workload","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","uid":"d4ed6467-b308-417e-a3db-ba7c4d554b0c","apiVersion":"kueue.x-k8s.io/v1beta2","resourceVersion":"142436"},"reason":"QuotaReserved"} {"level":"debug","ts":"2026-02-25T15:54:49.197950237Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Admitted by ClusterQueue test-clusterqueue, wait time since reservation was 0s","type":"Normal","object":{"kind":"Workload","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","uid":"d4ed6467-b308-417e-a3db-ba7c4d554b0c","apiVersion":"kueue.x-k8s.io/v1beta2","resourceVersion":"142436"},"reason":"Admitted"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.197982053Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"aba0751d-4afb-4f88-bdea-0bd660268a40"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.198033929Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"aba0751d-4afb-4f88-bdea-0bd660268a40","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.198049313Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted","prevStatus":"pending","clusterQueue":"test-clusterqueue"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.19816536Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-b7dnm","reconcileID":"85c41867-e185-49c3-9fef-2d92cfbcbde3","job":"kueue-managed-test/test-job-b7dnm","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.198272806Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"29d8e8b0-050a-4299-b3d0-d239db21fa9b"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.198263743Z","logger":"job","caller":"jobframework/reconciler.go:598","msg":"Job admitted, unsuspending","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-b7dnm","reconcileID":"85c41867-e185-49c3-9fef-2d92cfbcbde3","job":"kueue-managed-test/test-job-b7dnm","gvk":"batch/v1, Kind=Job"} {"level":"debug","ts":"2026-02-25T15:54:49.206567237Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Admitted by clusterQueue test-clusterqueue","type":"Normal","object":{"kind":"Job","namespace":"kueue-managed-test","name":"test-job-b7dnm","uid":"56c63b2f-0462-44a1-8757-92af7ef51831","apiVersion":"batch/v1","resourceVersion":"142438"},"reason":"Started"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.206971061Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-b7dnm","reconcileID":"4776d43d-087e-404c-8214-2893aa1127f0","job":"kueue-managed-test/test-job-b7dnm","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.243256359Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-b7dnm","reconcileID":"ba747b46-40ac-48c4-833b-6ad7ef303167","job":"kueue-managed-test/test-job-b7dnm","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.722378742Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-fw8gz","reconcileID":"e8634250-df79-458c-b6e0-20d6818064e0","job":"kueue-managed-test/test-job-fw8gz","gvk":"batch/v1, Kind=Job"} {"level":"debug","ts":"2026-02-25T15:54:49.733313598Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Created Workload: kueue-managed-test/job-test-job-fw8gz-cb598","type":"Normal","object":{"kind":"Job","namespace":"kueue-managed-test","name":"test-job-fw8gz","uid":"2143be44-4cc1-4a53-bd4c-846481cb3829","apiVersion":"batch/v1","resourceVersion":"142456"},"reason":"CreatedWorkload"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.733306929Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-fw8gz","reconcileID":"8c098a2d-c793-4810-aa1d-7c6e04d826f0","job":"kueue-managed-test/test-job-fw8gz","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.7340941Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"leader","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.734141311Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"a1c60a28-4e53-4025-bfae-4fb88cf2e30a"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.734163011Z","logger":"workload-reconciler","caller":"core/workload_controller.go:831","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.734165496Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"a1c60a28-4e53-4025-bfae-4fb88cf2e30a","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.734221126Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"2c057638-7caf-4779-87f5-938e00920b62"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.743795981Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"34ff050d-8d8b-47c4-99b4-abd07d22b62d"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.743842066Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"34ff050d-8d8b-47c4-99b4-abd07d22b62d","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.743799222Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.743886353Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.743936362Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"d4251cfc-7f58-423b-9a64-c877a7c8dc6b"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.745699197Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-fw8gz","reconcileID":"50ab82f2-fadb-4c3f-ba24-4d75c683c672","job":"kueue-managed-test/test-job-fw8gz","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:49.745805592Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-fw8gz","reconcileID":"ed961e85-a380-4254-bec8-dc1bb48d1505","job":"kueue-managed-test/test-job-fw8gz","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:54:50.188819001Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:167","msg":"Reconcile LocalQueue","replica-role":"leader","namespace":"kueue-managed-test","name":"test-queue","reconcileID":"df47b88f-0e0b-47d5-abd8-3493e771286c"} {"level":"Level(-2)","ts":"2026-02-25T15:54:50.188849945Z","logger":"clusterqueue-reconciler","caller":"core/clusterqueue_controller.go:161","msg":"Reconcile ClusterQueue","replica-role":"leader","name":"test-clusterqueue","reconcileID":"465c365e-f43c-40c8-889b-de3874d2b4e0"} {"level":"Level(-2)","ts":"2026-02-25T15:54:50.199373861Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:254","msg":"Queue update event","replica-role":"leader","localQueue":{"name":"test-queue","namespace":"kueue-managed-test"}} {"level":"Level(-2)","ts":"2026-02-25T15:54:50.199608309Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:346","msg":"ClusterQueue update event","replica-role":"leader","clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T15:54:50.199806984Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:167","msg":"Reconcile LocalQueue","replica-role":"leader","namespace":"kueue-managed-test","name":"test-queue","reconcileID":"7eecdd52-f7b3-4c0c-b099-f38a83c402fc"} {"level":"Level(-2)","ts":"2026-02-25T15:54:50.19990456Z","logger":"clusterqueue-reconciler","caller":"core/clusterqueue_controller.go:161","msg":"Reconcile ClusterQueue","replica-role":"leader","name":"test-clusterqueue","reconcileID":"ba17753b-8293-4318-9d43-63f0760709e9"} {"level":"Level(-2)","ts":"2026-02-25T15:54:55.925348272Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-b7dnm","reconcileID":"12851701-68ac-4b69-a60b-d44b853b1ec7","job":"kueue-managed-test/test-job-b7dnm","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.418695762Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:95","msg":"Reconcile ResourceFlavor","replica-role":"leader","name":"default","reconcileID":"17974fed-cd8e-4ee6-9120-c14ddc74aa84"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.419682372Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:95","msg":"Reconcile ResourceFlavor","replica-role":"leader","name":"default-flavor","reconcileID":"be068d1b-f680-44e6-8993-bf0484560a81"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.419727144Z","logger":"resourceflavor-reconciler","caller":"core/resourceflavor_controller.go:95","msg":"Reconcile ResourceFlavor","replica-role":"leader","name":"default-flavor2","reconcileID":"41129b42-6671-4fc3-a591-b2983da17939"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.52325002Z","logger":"clusterqueue-reconciler","caller":"core/clusterqueue_controller.go:161","msg":"Reconcile ClusterQueue","replica-role":"leader","name":"cluster-queue","reconcileID":"3ca0422d-b657-4ea8-94fc-f04c1f08b7a1"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.523220996Z","logger":"tas-node-failure-reconciler","caller":"tas/node_failure_controller.go:73","msg":"Reconcile Node","replica-role":"leader","name":"ip-10-0-10-243.us-east-2.compute.internal","reconcileID":"16cc0c68-b045-48dd-83b4-62516fa7cb6f"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.523494845Z","logger":"tas-node-failure-reconciler","caller":"tas/node_failure_controller.go:73","msg":"Reconcile Node","replica-role":"leader","name":"ip-10-0-30-56.us-east-2.compute.internal","reconcileID":"2a94d5c1-d2fa-4f75-bace-53ab74350824"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.523559524Z","logger":"tas-node-failure-reconciler","caller":"tas/node_failure_controller.go:73","msg":"Reconcile Node","replica-role":"leader","name":"ip-10-0-32-68.us-east-2.compute.internal","reconcileID":"39c76157-b61d-486e-8657-2e4facf4296e"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.523615638Z","logger":"tas-node-failure-reconciler","caller":"tas/node_failure_controller.go:73","msg":"Reconcile Node","replica-role":"leader","name":"ip-10-0-48-252.us-east-2.compute.internal","reconcileID":"4f2f77b6-b983-4e85-ab88-caed448a0336"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.52367458Z","logger":"tas-node-failure-reconciler","caller":"tas/node_failure_controller.go:73","msg":"Reconcile Node","replica-role":"leader","name":"ip-10-0-66-156.us-east-2.compute.internal","reconcileID":"306dac96-1460-4da1-ab7f-1d03c587257e"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.523719741Z","logger":"tas-node-failure-reconciler","caller":"tas/node_failure_controller.go:73","msg":"Reconcile Node","replica-role":"leader","name":"ip-10-0-94-129.us-east-2.compute.internal","reconcileID":"8a0ff90c-f2e8-406a-a064-fe9991064b70"} {"level":"Level(-2)","ts":"2026-02-25T15:56:01.533833314Z","logger":"clusterqueue-reconciler","caller":"core/clusterqueue_controller.go:161","msg":"Reconcile ClusterQueue","replica-role":"leader","name":"cluster-queue2","reconcileID":"66f989ac-1097-448b-8d87-d2e45c147959"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.084278552Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.084439381Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.084338738Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"3c463d5e-0853-4834-a033-7c5c4c2e90a4"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.084537368Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"3c463d5e-0853-4834-a033-7c5c4c2e90a4","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.084503955Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"4d2a674b-ccdc-41dc-8052-98cfc109865d"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.094103263Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"leader","workload":{"name":"job-test-job-fw8gz-cb598","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.094103197Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"6dba1531-6475-4ed2-b7fd-e67f42243334"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.094256343Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-fw8gz-cb598","reconcileID":"6dba1531-6475-4ed2-b7fd-e67f42243334","isDeleted":true} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.427221366Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"36711661-2013-44f7-bba5-60d02a8e509f"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.427295639Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"36711661-2013-44f7-bba5-60d02a8e509f","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.427238968Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted","clusterQueue":"test-clusterqueue"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.427470842Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"e9cb182b-cec6-4e9b-b8fb-116e861cdd2e"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.455526204Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"3498d9c9-0061-4439-83b5-cb57d068d9a8"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.455565004Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"leader","workload":{"name":"job-test-job-b7dnm-99f6f","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted"} {"level":"Level(-2)","ts":"2026-02-25T15:57:50.455594574Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-b7dnm-99f6f","reconcileID":"3498d9c9-0061-4439-83b5-cb57d068d9a8","isDeleted":true} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.427979086Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:167","msg":"Reconcile LocalQueue","replica-role":"leader","namespace":"kueue-managed-test","name":"test-queue","reconcileID":"0d7c7c53-f69c-44be-9ad0-681111dade23"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.427981178Z","logger":"clusterqueue-reconciler","caller":"core/clusterqueue_controller.go:161","msg":"Reconcile ClusterQueue","replica-role":"leader","name":"test-clusterqueue","reconcileID":"eccc175e-6e4c-4c82-94f3-29f6fc7b6cbb"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.438916396Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:254","msg":"Queue update event","replica-role":"leader","localQueue":{"name":"test-queue","namespace":"kueue-managed-test"}} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.439022964Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:167","msg":"Reconcile LocalQueue","replica-role":"leader","namespace":"kueue-managed-test","name":"test-queue","reconcileID":"7cc60293-6b0a-4698-90e5-e74ed5738bff"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.440744364Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:346","msg":"ClusterQueue update event","replica-role":"leader","clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.440900864Z","logger":"clusterqueue-reconciler","caller":"core/clusterqueue_controller.go:161","msg":"Reconcile ClusterQueue","replica-role":"leader","name":"test-clusterqueue","reconcileID":"23e92830-d39a-475c-a416-bdad48d43d60"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.824456514Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-8xmxz","reconcileID":"e0336057-e71f-417e-b62b-1cfbb68d6a48","job":"kueue-managed-test/test-job-8xmxz","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.834447588Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-8xmxz","reconcileID":"c4804e44-0061-4be2-b20b-d3eb82d21295","job":"kueue-managed-test/test-job-8xmxz","gvk":"batch/v1, Kind=Job"} {"level":"debug","ts":"2026-02-25T15:57:51.834555519Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Created Workload: kueue-managed-test/job-test-job-8xmxz-b852d","type":"Normal","object":{"kind":"Job","namespace":"kueue-managed-test","name":"test-job-8xmxz","uid":"e677d513-5e59-49b3-a4a6-8031f1657633","apiVersion":"batch/v1","resourceVersion":"143859"},"reason":"CreatedWorkload"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.834772837Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-8xmxz-b852d","reconcileID":"b32b985c-1328-44a5-b1a3-7b7b969ca2f3"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.834874736Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-8xmxz-b852d","reconcileID":"b32b985c-1328-44a5-b1a3-7b7b969ca2f3","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.834809681Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"leader","workload":{"name":"job-test-job-8xmxz-b852d","namespace":"kueue-managed-test"},"queue":"test-queue","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.83510326Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-8xmxz-b852d","reconcileID":"1bae949e-5f18-4ed3-8ab2-afafa226b005"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.835210722Z","logger":"scheduler","caller":"scheduler/scheduler.go:248","msg":"Attempting to schedule workload","replica-role":"follower","schedulingCycle":2,"workload":{"name":"job-test-job-8xmxz-b852d","namespace":"kueue-managed-test"},"clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.835276282Z","logger":"scheduler","caller":"scheduler/scheduler.go:697","msg":"Workload assumed in the cache","replica-role":"follower","schedulingCycle":2,"workload":{"name":"job-test-job-8xmxz-b852d","namespace":"kueue-managed-test"},"clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.846643224Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-8xmxz","reconcileID":"2c7eacb3-76ac-4f46-abcb-1c61a8b20b79","job":"kueue-managed-test/test-job-8xmxz","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.846748063Z","logger":"scheduler","caller":"scheduler/scheduler.go:660","msg":"Workload successfully admitted and assigned flavors","replica-role":"follower","schedulingCycle":2,"workload":{"name":"job-test-job-8xmxz-b852d","namespace":"kueue-managed-test"},"clusterQueue":{"name":"test-clusterqueue"},"assignments":[{"name":"main","flavors":{"cpu":"default","memory":"default"},"resourceUsage":{"cpu":"200m","memory":"512Mi"},"count":1}]} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.846778969Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-8xmxz-b852d","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted","prevStatus":"pending","clusterQueue":"test-clusterqueue"} {"level":"debug","ts":"2026-02-25T15:57:51.84683832Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Quota reserved in ClusterQueue test-clusterqueue, wait time since queued was 1s","type":"Normal","object":{"kind":"Workload","namespace":"kueue-managed-test","name":"job-test-job-8xmxz-b852d","uid":"be76319c-1657-4941-bd8c-b056e1db83e8","apiVersion":"kueue.x-k8s.io/v1beta2","resourceVersion":"143864"},"reason":"QuotaReserved"} {"level":"debug","ts":"2026-02-25T15:57:51.846897916Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Admitted by ClusterQueue test-clusterqueue, wait time since reservation was 0s","type":"Normal","object":{"kind":"Workload","namespace":"kueue-managed-test","name":"job-test-job-8xmxz-b852d","uid":"be76319c-1657-4941-bd8c-b056e1db83e8","apiVersion":"kueue.x-k8s.io/v1beta2","resourceVersion":"143864"},"reason":"Admitted"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.846850889Z","logger":"job","caller":"jobframework/reconciler.go:598","msg":"Job admitted, unsuspending","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-8xmxz","reconcileID":"2c7eacb3-76ac-4f46-abcb-1c61a8b20b79","job":"kueue-managed-test/test-job-8xmxz","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.846930825Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-8xmxz-b852d","reconcileID":"4481cf24-7ba1-4312-82f5-f24e04c5e735"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.846951987Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-8xmxz-b852d","reconcileID":"40546c1b-1006-4f87-8fc6-ae9188d38633"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.846999705Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-8xmxz-b852d","reconcileID":"40546c1b-1006-4f87-8fc6-ae9188d38633","isDeleted":false} {"level":"debug","ts":"2026-02-25T15:57:51.85601836Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Admitted by clusterQueue test-clusterqueue","type":"Normal","object":{"kind":"Job","namespace":"kueue-managed-test","name":"test-job-8xmxz","uid":"e677d513-5e59-49b3-a4a6-8031f1657633","apiVersion":"batch/v1","resourceVersion":"143866"},"reason":"Started"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.856069032Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-8xmxz","reconcileID":"36d011db-870f-4b23-a723-9589dc7463f8","job":"kueue-managed-test/test-job-8xmxz","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.856231329Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-8xmxz","reconcileID":"f49c770d-58da-48d5-b7df-5c7c7226b148","job":"kueue-managed-test/test-job-8xmxz","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:57:51.890242775Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-8xmxz","reconcileID":"55dd61b6-4eda-4776-b9e6-e89791000d5c","job":"kueue-managed-test/test-job-8xmxz","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.358416465Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-c74bb","reconcileID":"090ae3bd-e0b6-47c9-8e2a-f2dd982ead80","job":"kueue-managed-test/test-job-c74bb","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.366729836Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-c74bb-45a97","reconcileID":"e085382a-d043-4b1d-9d4a-748bad2bdaaf"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.366819823Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-c74bb-45a97","reconcileID":"e085382a-d043-4b1d-9d4a-748bad2bdaaf","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.36673599Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"leader","workload":{"name":"job-test-job-c74bb-45a97","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.36688869Z","logger":"workload-reconciler","caller":"core/workload_controller.go:831","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-c74bb-45a97","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.366938364Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-c74bb-45a97","reconcileID":"30cb6a9d-3060-49c2-af53-a1facce8bb90"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.367964129Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-c74bb","reconcileID":"1b1e76e3-10a7-44a2-93ae-a04dfcf0a31b","job":"kueue-managed-test/test-job-c74bb","gvk":"batch/v1, Kind=Job"} {"level":"debug","ts":"2026-02-25T15:57:52.367974739Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Created Workload: kueue-managed-test/job-test-job-c74bb-45a97","type":"Normal","object":{"kind":"Job","namespace":"kueue-managed-test","name":"test-job-c74bb","uid":"80ee4f54-69a7-479d-8010-36d76047a2cb","apiVersion":"batch/v1","resourceVersion":"143882"},"reason":"CreatedWorkload"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.37620912Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-c74bb-45a97","reconcileID":"a38ea54f-af34-4775-877b-16ca7693e7d0"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.376252802Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-c74bb-45a97","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.376267207Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-c74bb-45a97","reconcileID":"a38ea54f-af34-4775-877b-16ca7693e7d0","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.376306638Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-c74bb-45a97","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.376360532Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-c74bb-45a97","reconcileID":"b9177d6d-9084-4e8a-8ee3-73ef18635a31"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.37635675Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-c74bb","reconcileID":"781576ad-9c0e-4eda-ad5f-cf074b7e8882","job":"kueue-managed-test/test-job-c74bb","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.835840643Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:167","msg":"Reconcile LocalQueue","replica-role":"leader","namespace":"kueue-managed-test","name":"test-queue","reconcileID":"1b318c95-922c-4b7a-9ebf-7cf38ba278f9"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.835842794Z","logger":"clusterqueue-reconciler","caller":"core/clusterqueue_controller.go:161","msg":"Reconcile ClusterQueue","replica-role":"leader","name":"test-clusterqueue","reconcileID":"c029edd9-d514-4b81-84ad-e0e7d53d535a"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.845241842Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:254","msg":"Queue update event","replica-role":"leader","localQueue":{"name":"test-queue","namespace":"kueue-managed-test"}} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.845380277Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:167","msg":"Reconcile LocalQueue","replica-role":"leader","namespace":"kueue-managed-test","name":"test-queue","reconcileID":"eb2b3439-1685-4aa9-87ea-d64f9839f552"} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.846865127Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:346","msg":"ClusterQueue update event","replica-role":"leader","clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T15:57:52.847035515Z","logger":"clusterqueue-reconciler","caller":"core/clusterqueue_controller.go:161","msg":"Reconcile ClusterQueue","replica-role":"leader","name":"test-clusterqueue","reconcileID":"7f24134e-1a0a-4747-87ec-b97080828282"} {"level":"Level(-2)","ts":"2026-02-25T15:57:54.483219685Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-8xmxz","reconcileID":"209edf6a-e8c5-4ec9-afc0-93432d9c7631","job":"kueue-managed-test/test-job-8xmxz","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T16:00:52.702921597Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-c74bb-45a97","reconcileID":"56c32a69-0807-4839-b190-a3e56bc5d26a"} {"level":"Level(-2)","ts":"2026-02-25T16:00:52.703064735Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-c74bb-45a97","reconcileID":"56c32a69-0807-4839-b190-a3e56bc5d26a","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T16:00:52.702994165Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-c74bb-45a97","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T16:00:52.703156108Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-c74bb-45a97","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T16:00:52.703225822Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-c74bb-45a97","reconcileID":"9bc7a4ee-6c2d-429e-bacb-861ad02a9cb5"} {"level":"Level(-2)","ts":"2026-02-25T16:00:52.714842149Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"leader","workload":{"name":"job-test-job-c74bb-45a97","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T16:00:52.714838594Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-c74bb-45a97","reconcileID":"879472a5-6ce5-440f-bd31-f4ed7ef6d4c8"} {"level":"Level(-2)","ts":"2026-02-25T16:00:52.714961967Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-c74bb-45a97","reconcileID":"879472a5-6ce5-440f-bd31-f4ed7ef6d4c8","isDeleted":true} {"level":"Level(-2)","ts":"2026-02-25T16:00:53.042959562Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-8xmxz-b852d","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted","clusterQueue":"test-clusterqueue"} {"level":"Level(-2)","ts":"2026-02-25T16:00:53.043024628Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-8xmxz-b852d","reconcileID":"60e5ee9a-1d47-430f-aed1-aecbcaf005eb"} {"level":"Level(-2)","ts":"2026-02-25T16:00:53.043090876Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-8xmxz-b852d","reconcileID":"60e5ee9a-1d47-430f-aed1-aecbcaf005eb","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T16:00:53.043204166Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-8xmxz-b852d","reconcileID":"a4847ba6-7196-42e0-ac48-11522922d227"} {"level":"Level(-2)","ts":"2026-02-25T16:00:53.070454093Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-8xmxz-b852d","reconcileID":"1a216286-193d-4f40-8279-dd73d0d47b0f"} {"level":"Level(-2)","ts":"2026-02-25T16:00:53.070568122Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-8xmxz-b852d","reconcileID":"1a216286-193d-4f40-8279-dd73d0d47b0f","isDeleted":true} {"level":"Level(-2)","ts":"2026-02-25T16:00:53.070501725Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"leader","workload":{"name":"job-test-job-8xmxz-b852d","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted"} {"level":"Level(-2)","ts":"2026-02-25T16:00:54.043555485Z","logger":"clusterqueue-reconciler","caller":"core/clusterqueue_controller.go:161","msg":"Reconcile ClusterQueue","replica-role":"leader","name":"test-clusterqueue","reconcileID":"766a19a9-d560-4f72-a28b-ef337f590ceb"} {"level":"Level(-2)","ts":"2026-02-25T16:00:54.043551639Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:167","msg":"Reconcile LocalQueue","replica-role":"leader","namespace":"kueue-managed-test","name":"test-queue","reconcileID":"75c0e86b-7bb4-45f7-ac2b-cff9fb711950"} {"level":"Level(-2)","ts":"2026-02-25T16:00:54.052577465Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:254","msg":"Queue update event","replica-role":"leader","localQueue":{"name":"test-queue","namespace":"kueue-managed-test"}} {"level":"Level(-2)","ts":"2026-02-25T16:00:54.052640447Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:167","msg":"Reconcile LocalQueue","replica-role":"leader","namespace":"kueue-managed-test","name":"test-queue","reconcileID":"f6de9c74-8def-4158-8b58-2b8ec6ef0f7c"} {"level":"Level(-2)","ts":"2026-02-25T16:00:54.053928105Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:346","msg":"ClusterQueue update event","replica-role":"leader","clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T16:00:54.054072115Z","logger":"clusterqueue-reconciler","caller":"core/clusterqueue_controller.go:161","msg":"Reconcile ClusterQueue","replica-role":"leader","name":"test-clusterqueue","reconcileID":"be534a51-b758-40b3-b495-49925e48508d"} {"level":"Level(-2)","ts":"2026-02-25T16:00:54.701622869Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-869lt","reconcileID":"69f68706-5ee7-4ad3-ac3e-ed58525274a7","job":"kueue-managed-test/test-job-869lt","gvk":"batch/v1, Kind=Job"} {"level":"debug","ts":"2026-02-25T16:00:54.711795787Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Created Workload: kueue-managed-test/job-test-job-869lt-9a3b7","type":"Normal","object":{"kind":"Job","namespace":"kueue-managed-test","name":"test-job-869lt","uid":"4a2dca97-8326-41ed-9c11-905797eefd97","apiVersion":"batch/v1","resourceVersion":"144866"},"reason":"CreatedWorkload"} {"level":"Level(-2)","ts":"2026-02-25T16:00:54.711853211Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-869lt","reconcileID":"9556e70e-6658-41f5-8934-c20cab4f046b","job":"kueue-managed-test/test-job-869lt","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T16:00:54.712425448Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"leader","workload":{"name":"job-test-job-869lt-9a3b7","namespace":"kueue-managed-test"},"queue":"test-queue","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T16:00:54.712438139Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-869lt-9a3b7","reconcileID":"48302440-c357-48e5-8472-20365b6d56d2"} {"level":"Level(-2)","ts":"2026-02-25T16:00:54.712474052Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-869lt-9a3b7","reconcileID":"48302440-c357-48e5-8472-20365b6d56d2","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T16:00:54.712560415Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-869lt-9a3b7","reconcileID":"db66e9b5-14d4-441b-9c44-229bd88f98f6"} {"level":"Level(-2)","ts":"2026-02-25T16:00:54.712640571Z","logger":"scheduler","caller":"scheduler/scheduler.go:248","msg":"Attempting to schedule workload","replica-role":"follower","schedulingCycle":3,"workload":{"name":"job-test-job-869lt-9a3b7","namespace":"kueue-managed-test"},"clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T16:00:54.71267574Z","logger":"scheduler","caller":"scheduler/scheduler.go:697","msg":"Workload assumed in the cache","replica-role":"follower","schedulingCycle":3,"workload":{"name":"job-test-job-869lt-9a3b7","namespace":"kueue-managed-test"},"clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T16:00:54.723243442Z","logger":"scheduler","caller":"scheduler/scheduler.go:660","msg":"Workload successfully admitted and assigned flavors","replica-role":"follower","schedulingCycle":3,"workload":{"name":"job-test-job-869lt-9a3b7","namespace":"kueue-managed-test"},"clusterQueue":{"name":"test-clusterqueue"},"assignments":[{"name":"main","flavors":{"cpu":"default","memory":"default"},"resourceUsage":{"cpu":"200m","memory":"512Mi"},"count":1}]} {"level":"debug","ts":"2026-02-25T16:00:54.723280807Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Quota reserved in ClusterQueue test-clusterqueue, wait time since queued was 1s","type":"Normal","object":{"kind":"Workload","namespace":"kueue-managed-test","name":"job-test-job-869lt-9a3b7","uid":"f04e442d-5899-4c80-a26b-47f72649a8d5","apiVersion":"kueue.x-k8s.io/v1beta2","resourceVersion":"144871"},"reason":"QuotaReserved"} {"level":"debug","ts":"2026-02-25T16:00:54.723296596Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Admitted by ClusterQueue test-clusterqueue, wait time since reservation was 0s","type":"Normal","object":{"kind":"Workload","namespace":"kueue-managed-test","name":"job-test-job-869lt-9a3b7","uid":"f04e442d-5899-4c80-a26b-47f72649a8d5","apiVersion":"kueue.x-k8s.io/v1beta2","resourceVersion":"144871"},"reason":"Admitted"} {"level":"Level(-2)","ts":"2026-02-25T16:00:54.723370112Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-869lt-9a3b7","reconcileID":"7a7ece99-6233-4af3-8e46-82644a60c506"} {"level":"Level(-2)","ts":"2026-02-25T16:00:54.723402951Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-869lt-9a3b7","reconcileID":"7a7ece99-6233-4af3-8e46-82644a60c506","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T16:00:54.723436752Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-869lt-9a3b7","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted","prevStatus":"pending","clusterQueue":"test-clusterqueue"} {"level":"Level(-2)","ts":"2026-02-25T16:00:54.723560388Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-869lt-9a3b7","reconcileID":"da5cb21f-8019-418d-9def-7ce59d4a0f61"} {"level":"Level(-2)","ts":"2026-02-25T16:00:54.724372702Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-869lt","reconcileID":"6f16a58b-785d-40b0-a04a-1b25f0a897f5","job":"kueue-managed-test/test-job-869lt","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T16:00:54.724457486Z","logger":"job","caller":"jobframework/reconciler.go:598","msg":"Job admitted, unsuspending","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-869lt","reconcileID":"6f16a58b-785d-40b0-a04a-1b25f0a897f5","job":"kueue-managed-test/test-job-869lt","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T16:00:54.735242014Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-869lt","reconcileID":"01dbdc7b-a893-4f21-aa69-134cbdad17c5","job":"kueue-managed-test/test-job-869lt","gvk":"batch/v1, Kind=Job"} {"level":"debug","ts":"2026-02-25T16:00:54.735271246Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Admitted by clusterQueue test-clusterqueue","type":"Normal","object":{"kind":"Job","namespace":"kueue-managed-test","name":"test-job-869lt","uid":"4a2dca97-8326-41ed-9c11-905797eefd97","apiVersion":"batch/v1","resourceVersion":"144874"},"reason":"Started"} {"level":"Level(-2)","ts":"2026-02-25T16:00:54.767928738Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-869lt","reconcileID":"f62381ba-cd83-41d8-b920-19ada850348b","job":"kueue-managed-test/test-job-869lt","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T16:00:55.205470526Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-2bkxc","reconcileID":"e4909c20-0a3a-4ccf-8d4c-d62e9c852650","job":"kueue-managed-test/test-job-2bkxc","gvk":"batch/v1, Kind=Job"} {"level":"debug","ts":"2026-02-25T16:00:55.213072495Z","logger":"events","caller":"recorder/recorder.go:104","msg":"Created Workload: kueue-managed-test/job-test-job-2bkxc-20dff","type":"Normal","object":{"kind":"Job","namespace":"kueue-managed-test","name":"test-job-2bkxc","uid":"c62b3881-45db-4018-9b0c-308c31e4f6e3","apiVersion":"batch/v1","resourceVersion":"144884"},"reason":"CreatedWorkload"} {"level":"Level(-2)","ts":"2026-02-25T16:00:55.213226698Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-2bkxc-20dff","reconcileID":"16f172ed-4b91-4ceb-b0a6-c40d80cfde98"} {"level":"Level(-2)","ts":"2026-02-25T16:00:55.213242577Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-2bkxc","reconcileID":"0f39b456-8ada-418f-a675-db4bbb81b670","job":"kueue-managed-test/test-job-2bkxc","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T16:00:55.213268294Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-2bkxc-20dff","reconcileID":"16f172ed-4b91-4ceb-b0a6-c40d80cfde98","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T16:00:55.213281425Z","logger":"workload-reconciler","caller":"core/workload_controller.go:811","msg":"Workload create event","replica-role":"leader","workload":{"name":"job-test-job-2bkxc-20dff","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T16:00:55.213336423Z","logger":"workload-reconciler","caller":"core/workload_controller.go:831","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-2bkxc-20dff","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T16:00:55.213392818Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-2bkxc-20dff","reconcileID":"22e29062-3e40-472e-9143-b96d75336a45"} {"level":"Level(-2)","ts":"2026-02-25T16:00:55.214758848Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-2bkxc","reconcileID":"4a6ad0be-508a-4d68-8fd8-0331163e1e11","job":"kueue-managed-test/test-job-2bkxc","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T16:00:55.222077864Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-2bkxc-20dff","reconcileID":"042afdf8-2849-4001-b924-f5a305a4e0f0"} {"level":"Level(-2)","ts":"2026-02-25T16:00:55.222132769Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-2bkxc-20dff","reconcileID":"042afdf8-2849-4001-b924-f5a305a4e0f0","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T16:00:55.222137913Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-2bkxc-20dff","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T16:00:55.22216899Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-2bkxc-20dff","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T16:00:55.222212129Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-2bkxc-20dff","reconcileID":"57ad5072-0710-4ecd-b3b4-eaf0d32ef2fb"} {"level":"Level(-2)","ts":"2026-02-25T16:00:55.222224939Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-2bkxc","reconcileID":"71290d9f-ae5a-4f2d-bbb8-d6e84a95f08f","job":"kueue-managed-test/test-job-2bkxc","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T16:00:55.71331526Z","logger":"clusterqueue-reconciler","caller":"core/clusterqueue_controller.go:161","msg":"Reconcile ClusterQueue","replica-role":"leader","name":"test-clusterqueue","reconcileID":"87a91342-616c-419a-a5bd-634330d37dcc"} {"level":"Level(-2)","ts":"2026-02-25T16:00:55.713315264Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:167","msg":"Reconcile LocalQueue","replica-role":"leader","namespace":"kueue-managed-test","name":"test-queue","reconcileID":"e5392020-5d6e-4c31-9f90-fbe399d252fc"} {"level":"Level(-2)","ts":"2026-02-25T16:00:55.723547175Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:254","msg":"Queue update event","replica-role":"leader","localQueue":{"name":"test-queue","namespace":"kueue-managed-test"}} {"level":"Level(-2)","ts":"2026-02-25T16:00:55.724179628Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:167","msg":"Reconcile LocalQueue","replica-role":"leader","namespace":"kueue-managed-test","name":"test-queue","reconcileID":"a8e5845f-7e52-4956-a4e6-17a2ea9a88f2"} {"level":"Level(-2)","ts":"2026-02-25T16:00:55.724721906Z","logger":"cluster-queue-reconciler","caller":"core/clusterqueue_controller.go:346","msg":"ClusterQueue update event","replica-role":"leader","clusterQueue":{"name":"test-clusterqueue"}} {"level":"Level(-2)","ts":"2026-02-25T16:00:55.725160217Z","logger":"clusterqueue-reconciler","caller":"core/clusterqueue_controller.go:161","msg":"Reconcile ClusterQueue","replica-role":"leader","name":"test-clusterqueue","reconcileID":"56b0f0dd-3d27-42c6-8336-01753a06d6de"} {"level":"Level(-2)","ts":"2026-02-25T16:00:57.265565241Z","logger":"job","caller":"jobframework/reconciler.go:414","msg":"Reconciling Job","replica-role":"leader","namespace":"kueue-managed-test","name":"test-job-869lt","reconcileID":"518f36cb-c894-45e2-830b-5f2bf35750e1","job":"kueue-managed-test/test-job-869lt","gvk":"batch/v1, Kind=Job"} {"level":"Level(-2)","ts":"2026-02-25T16:03:55.554936604Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-2bkxc-20dff","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T16:03:55.555079029Z","logger":"workload-reconciler","caller":"core/workload_controller.go:924","msg":"ignored an error for now","replica-role":"leader","workload":{"name":"job-test-job-2bkxc-20dff","namespace":"kueue-managed-test"},"queue":"","status":"pending","error":"localQueue doesn't exist or inactive"} {"level":"Level(-2)","ts":"2026-02-25T16:03:55.554936655Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-2bkxc-20dff","reconcileID":"c292f17a-5038-4428-9354-a91102da047e"} {"level":"Level(-2)","ts":"2026-02-25T16:03:55.555151102Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-2bkxc-20dff","reconcileID":"c292f17a-5038-4428-9354-a91102da047e","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T16:03:55.555182191Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-2bkxc-20dff","reconcileID":"ee75bb9e-aa73-4c8a-a262-cd7fe7348e92"} {"level":"Level(-2)","ts":"2026-02-25T16:03:55.566906133Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"leader","workload":{"name":"job-test-job-2bkxc-20dff","namespace":"kueue-managed-test"},"queue":"","status":"pending"} {"level":"Level(-2)","ts":"2026-02-25T16:03:55.566934562Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-2bkxc-20dff","reconcileID":"72f67152-a0d9-4d28-ba38-ca971a91366b"} {"level":"Level(-2)","ts":"2026-02-25T16:03:55.56697926Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-2bkxc-20dff","reconcileID":"72f67152-a0d9-4d28-ba38-ca971a91366b","isDeleted":true} {"level":"Level(-2)","ts":"2026-02-25T16:03:55.919250628Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-869lt-9a3b7","reconcileID":"2cd5dd4c-7ede-4a5b-8584-9d880088bb21"} {"level":"Level(-2)","ts":"2026-02-25T16:03:55.919325823Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-869lt-9a3b7","reconcileID":"2cd5dd4c-7ede-4a5b-8584-9d880088bb21","isDeleted":false} {"level":"Level(-2)","ts":"2026-02-25T16:03:55.919265378Z","logger":"workload-reconciler","caller":"core/workload_controller.go:891","msg":"Workload update event","replica-role":"leader","workload":{"name":"job-test-job-869lt-9a3b7","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted","clusterQueue":"test-clusterqueue"} {"level":"Level(-2)","ts":"2026-02-25T16:03:55.919455216Z","logger":"workload-reconciler","caller":"core/workload_controller.go:186","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-869lt-9a3b7","reconcileID":"77721d4d-5463-44b1-af6d-6baba8e75a34"} {"level":"Level(-2)","ts":"2026-02-25T16:03:55.957738343Z","logger":"workload-reconciler","caller":"core/workload_controller.go:849","msg":"Workload delete event","replica-role":"leader","workload":{"name":"job-test-job-869lt-9a3b7","namespace":"kueue-managed-test"},"queue":"test-queue","status":"admitted"} {"level":"Level(-2)","ts":"2026-02-25T16:03:55.957742007Z","logger":"multikueue-workload","caller":"multikueue/workload.go:157","msg":"Reconcile Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-869lt-9a3b7","reconcileID":"fdd676f1-1019-4611-902b-b4084a45f716"} {"level":"Level(-2)","ts":"2026-02-25T16:03:55.957896263Z","logger":"multikueue-workload","caller":"multikueue/workload.go:183","msg":"Skip Workload","replica-role":"leader","namespace":"kueue-managed-test","name":"job-test-job-869lt-9a3b7","reconcileID":"fdd676f1-1019-4611-902b-b4084a45f716","isDeleted":true} {"level":"Level(-2)","ts":"2026-02-25T16:03:56.92006892Z","logger":"localqueue-reconciler","caller":"core/localqueue_controller.go:167","msg":"Reconcile LocalQueue","replica-role":"leader","namespace":"kueue-managed-test","name":"test-queue","reconcileID":"1cc8647a-707d-477c-86d1-454af6104868"} {"level":"Level(-2)","ts":"2026-02-25T16:03:56.920096218Z","logger":"clusterqueue-reconciler","caller":"core/clusterqueue_controller.go:161","msg":"Reconcile ClusterQueue","replica-role":"leader","name":"test-clusterqueue","reconcileID":"b1aef5d4-bf7a-4b46-ad99-f061b38ebf3e"} STEP: Destroying LocalQueue kueue-managed-test/test-queue @ 02/25/26 13:03:58.574 STEP: Deleting namespace kueue-managed-test @ 02/25/26 13:03:59.123 STEP: Deleting namespace kueue-unmanaged-test @ 02/25/26 13:04:30.681 STEP: Destroying ClusterQueue test-clusterqueue @ 02/25/26 13:04:31.017 STEP: Destroying ResourceFlavor default @ 02/25/26 13:04:31.529 • [FAILED] [583.486 seconds] Kueue Operator when enable webhook via opt-in namespaces [It] should manage jobs only in labeled namespaces [operator] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:533 [FAILED] Timed out after 180.001s. Job not managed by Kueue in labeled namespace Value for field 'Active' failed to satisfy matcher. Expected : 0 to be >= : 1 In [It] at: /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:559 @ 02/25/26 13:03:55.959 There were additional failures detected. To view them in detail run ginkgo -vv ------------------------------ S [SKIPPED] [0.000 seconds] Kueue Operator when enable webhook via opt-in namespaces [It] should manage pods only in labeled namespaces [operator] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:573 [SKIPPED] Spec skipped because an earlier spec in an ordered container failed In [It] at: /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:573 @ 02/25/26 13:04:32.038 ------------------------------ S [SKIPPED] [0.000 seconds] Kueue Operator when enable webhook via opt-in namespaces [It] should manage deployments only in labeled namespaces [operator] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:613 [SKIPPED] Spec skipped because an earlier spec in an ordered container failed In [It] at: /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:613 @ 02/25/26 13:04:32.039 ------------------------------ S [SKIPPED] [0.000 seconds] Kueue Operator when enable webhook via opt-in namespaces [It] should manage statefulsets only in labeled namespaces [operator] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:669 [SKIPPED] Spec skipped because an earlier spec in an ordered container failed In [It] at: /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:669 @ 02/25/26 13:04:32.04 ------------------------------ S [SKIPPED] [0.000 seconds] Kueue Operator when enable webhook via opt-in namespaces [It] should suspend jobsets only in labeled namespaces when labelPolicy=None [operator] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:728 [SKIPPED] Spec skipped because an earlier spec in an ordered container failed In [It] at: /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:728 @ 02/25/26 13:04:32.04 ------------------------------ S [SKIPPED] [0.000 seconds] Kueue Operator when enable webhook via opt-in namespaces [It] should manage LeaderWorkerSet without queue name in labeled namespace when labelPolicy=None [operator] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:784 [SKIPPED] Spec skipped because an earlier spec in an ordered container failed In [It] at: /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:784 @ 02/25/26 13:04:32.041 ------------------------------ S [SKIPPED] [0.000 seconds] Kueue Operator when enable webhook via opt-in namespaces [It] should not manage LeaderWorkerSet without queue name in unlabeled namespace when labelPolicy=None [operator] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:856 [SKIPPED] Spec skipped because an earlier spec in an ordered container failed In [It] at: /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:856 @ 02/25/26 13:04:32.041 ------------------------------ S [SKIPPED] [0.000 seconds] Kueue Operator when enable webhook via opt-in namespaces [It] should expose metrics endpoint with TLS [operator] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:914 [SKIPPED] Spec skipped because an earlier spec in an ordered container failed In [It] at: /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:914 @ 02/25/26 13:04:32.041 ------------------------------ SS ------------------------------ Preemption when Preemption is Fair Sharing should preempt workloads [preemption] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_preemption_test.go:59 STEP: Saving initial Kueue configuration @ 02/25/26 13:04:32.041 STEP: Updating Kueue configuration to use FairSharing preemption @ 02/25/26 13:04:32.209 STEP: Feching Kueue Instance @ 02/25/26 13:04:32.209 STEP: Updating Kueue config @ 02/25/26 13:04:32.72 STEP: Waiting for kueue-controller-manager resource version to change from 141880 @ 02/25/26 13:04:32.9 STEP: Checking if deployment replicas: 3 matches amount of ready replicas: 1 @ 02/25/26 13:04:43.568 STEP: Checking if deployment replicas: 2 matches amount of ready replicas: 2 @ 02/25/26 13:04:53.904 STEP: Waiting for webhook configurations to exist @ 02/25/26 13:04:53.904 STEP: Waiting for webhook to handle requests successfully @ 02/25/26 13:04:54.243 I0225 13:04:54.594825 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-b5dlk I0225 13:04:54.758893 45539 e2e_operator_test.go:1494] Webhook success 1/3 I0225 13:04:58.121925 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-d7tfp I0225 13:04:58.312732 45539 e2e_operator_test.go:1494] Webhook success 2/3 I0225 13:05:01.672661 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-kcbp4 I0225 13:05:01.853812 45539 e2e_operator_test.go:1491] Webhook stable after 3 consecutive successes STEP: Creating Resource Flavor @ 02/25/26 13:05:01.854 STEP: Creating ClusterQueue, Namespace and LocalQueue for A @ 02/25/26 13:05:02.034 STEP: Creating LocalQueue local-queue-a in namespace preemption-a-vwf2r @ 02/25/26 13:05:02.4 STEP: Creating ClusterQueue, Namespace and LocalQueue for B @ 02/25/26 13:05:02.566 STEP: Creating LocalQueue local-queue-b in namespace preemption-b-f29cj @ 02/25/26 13:05:02.912 STEP: Creating a job on A that borrows resources from the cohort @ 02/25/26 13:05:03.093 STEP: Verifying borrowing job workload is admitted @ 02/25/26 13:05:03.28 STEP: Verifying clusterQueueA borrowed 250m CPU @ 02/25/26 13:05:03.44 STEP: Creating a job on B that will reclaim quota from A @ 02/25/26 13:05:13.781 STEP: Verifying reclaim job workload is admitted @ 02/25/26 13:05:13.957 STEP: Verifying borrowing job on A was preempted @ 02/25/26 13:05:24.27 STEP: Waiting for reclaim job to finish @ 02/25/26 13:05:24.43 STEP: Verifying borrowing job on A is re-admitted after reclaim job finishes @ 02/25/26 13:06:05.294 STEP: Restoring initial Kueue configuration @ 02/25/26 13:06:05.463 STEP: Feching Kueue Instance @ 02/25/26 13:06:05.463 STEP: Updating Kueue config @ 02/25/26 13:06:05.949 STEP: Waiting for kueue-controller-manager resource version to change from 146345 @ 02/25/26 13:06:06.135 STEP: Checking if deployment replicas: 3 matches amount of ready replicas: 1 @ 02/25/26 13:06:16.812 STEP: Checking if deployment replicas: 2 matches amount of ready replicas: 2 @ 02/25/26 13:06:27.181 STEP: Waiting for webhook configurations to exist @ 02/25/26 13:06:27.182 STEP: Waiting for webhook to handle requests successfully @ 02/25/26 13:06:27.531 I0225 13:06:27.888425 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-2mjfz I0225 13:06:28.060638 45539 e2e_operator_test.go:1494] Webhook success 1/3 I0225 13:06:31.420251 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-qjvct I0225 13:06:31.594107 45539 e2e_operator_test.go:1494] Webhook success 2/3 I0225 13:06:34.957574 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-4msd8 I0225 13:06:35.127844 45539 e2e_operator_test.go:1491] Webhook stable after 3 consecutive successes STEP: Deleting Job reclaim-job in namespace preemption-b-f29cj @ 02/25/26 13:06:35.128 STEP: Deleting Job borrowing-job in namespace preemption-a-vwf2r @ 02/25/26 13:06:35.456 STEP: Destroying LocalQueue preemption-b-f29cj/local-queue-b @ 02/25/26 13:06:35.78 STEP: Deleting namespace preemption-b-f29cj @ 02/25/26 13:06:36.28 STEP: Destroying ClusterQueue cluster-queue-vs4sk @ 02/25/26 13:06:36.607 STEP: Destroying LocalQueue preemption-a-vwf2r/local-queue-a @ 02/25/26 13:06:37.091 STEP: Deleting namespace preemption-a-vwf2r @ 02/25/26 13:06:37.579 STEP: Destroying ClusterQueue cluster-queue-2mwz7 @ 02/25/26 13:06:48.062 STEP: Destroying ResourceFlavor resource-flavor-s65fj @ 02/25/26 13:06:48.599 • [137.123 seconds] ------------------------------ LocalQueueDefaulting when labelPolicy=None and LocalQueue default in a managed namespace should label and admit Pod and Job [local-queue-default] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_local_queue_defaulting_test.go:78 STEP: Feching Kueue Instance @ 02/25/26 13:06:49.337 STEP: Updating Kueue config @ 02/25/26 13:06:49.844 STEP: Skipping deployment update wait - config unchanged @ 02/25/26 13:06:50.024 STEP: Checking if deployment replicas: 2 matches amount of ready replicas: 2 @ 02/25/26 13:06:50.201 STEP: Waiting for webhook configurations to exist @ 02/25/26 13:06:50.201 STEP: Waiting for webhook to handle requests successfully @ 02/25/26 13:06:50.558 I0225 13:06:50.951922 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-s6nlh I0225 13:06:51.143544 45539 e2e_operator_test.go:1494] Webhook success 1/3 I0225 13:06:54.515130 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-46nkv I0225 13:06:54.689811 45539 e2e_operator_test.go:1494] Webhook success 2/3 I0225 13:06:58.066614 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-4kwgg I0225 13:06:58.236788 45539 e2e_operator_test.go:1491] Webhook stable after 3 consecutive successes STEP: Created namespace e2e-kueue-wjz5r @ 02/25/26 13:06:58.764 STEP: Creating LocalQueue default in namespace e2e-kueue-wjz5r @ 02/25/26 13:06:58.765 STEP: creating job without queue name @ 02/25/26 13:06:58.936 STEP: namespace with labels: e2e-kueue-wjz5r @ 02/25/26 13:06:58.937 STEP: verifying workload created in namespace e2e-kueue-wjz5r and uid 287fb217-58bc-4152-a5e9-5e94f794c885 @ 02/25/26 13:06:59.139 STEP: creating pod without queue name @ 02/25/26 13:06:59.49 STEP: verifying workload created in namespace e2e-kueue-wjz5r and uid cf19b91b-52ed-4ab5-9547-6fa92d483e8a @ 02/25/26 13:06:59.682 STEP: Destroying LocalQueue e2e-kueue-wjz5r/default @ 02/25/26 13:07:00.035 STEP: Deleting namespace e2e-kueue-wjz5r @ 02/25/26 13:07:00.537 • [52.393 seconds] ------------------------------ LocalQueueDefaulting when labelPolicy=None and LocalQueue default in a managed namespace should label and admit LeaderWorkerSet [local-queue-default] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_local_queue_defaulting_test.go:95 STEP: Created namespace e2e-kueue-hjjv6 @ 02/25/26 13:07:41.716 STEP: Creating LocalQueue default in namespace e2e-kueue-hjjv6 @ 02/25/26 13:07:41.716 STEP: creating LeaderWorkerSet without queue name @ 02/25/26 13:07:41.876 STEP: verifying LeaderWorkerSet has queue label added @ 02/25/26 13:07:42.072 STEP: verifying workload is created for LeaderWorkerSet @ 02/25/26 13:07:42.239 STEP: verifying workload created in namespace e2e-kueue-hjjv6 and uid 54e8a871-e45a-457a-95e4-ac21c411668a @ 02/25/26 13:07:42.239 STEP: Destroying Object test-lws-bqzsl @ 02/25/26 13:07:42.775 STEP: Destroying LocalQueue e2e-kueue-hjjv6/default @ 02/25/26 13:07:43.282 STEP: Deleting namespace e2e-kueue-hjjv6 @ 02/25/26 13:07:43.806 • [43.764 seconds] ------------------------------ LocalQueueDefaulting when labelPolicy=None and LocalQueue default in a managed namespace should allow other local queues in same namespace without interfering [local-queue-default] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_local_queue_defaulting_test.go:119 STEP: Created namespace e2e-kueue-d4jgg @ 02/25/26 13:08:25.493 STEP: Creating LocalQueue default in namespace e2e-kueue-d4jgg @ 02/25/26 13:08:25.493 STEP: creating job without queue name @ 02/25/26 13:08:25.674 STEP: verifying workload created in namespace e2e-kueue-d4jgg and uid a62180e0-d3f9-428d-adf2-a603700cd63f @ 02/25/26 13:08:25.858 STEP: creating pod without queue name @ 02/25/26 13:08:26.222 STEP: verifying workload created in namespace e2e-kueue-d4jgg and uid 0be77f04-f14c-44eb-8f65-1292cead8402 @ 02/25/26 13:08:26.421 STEP: Creating LocalQueue test-queue-2 in namespace e2e-kueue-d4jgg @ 02/25/26 13:08:26.794 STEP: creating job in other local queue @ 02/25/26 13:08:27.461 STEP: verifying workload created in namespace e2e-kueue-d4jgg and uid 0be77f04-f14c-44eb-8f65-1292cead8402 @ 02/25/26 13:08:27.632 STEP: creating pod in other local queue @ 02/25/26 13:08:27.959 STEP: verifying workload created in namespace e2e-kueue-d4jgg and uid bf3c8c8c-e349-485b-8693-55fefede0bf2 @ 02/25/26 13:08:28.155 STEP: Destroying LocalQueue e2e-kueue-d4jgg/test-queue-2 @ 02/25/26 13:08:28.513 STEP: Destroying LocalQueue e2e-kueue-d4jgg/default @ 02/25/26 13:08:29.045 STEP: Deleting namespace e2e-kueue-d4jgg @ 02/25/26 13:08:29.617 • [46.027 seconds] ------------------------------ LocalQueueDefaulting when labelPolicy=None and LocalQueue default in a managed namespace should allow to label pod and job with default localqueue after they're created [local-queue-default] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_local_queue_defaulting_test.go:161 STEP: Created namespace e2e-kueue-774vf @ 02/25/26 13:09:11.523 STEP: Creating LocalQueue default in namespace e2e-kueue-774vf @ 02/25/26 13:09:11.523 STEP: Creating a new job without localQueue @ 02/25/26 13:09:11.71 STEP: Creating a new pod without localQueue @ 02/25/26 13:09:12.241 STEP: Creating localQueue Default @ 02/25/26 13:09:12.622 STEP: Creating LocalQueue default in namespace e2e-kueue-774vf @ 02/25/26 13:09:12.623 STEP: verifying workload created in namespace e2e-kueue-774vf and uid 01a9cfbf-fbab-4da3-a32d-fa748be590ac @ 02/25/26 13:09:13.135 STEP: Checking that Job and Pod were not automatically labeled @ 02/25/26 13:09:13.487 STEP: Adding localQueue Default label to Pod scheduled: test-pod-kmxwb @ 02/25/26 13:09:13.824 STEP: verifying workload created in namespace e2e-kueue-774vf and uid 80d8aa8a-9588-44fe-bb67-6825bdf870a2 @ 02/25/26 13:09:14.341 STEP: Adding localQueue Default label to Job suspended: test-job-lcd74 @ 02/25/26 13:09:14.689 STEP: verifying workload created in namespace e2e-kueue-774vf and uid 20c2bda0-af47-402c-a521-bcffcf0e723a @ 02/25/26 13:09:15.188 STEP: Destroying LocalQueue e2e-kueue-774vf/default @ 02/25/26 13:09:15.526 STEP: Deleting namespace e2e-kueue-774vf @ 02/25/26 13:09:16.036 STEP: deleting cluster queue @ 02/25/26 13:09:57.693 STEP: Destroying ClusterQueue test-clusterqueue @ 02/25/26 13:09:57.693 STEP: deleting resource flavor @ 02/25/26 13:10:03.424 STEP: Destroying ResourceFlavor default @ 02/25/26 13:10:03.424 STEP: Feching Kueue Instance @ 02/25/26 13:10:03.942 STEP: Updating Kueue config @ 02/25/26 13:10:04.455 STEP: Skipping deployment update wait - config unchanged @ 02/25/26 13:10:04.645 STEP: Checking if deployment replicas: 2 matches amount of ready replicas: 2 @ 02/25/26 13:10:04.982 STEP: Waiting for webhook configurations to exist @ 02/25/26 13:10:04.982 STEP: Waiting for webhook to handle requests successfully @ 02/25/26 13:10:05.335 I0225 13:10:05.723986 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-zjvqt I0225 13:10:05.888899 45539 e2e_operator_test.go:1494] Webhook success 1/3 I0225 13:10:09.259947 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-dbxnl I0225 13:10:09.445304 45539 e2e_operator_test.go:1494] Webhook success 2/3 I0225 13:10:12.805901 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-v95mc I0225 13:10:12.981968 45539 e2e_operator_test.go:1491] Webhook stable after 3 consecutive successes • [61.642 seconds] ------------------------------ LocalQueueDefaulting when labelPolicy is not defined and default LocalQueue is in a managed namespace should label and admit Job [local-queue-default] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_local_queue_defaulting_test.go:248 STEP: Created namespace e2e-kueue-vqtp5 @ 02/25/26 13:10:13.52 STEP: Creating LocalQueue default in namespace e2e-kueue-vqtp5 @ 02/25/26 13:10:13.52 STEP: Creating job without queue name @ 02/25/26 13:10:13.691 STEP: verifying workload created in namespace e2e-kueue-vqtp5 and uid b77fbe7b-d272-4209-bf17-8c57df1e0a77 @ 02/25/26 13:10:14.048 STEP: Destroying LocalQueue e2e-kueue-vqtp5/default @ 02/25/26 13:10:14.4 STEP: Deleting namespace e2e-kueue-vqtp5 @ 02/25/26 13:10:14.907 STEP: deleting cluster queue @ 02/25/26 13:10:55.925 STEP: Destroying ClusterQueue test-clusterqueue @ 02/25/26 13:10:55.925 STEP: deleting resource flavor @ 02/25/26 13:11:01.627 STEP: Destroying ResourceFlavor default @ 02/25/26 13:11:01.627 • [49.187 seconds] ------------------------------ LocalQueueDefaulting when labelPolicy is not defined and default LocalQueue is in a managed namespace should label and admit JobSet [local-queue-default] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_local_queue_defaulting_test.go:260 STEP: Created namespace e2e-kueue-qk8fn @ 02/25/26 13:11:02.698 STEP: Creating LocalQueue default in namespace e2e-kueue-qk8fn @ 02/25/26 13:11:02.698 STEP: Creating JobSet without queue name @ 02/25/26 13:11:02.874 STEP: verifying jobset did start in labeled namespace @ 02/25/26 13:11:03.074 STEP: verifying workload created in namespace e2e-kueue-qk8fn and uid dba871d5-11ae-4d9a-a915-538ffe673734 @ 02/25/26 13:11:03.253 STEP: Destroying Object test-jobset-6h6fs @ 02/25/26 13:11:03.606 STEP: Destroying LocalQueue e2e-kueue-qk8fn/default @ 02/25/26 13:11:04.133 STEP: Deleting namespace e2e-kueue-qk8fn @ 02/25/26 13:11:04.666 STEP: deleting cluster queue @ 02/25/26 13:11:15.192 STEP: Destroying ClusterQueue test-clusterqueue @ 02/25/26 13:11:15.192 STEP: deleting resource flavor @ 02/25/26 13:11:15.724 STEP: Destroying ResourceFlavor default @ 02/25/26 13:11:15.724 • [14.075 seconds] ------------------------------ LocalQueueDefaulting when labelPolicy is not defined and default LocalQueue is in an unmanaged namespace should label and admit Pod [local-queue-default] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_local_queue_defaulting_test.go:292 STEP: Created namespace e2e-kueue-krz86 @ 02/25/26 13:11:16.796 STEP: Creating LocalQueue default in namespace e2e-kueue-krz86 @ 02/25/26 13:11:16.796 STEP: creating pod without queue name @ 02/25/26 13:11:16.994 STEP: Destroying LocalQueue e2e-kueue-krz86/default @ 02/25/26 13:11:27.541 STEP: Deleting namespace e2e-kueue-krz86 @ 02/25/26 13:11:28.058 STEP: deleting cluster queue @ 02/25/26 13:12:09.095 STEP: Destroying ClusterQueue test-clusterqueue @ 02/25/26 13:12:09.095 STEP: deleting resource flavor @ 02/25/26 13:12:09.656 STEP: Destroying ResourceFlavor default @ 02/25/26 13:12:09.656 • [53.965 seconds] ------------------------------ VisibilityOnDemand when kueue.openshift.io/allow-nominal-concurrency-shares-update annotation is set to true should allow modification of the nominal concurrency shares to 0 [visibility-on-demand] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_visibility_on_demand_test.go:103 STEP: Creating LocalQueue test-queue in namespace e2e-kueue-visibility-on-demand-xdmsz @ 02/25/26 13:12:10.578 STEP: Modifying the PriorityLevelConfiguration with nominal concurrency shares set to 0 @ 02/25/26 13:12:11.143 STEP: Ensure the value of nominal concurrency shares is changed to 0 @ 02/25/26 13:12:11.528 STEP: Try to access the pending workload @ 02/25/26 13:12:11.698 • [178.376 seconds] ------------------------------ VisibilityOnDemand when kueue.openshift.io/allow-nominal-concurrency-shares-update annotation is set to true should allow modification of the nominal concurrency shares to 5 [visibility-on-demand] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_visibility_on_demand_test.go:117 STEP: Modifying the PriorityLevelConfiguration with nominal concurrency shares set to 5 @ 02/25/26 13:15:08.574 STEP: Try to access the pending workload @ 02/25/26 13:15:08.971 STEP: Check is able to access the pending workload @ 02/25/26 13:15:09.166 • [0.765 seconds] ------------------------------ VisibilityOnDemand when kueue.openshift.io/allow-nominal-concurrency-shares-update annotation is set to true should not allow modification of the nominal concurrency shares to 1 [visibility-on-demand] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_visibility_on_demand_test.go:130 STEP: Modifying the PriorityLevelConfiguration with nominal concurrency shares set to 1 @ 02/25/26 13:15:09.34 STEP: Wait to verify the value of nominal concurrency shares is changed back to the default @ 02/25/26 13:15:09.731 • [10.718 seconds] ------------------------------ VisibilityOnDemand when kueue.openshift.io/allow-nominal-concurrency-shares-update annotation is set to true should not allow modification of the nominal concurrency shares to 6 [visibility-on-demand] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_visibility_on_demand_test.go:142 STEP: Modifying the PriorityLevelConfiguration with nominal concurrency shares set to 6 @ 02/25/26 13:15:20.058 STEP: Wait to verify the value of nominal concurrency shares is changed back to the default @ 02/25/26 13:15:20.389 • [0.498 seconds] ------------------------------ VisibilityOnDemand when kueue.openshift.io/allow-nominal-concurrency-shares-update annotation is set to true should not allow modification of the nominal concurrency shares if it's not admin [visibility-on-demand] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_visibility_on_demand_test.go:155 STEP: Creating Kubernetes client with impersonation for non-admin service account @ 02/25/26 13:15:20.557 STEP: Attempting to modify PriorityLevelConfiguration with nominal concurrency shares set to 4 @ 02/25/26 13:15:20.567 STEP: Destroying LocalQueue e2e-kueue-visibility-on-demand-xdmsz/test-queue @ 02/25/26 13:15:20.722 STEP: Deleting namespace e2e-kueue-visibility-on-demand-xdmsz @ 02/25/26 13:15:21.208 STEP: Destroying ClusterQueue test-clusterqueue @ 02/25/26 13:15:21.553 STEP: Destroying ResourceFlavor default @ 02/25/26 13:15:22.044 • [1.984 seconds] ------------------------------ VisibilityOnDemand when PendingWorkloads list should be checked for a ClusterQueue and LocalQueue Should allow admin to access ClusterQueues, deny user access, and order pending workloads by priority [visibility-on-demand] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_visibility_on_demand_test.go:177 STEP: Creating Cluster Resources @ 02/25/26 13:15:22.543 STEP: Creating Namespaces and LocalQueues @ 02/25/26 13:15:23.049 STEP: Creating LocalQueue local-queue-a in namespace namespace-a-vvlwd @ 02/25/26 13:15:23.217 STEP: Creating LocalQueue local-queue-b in namespace namespace-b-l8ncn @ 02/25/26 13:15:23.571 STEP: Creating Priority Classes @ 02/25/26 13:15:23.741 STEP: Creating RBAC kueue-batch-admin-role for Visibility API @ 02/25/26 13:15:24.258 STEP: Creating RBAC kueue-batch-user-role for Visibility API @ 02/25/26 13:15:24.597 STEP: Creating custom visibility client for ClusterQueue user @ 02/25/26 13:15:24.933 STEP: Creating testing data @ 02/25/26 13:15:24.944 STEP: verifying workload created in namespace namespace-a-vvlwd and uid ce447f88-dfd4-47db-8345-12e40666528a @ 02/25/26 13:15:25.142 STEP: verifying workload created in namespace namespace-b-l8ncn and uid 4d06d72b-2ac7-44ba-84c1-ff4ef96786ea @ 02/25/26 13:15:57.288 STEP: Checking the pending workloads for cluster queue cluster-queue-g7xv7 @ 02/25/26 13:15:57.839 STEP: Verifying number of pending workloads and their priority ordering @ 02/25/26 13:16:03.235 STEP: Checking the pending workloads for cluster queue cluster-queue-4rrqh @ 02/25/26 13:16:03.236 STEP: Verifying number of pending workloads @ 02/25/26 13:16:03.581 STEP: All workloads should have been created @ 02/25/26 13:16:03.581 STEP: verifying workload created in namespace namespace-a-vvlwd and uid a1aa5fdf-7fa6-41c2-a2ce-0cb4e9d71c78 @ 02/25/26 13:16:03.581 STEP: verifying workload created in namespace namespace-a-vvlwd and uid 77e340c4-837c-494f-8720-69f31a88d4ba @ 02/25/26 13:16:44.593 STEP: verifying workload created in namespace namespace-a-vvlwd and uid 8fbfa459-f0b4-4c59-bdbd-19f229971559 @ 02/25/26 13:16:44.956 STEP: verifying workload created in namespace namespace-b-l8ncn and uid 41241708-16b6-40bf-ba42-11c56ce5c44b @ 02/25/26 13:17:26.014 STEP: Verifying pending workloads lists are empty @ 02/25/26 13:17:26.392 STEP: Verifying a unauthorized user cannot access the pending workloads @ 02/25/26 13:17:33.284 STEP: Verifying user with kueue-batch-user-role cannot access ClusterQueue pending workloads @ 02/25/26 13:17:33.464 STEP: Deleting Job job-high-b in namespace namespace-b-l8ncn @ 02/25/26 13:17:33.643 STEP: Deleting Job job-blocker-b in namespace namespace-b-l8ncn @ 02/25/26 13:17:34.003 STEP: Deleting Job job-low-a in namespace namespace-a-vvlwd @ 02/25/26 13:17:34.356 STEP: Deleting Job job-medium-a in namespace namespace-a-vvlwd @ 02/25/26 13:17:34.71 STEP: Deleting Job job-high-a in namespace namespace-a-vvlwd @ 02/25/26 13:17:35.07 STEP: Deleting Job job-blocker in namespace namespace-a-vvlwd @ 02/25/26 13:17:35.437 STEP: Deleting ClusterRoleBinding kueue-cluster-role-binding-tq4h6 @ 02/25/26 13:17:35.796 STEP: Deleting ClusterRoleBinding kueue-cluster-role-binding-ncn9p @ 02/25/26 13:17:36.136 STEP: Deleting PriorityClass priority-class-4524q @ 02/25/26 13:17:36.49 STEP: Deleting PriorityClass priority-class-4zwms @ 02/25/26 13:17:36.87 STEP: Deleting PriorityClass priority-class-qktz7 @ 02/25/26 13:17:37.209 STEP: Destroying LocalQueue namespace-b-l8ncn/local-queue-b @ 02/25/26 13:17:37.577 STEP: Deleting namespace namespace-b-l8ncn @ 02/25/26 13:17:38.129 STEP: Destroying LocalQueue namespace-a-vvlwd/local-queue-a @ 02/25/26 13:17:38.489 STEP: Deleting namespace namespace-a-vvlwd @ 02/25/26 13:17:39.03 STEP: Destroying ClusterQueue cluster-queue-4rrqh @ 02/25/26 13:18:09.914 STEP: Destroying ClusterQueue cluster-queue-g7xv7 @ 02/25/26 13:18:10.467 STEP: Destroying ResourceFlavor resource-flavor-jx4ls @ 02/25/26 13:18:11.043 • [169.025 seconds] ------------------------------ VisibilityOnDemand when PendingWorkloads list should be checked for a ClusterQueue and LocalQueue Should allow access to LocalQueues in bound namespaces and deny access to unbound namespaces [visibility-on-demand] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_visibility_on_demand_test.go:373 STEP: Creating Cluster Resources @ 02/25/26 13:18:11.561 STEP: Creating namespace-a and LocalQueue-a @ 02/25/26 13:18:11.934 STEP: Creating LocalQueue local-queue-a in namespace namespace-a-2qprf @ 02/25/26 13:18:12.107 STEP: Creating namespace-b and LocalQueue-b @ 02/25/26 13:18:12.288 STEP: Creating LocalQueue local-queue-b in namespace namespace-b-5bdsb @ 02/25/26 13:18:12.464 STEP: Creating RBAC kueue-batch-admin-role for Visibility API @ 02/25/26 13:18:12.636 STEP: Creating custom visibility client for ClusterQueue user @ 02/25/26 13:18:13.339 STEP: Creating Priority Classes @ 02/25/26 13:18:13.352 STEP: Creating testing data @ 02/25/26 13:18:13.696 STEP: verifying workload created in namespace namespace-a-2qprf and uid 46655433-abab-4c9e-ac24-bf0d1567cbe1 @ 02/25/26 13:18:13.888 STEP: verifying workload created in namespace namespace-b-5bdsb and uid 752fb18d-c73a-4953-929a-8308484f2a61 @ 02/25/26 13:18:14.587 STEP: Waiting for job-high-a workload to be created @ 02/25/26 13:19:05.983 I0225 13:19:06.154954 45539 warnings.go:110] "Warning: This version is deprecated. Use v1beta2 instead." STEP: Waiting for job-low-b workload to be created @ 02/25/26 13:19:06.156 I0225 13:19:06.323449 45539 warnings.go:110] "Warning: This version is deprecated. Use v1beta2 instead." STEP: Checking the pending workloads for local queue local-queue-a in namespace namespace-a-2qprf @ 02/25/26 13:19:06.323 STEP: Verifying pending workloads for LocalQueue A @ 02/25/26 13:19:06.675 STEP: Checking the pending workloads for local queue local-queue-b in namespace namespace-b-5bdsb @ 02/25/26 13:19:06.675 STEP: Verifying pending workloads for LocalQueue B @ 02/25/26 13:19:07.011 STEP: Check if not allowed users does not have access to LocalQueueA @ 02/25/26 13:19:07.011 STEP: Check if not allowed users does not have access to LocalQueueB @ 02/25/26 13:19:07.171 STEP: All workloads should have been created @ 02/25/26 13:19:07.326 STEP: verifying workload created in namespace namespace-a-2qprf and uid 639b3c35-7e3b-4856-9e3b-aa1ef89794a5 @ 02/25/26 13:19:07.326 STEP: verifying workload created in namespace namespace-b-5bdsb and uid d9f2f520-9cd5-476e-b031-ac37e3b3a1cd @ 02/25/26 13:19:48.326 STEP: Verifying pending workloads lists are empty @ 02/25/26 13:19:48.68 STEP: Deleting Job job-low-b in namespace namespace-b-5bdsb @ 02/25/26 13:19:49.011 STEP: Deleting Job job-blocker-b in namespace namespace-b-5bdsb @ 02/25/26 13:19:49.358 STEP: Deleting Job job-high-a in namespace namespace-a-2qprf @ 02/25/26 13:19:49.715 STEP: Deleting Job job-blocker in namespace namespace-a-2qprf @ 02/25/26 13:19:50.047 STEP: Deleting PriorityClass priority-class-hmcnl @ 02/25/26 13:19:50.391 STEP: Deleting PriorityClass priority-class-6972r @ 02/25/26 13:19:50.735 STEP: Deleting RoleBinding kueue-role-binding-8lsh2 in namespace namespace-b-5bdsb @ 02/25/26 13:19:51.063 STEP: Deleting RoleBinding kueue-role-binding-lp9wn in namespace namespace-a-2qprf @ 02/25/26 13:19:51.412 STEP: Destroying LocalQueue namespace-b-5bdsb/local-queue-b @ 02/25/26 13:19:51.76 STEP: Deleting namespace namespace-b-5bdsb @ 02/25/26 13:19:52.283 STEP: Destroying LocalQueue namespace-a-2qprf/local-queue-a @ 02/25/26 13:20:23.12 STEP: Deleting namespace namespace-a-2qprf @ 02/25/26 13:20:23.628 STEP: Destroying ClusterQueue cluster-queue-xkbwj @ 02/25/26 13:20:23.968 STEP: Destroying ResourceFlavor resource-flavor-9mtvd @ 02/25/26 13:20:24.476 • [133.427 seconds] ------------------------------ VisibilityOnDemand when PendingWorkloads Endpoints should be checked Should use the correct PriorityLevelConfiguration and FlowSchema for ClusterQueue and LocalQueue [visibility-on-demand] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_visibility_on_demand_test.go:583 STEP: Creating Cluster Resources @ 02/25/26 13:20:24.978 STEP: Creating Namespaces and LocalQueues @ 02/25/26 13:20:25.323 STEP: Creating LocalQueue local-queue in namespace namespace-r9mbk @ 02/25/26 13:20:25.495 STEP: Creating RBAC kueue-batch-admin-role for Visibility API @ 02/25/26 13:20:25.667 STEP: Creating custom visibility client for ClusterQueue user @ 02/25/26 13:20:25.845 STEP: Getting the PriorityLevelConfiguration and FlowSchema @ 02/25/26 13:20:25.853 STEP: Setting up HTTP header capture @ 02/25/26 13:20:26.201 STEP: Getting the pending workloads for the ClusterQueue API response @ 02/25/26 13:20:26.201 STEP: Extracting response headers from API call @ 02/25/26 13:20:26.364 STEP: Checking that response headers match expected PriorityLevelConfiguration and FlowSchema UIDs for ClusterQueue @ 02/25/26 13:20:26.364 STEP: Getting the pending workloads for the LocalQueue API response @ 02/25/26 13:20:26.365 STEP: Extracting response headers from API call @ 02/25/26 13:20:26.532 STEP: Checking that response headers match expected PriorityLevelConfiguration and FlowSchema UIDs for LocalQueue @ 02/25/26 13:20:26.532 STEP: Deleting ClusterRoleBinding kueue-cluster-role-binding-wsgwb @ 02/25/26 13:20:26.532 STEP: Destroying LocalQueue namespace-r9mbk/local-queue @ 02/25/26 13:20:26.875 STEP: Deleting namespace namespace-r9mbk @ 02/25/26 13:20:27.36 STEP: Destroying ClusterQueue cluster-queue-ncbh6 @ 02/25/26 13:20:27.704 STEP: Destroying ResourceFlavor resource-flavor-9z2wd @ 02/25/26 13:20:28.211 • [3.726 seconds] ------------------------------ VisibilityOnDemand when PendingWorkloads list should be checked for LWS workloads Should show pending LWS workloads ordered by priority and admit them sequentially based on resource availability [visibility-on-demand] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_visibility_on_demand_test.go:685 STEP: Creating Cluster Resources @ 02/25/26 13:20:28.704 STEP: Creating Namespaces and LocalQueues @ 02/25/26 13:20:29.04 STEP: Creating LocalQueue local-queue-a in namespace namespace-a-n6ztd @ 02/25/26 13:20:29.199 STEP: Creating LocalQueue local-queue-b in namespace namespace-b-vrw4w @ 02/25/26 13:20:29.531 STEP: Creating Priority Classes @ 02/25/26 13:20:29.703 STEP: Creating RBAC kueue-batch-admin-role for Visibility API @ 02/25/26 13:20:30.2 STEP: Creating RBAC kueue-batch-user-role for Visibility API @ 02/25/26 13:20:30.546 STEP: Creating custom visibility client for ClusterQueue access @ 02/25/26 13:20:31.228 STEP: Creating custom visibility client for LocalQueue A access @ 02/25/26 13:20:31.235 STEP: Creating custom visibility client for LocalQueue B access @ 02/25/26 13:20:31.239 STEP: Creating blocker LWS in namespace A @ 02/25/26 13:20:31.244 STEP: verifying workload created in namespace namespace-a-n6ztd and uid 5879ee28-0685-47f1-8bb2-56317a580922 @ 02/25/26 13:20:31.436 STEP: Waiting for blocker LWS pods to be created @ 02/25/26 13:20:31.928 STEP: Creating pending LWS workloads after blocker pods are ready @ 02/25/26 13:20:32.253 STEP: Verifying all pending workloads are created @ 02/25/26 13:20:32.796 STEP: verifying workload is created in namespace namespace-b-vrw4w @ 02/25/26 13:20:32.796 STEP: verifying workload is created in namespace namespace-a-n6ztd @ 02/25/26 13:20:32.964 STEP: verifying workload is created in namespace namespace-a-n6ztd @ 02/25/26 13:20:33.165 STEP: Checking the pending workloads for cluster queue cluster-queue-gmcgr @ 02/25/26 13:20:33.334 STEP: Verifying number of pending workloads and their priority ordering for ClusterQueue @ 02/25/26 13:20:33.504 STEP: Checking the pending workloads for local queue local-queue-a in namespace namespace-a-n6ztd @ 02/25/26 13:20:33.504 STEP: Verifying pending workloads for LocalQueue A @ 02/25/26 13:20:33.668 STEP: Checking the pending workloads for local queue local-queue-b in namespace namespace-b-vrw4w @ 02/25/26 13:20:33.668 STEP: Verifying pending workloads for LocalQueue B @ 02/25/26 13:20:33.832 STEP: Deleting blocker LWS to free resources @ 02/25/26 13:20:33.832 STEP: Waiting for blocker LWS deletion and resource release @ 02/25/26 13:20:34.003 STEP: Waiting for blocker LWS pods to be deleted @ 02/25/26 13:20:34.164 STEP: Verifying high priority workload is admitted @ 02/25/26 13:21:38.11 STEP: verifying workload created in namespace namespace-b-vrw4w and uid 764337ef-c1f7-41db-8619-524f8f6b2c84 @ 02/25/26 13:21:38.112 STEP: Verifying high priority LWS pods are running after admission @ 02/25/26 13:21:38.63 STEP: Verifying medium priority workload is admitted @ 02/25/26 13:21:49.137 STEP: verifying workload created in namespace namespace-a-n6ztd and uid a0484ae2-adc6-4b09-858f-952d2b636302 @ 02/25/26 13:21:49.137 STEP: Verifying medium priority LWS pods are running after admission @ 02/25/26 13:21:49.625 STEP: Verifying low priority LWS is still pending @ 02/25/26 13:21:49.797 STEP: Deleting medium priority LWS to free resources for low priority @ 02/25/26 13:21:49.97 STEP: Waiting for medium priority LWS deletion @ 02/25/26 13:21:50.14 STEP: Waiting for medium priority LWS pods to be deleted @ 02/25/26 13:21:50.296 STEP: Verifying low priority workload is admitted @ 02/25/26 13:22:53.296 STEP: verifying workload created in namespace namespace-a-n6ztd and uid 81c66fad-9c12-4dbf-903f-c1b57eca44d3 @ 02/25/26 13:22:53.296 STEP: Verifying low priority LWS pods are running after admission @ 02/25/26 13:22:53.788 STEP: Verifying pending workloads list is empty @ 02/25/26 13:23:04.274 STEP: Deleting RoleBinding kueue-role-binding-wh825 in namespace namespace-b-vrw4w @ 02/25/26 13:23:04.443 STEP: Deleting RoleBinding kueue-role-binding-ng2fj in namespace namespace-a-n6ztd @ 02/25/26 13:23:04.767 STEP: Deleting ClusterRoleBinding kueue-cluster-role-binding-5pdgx @ 02/25/26 13:23:05.159 STEP: Deleting PriorityClass priority-class-fxfzz @ 02/25/26 13:23:05.498 STEP: Deleting PriorityClass priority-class-6rhnk @ 02/25/26 13:23:05.83 STEP: Deleting PriorityClass priority-class-nbhbr @ 02/25/26 13:23:06.15 STEP: Destroying LocalQueue namespace-b-vrw4w/local-queue-b @ 02/25/26 13:23:06.488 STEP: Deleting namespace namespace-b-vrw4w @ 02/25/26 13:23:06.999 STEP: Destroying LocalQueue namespace-a-n6ztd/local-queue-a @ 02/25/26 13:23:48.515 STEP: Deleting namespace namespace-a-n6ztd @ 02/25/26 13:23:48.998 STEP: Destroying ClusterQueue cluster-queue-gmcgr @ 02/25/26 13:24:30.59 STEP: Destroying ResourceFlavor resource-flavor-q5mgx @ 02/25/26 13:24:36.245 • [248.048 seconds] ------------------------------ VisibilityOnDemand when JobSet is suspended Should show on pending workloads list for local queue and cluster queue [visibility-on-demand] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_visibility_on_demand_test.go:1020 STEP: Creating Cluster Resources @ 02/25/26 13:24:36.744 STEP: Creating Namespaces and LocalQueues @ 02/25/26 13:24:37.089 STEP: Creating LocalQueue local-queue in namespace test-namespace-86dfc @ 02/25/26 13:24:37.254 STEP: Creating RBAC kueue-batch-admin-role for Visibility API @ 02/25/26 13:24:37.429 STEP: Creating RBAC kueue-batch-user-role for Visibility API @ 02/25/26 13:24:37.765 STEP: Creating custom visibility client for ClusterQueue access @ 02/25/26 13:24:38.101 STEP: Creating custom visibility client for LocalQueue access @ 02/25/26 13:24:38.117 STEP: Creating custom jobset @ 02/25/26 13:24:38.121 STEP: Verifying all pending workloads are created @ 02/25/26 13:24:38.315 STEP: verifying workload is created in namespace test-namespace-86dfc @ 02/25/26 13:24:38.315 STEP: Checking the pending workloads for local queue local-queue in namespace test-namespace-86dfc @ 02/25/26 13:24:38.476 STEP: Verifying pending workloads for LocalQueue @ 02/25/26 13:24:38.649 STEP: Checking the pending workloads for cluster queue cluster-queue-gtrqc @ 02/25/26 13:24:38.649 STEP: Verifying pending workloads for ClusterQueue @ 02/25/26 13:24:38.812 STEP: Destroying Object test-jobset-nzbhz @ 02/25/26 13:24:38.813 STEP: Deleting RoleBinding kueue-role-binding-kdscw in namespace test-namespace-86dfc @ 02/25/26 13:24:39.309 STEP: Deleting ClusterRoleBinding kueue-cluster-role-binding-gh7mt @ 02/25/26 13:24:39.653 STEP: Destroying LocalQueue test-namespace-86dfc/local-queue @ 02/25/26 13:24:39.982 STEP: Deleting namespace test-namespace-86dfc @ 02/25/26 13:24:40.49 STEP: Destroying ClusterQueue cluster-queue-gtrqc @ 02/25/26 13:24:40.825 STEP: Destroying ResourceFlavor resource-flavor-gt9jb @ 02/25/26 13:24:41.318 • [5.062 seconds] ------------------------------ Scheduling Gate when workloads are submitted to a non-existent LocalQueue should verify Kueue is configured with only Deployment, StatefulSet, and LeaderWorkerSet integrations [scheduling-gate] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_scheduling_gate_test.go:130 STEP: Updating Kueue configuration to only include Deployment, StatefulSet, and LeaderWorkerSet @ 02/25/26 13:24:41.807 STEP: Feching Kueue Instance @ 02/25/26 13:24:41.807 STEP: Updating Kueue config @ 02/25/26 13:24:42.347 STEP: Waiting for kueue-controller-manager resource version to change from 151691 @ 02/25/26 13:24:42.527 STEP: Checking if deployment replicas: 3 matches amount of ready replicas: 1 @ 02/25/26 13:24:53.251 STEP: Checking if deployment replicas: 2 matches amount of ready replicas: 2 @ 02/25/26 13:25:03.574 STEP: Waiting for webhook configurations to exist @ 02/25/26 13:25:03.574 STEP: Waiting for webhook to handle requests successfully @ 02/25/26 13:25:03.932 I0225 13:25:04.309021 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-w6rtk I0225 13:25:04.486079 45539 e2e_operator_test.go:1494] Webhook success 1/3 I0225 13:25:07.841393 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-6xdpp I0225 13:25:08.026973 45539 e2e_operator_test.go:1494] Webhook success 2/3 I0225 13:25:11.392830 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-bzczt I0225 13:25:11.573969 45539 e2e_operator_test.go:1491] Webhook stable after 3 consecutive successes STEP: Creating namespace scheduling-gate-test with managed label @ 02/25/26 13:25:11.574 I0225 13:25:12.099468 45539 e2e_scheduling_gate_test.go:92] Setup complete - no LocalQueue created in namespace scheduling-gate-test STEP: Verifying kueue-manager-config ConfigMap contains only expected integrations @ 02/25/26 13:25:12.099 • [30.476 seconds] ------------------------------ Scheduling Gate when workloads are submitted to a non-existent LocalQueue should keep Deployment pods scheduling gated when submitted to non-existent queue [scheduling-gate] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_scheduling_gate_test.go:150 STEP: Creating deployment with queue label pointing to non-existent queue @ 02/25/26 13:25:12.281 STEP: Waiting for deployment pods to be created @ 02/25/26 13:25:12.484 STEP: Verifying deployment pods have scheduling gates @ 02/25/26 13:25:12.654 STEP: Verifying pods remain scheduling gated (not scheduled) @ 02/25/26 13:25:12.826 I0225 13:25:12.826083 45539 e2e_scheduling_gate_test.go:197] Deployment pod test-deploy-2dpx4-6578975d-bz2d5 has scheduling gate as expected STEP: Destroying Object test-deploy-2dpx4 @ 02/25/26 13:25:42.827 • [31.060 seconds] ------------------------------ Scheduling Gate when workloads are submitted to a non-existent LocalQueue should keep StatefulSet pods scheduling gated when submitted to non-existent queue [scheduling-gate] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_scheduling_gate_test.go:217 STEP: Creating statefulset with queue label pointing to non-existent queue @ 02/25/26 13:25:43.34 STEP: Waiting for statefulset pods to be created @ 02/25/26 13:25:43.541 STEP: Verifying statefulset pods have scheduling gates @ 02/25/26 13:25:43.715 STEP: Verifying pods remain scheduling gated (not scheduled) @ 02/25/26 13:25:43.893 I0225 13:25:43.893842 45539 e2e_scheduling_gate_test.go:264] StatefulSet pod test-ss-2tmgg-0 has scheduling gate as expected STEP: Destroying Object test-ss-2tmgg @ 02/25/26 13:26:13.893 • [31.077 seconds] ------------------------------ Scheduling Gate when workloads are submitted to a non-existent LocalQueue should keep LeaderWorkerSet pods scheduling gated when submitted to non-existent queue [scheduling-gate] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_scheduling_gate_test.go:284 STEP: Creating LeaderWorkerSet with queue label pointing to non-existent queue @ 02/25/26 13:26:14.415 STEP: Waiting for LeaderWorkerSet pods to be created @ 02/25/26 13:26:14.624 STEP: Verifying LeaderWorkerSet pods have scheduling gates @ 02/25/26 13:26:24.966 I0225 13:26:25.172468 45539 e2e_scheduling_gate_test.go:334] LeaderWorkerSet pod test-lws-8p225-0 has scheduling gate as expected I0225 13:26:25.343376 45539 e2e_scheduling_gate_test.go:334] LeaderWorkerSet pod test-lws-8p225-0-1 has scheduling gate as expected STEP: Verifying pods remain scheduling gated (not scheduled) @ 02/25/26 13:26:25.343 STEP: Destroying Object test-lws-8p225 @ 02/25/26 13:26:55.394 STEP: Cleaning up test resources @ 02/25/26 13:26:56.036 STEP: Deleting namespace scheduling-gate-test @ 02/25/26 13:26:56.036 STEP: Destroying ClusterQueue test-clusterqueue @ 02/25/26 13:26:56.409 STEP: Destroying ResourceFlavor default @ 02/25/26 13:26:56.998 STEP: Restoring default Kueue configuration @ 02/25/26 13:26:57.584 STEP: Feching Kueue Instance @ 02/25/26 13:26:57.584 STEP: Updating Kueue config @ 02/25/26 13:26:58.09 STEP: Waiting for kueue-controller-manager resource version to change from 156506 @ 02/25/26 13:26:58.279 STEP: Checking if deployment replicas: 3 matches amount of ready replicas: 1 @ 02/25/26 13:27:08.994 STEP: Checking if deployment replicas: 2 matches amount of ready replicas: 2 @ 02/25/26 13:27:19.324 STEP: Waiting for webhook configurations to exist @ 02/25/26 13:27:19.324 STEP: Waiting for webhook to handle requests successfully @ 02/25/26 13:27:19.652 I0225 13:27:20.008618 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-mllsq I0225 13:27:20.163129 45539 e2e_operator_test.go:1494] Webhook success 1/3 I0225 13:27:23.517816 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-89fbx I0225 13:27:23.689485 45539 e2e_operator_test.go:1494] Webhook success 2/3 I0225 13:27:27.051011 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-4n7t5 I0225 13:27:27.226105 45539 e2e_operator_test.go:1491] Webhook stable after 3 consecutive successes • [72.814 seconds] ------------------------------ Gangscheduling when Policy is ByWorkload and Admission is Sequential should apply all-or-nothing admission for workloads [gangscheduling] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_gangscheduling_test.go:81 STEP: Saving initial Kueue configuration @ 02/25/26 13:27:27.227 STEP: Configuring Kueue with gangScheduling: policy=ByWorkload, admission=Sequential @ 02/25/26 13:27:27.393 STEP: Feching Kueue Instance @ 02/25/26 13:27:27.393 STEP: Updating Kueue config @ 02/25/26 13:27:28.059 STEP: Waiting for kueue-controller-manager resource version to change from 157657 @ 02/25/26 13:27:28.231 STEP: Checking if deployment replicas: 3 matches amount of ready replicas: 1 @ 02/25/26 13:27:39.206 STEP: Checking if deployment replicas: 2 matches amount of ready replicas: 2 @ 02/25/26 13:27:49.531 STEP: Waiting for webhook configurations to exist @ 02/25/26 13:27:49.531 STEP: Waiting for webhook to handle requests successfully @ 02/25/26 13:27:49.887 I0225 13:27:50.266636 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-dqttg I0225 13:27:50.436774 45539 e2e_operator_test.go:1494] Webhook success 1/3 I0225 13:27:53.807239 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-jftjg I0225 13:27:53.983561 45539 e2e_operator_test.go:1494] Webhook success 2/3 I0225 13:27:57.367534 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-l8vm2 I0225 13:27:57.552744 45539 e2e_operator_test.go:1491] Webhook stable after 3 consecutive successes STEP: Waiting for Kueue configuration to be applied @ 02/25/26 13:27:57.552 STEP: Creating Cluster Resources @ 02/25/26 13:27:57.759 STEP: Creating Namespaces and LocalQueues @ 02/25/26 13:27:58.086 STEP: Creating LocalQueue local-queue in namespace gangscheduling-tftql @ 02/25/26 13:27:58.24 STEP: Admitting a Job that consumes partial quota @ 02/25/26 13:27:58.412 STEP: Verifying first job workload is created and admitted @ 02/25/26 13:27:58.582 STEP: verifying workload created in namespace gangscheduling-tftql and uid 80f6b159-c688-4a4c-bafe-72de42c59d22 @ 02/25/26 13:27:58.582 STEP: Creating a gang job (parallelism=2) that exceeds remaining quota @ 02/25/26 13:27:58.92 STEP: Verifying the gang job workload is created but NOT admitted (not enough quota for all pods) @ 02/25/26 13:27:59.124 STEP: verifying workload is created in namespace gangscheduling-tftql @ 02/25/26 13:27:59.124 STEP: Waiting for first job to complete and free up quota @ 02/25/26 13:28:04.352 STEP: Verifying the second gang job is now admitted after quota is freed (all pods admitted together) @ 02/25/26 13:28:35.01 STEP: verifying workload created in namespace gangscheduling-tftql and uid 233fa949-ae18-4d01-af01-636a7403b9df @ 02/25/26 13:28:35.011 I0225 13:28:35.493395 45539 warnings.go:110] "Warning: child pods are preserved by default when jobs are deleted; set propagationPolicy=Background to remove them or set propagationPolicy=Orphan to suppress this warning" I0225 13:28:35.655664 45539 warnings.go:110] "Warning: child pods are preserved by default when jobs are deleted; set propagationPolicy=Background to remove them or set propagationPolicy=Orphan to suppress this warning" STEP: Destroying LocalQueue gangscheduling-tftql/local-queue @ 02/25/26 13:28:35.655 STEP: Deleting namespace gangscheduling-tftql @ 02/25/26 13:28:36.17 STEP: Destroying ClusterQueue cluster-queue-2vt26 @ 02/25/26 13:29:07.187 STEP: Destroying ResourceFlavor resource-flavor-gvnvk @ 02/25/26 13:29:23.214 • [116.491 seconds] ------------------------------ Gangscheduling when Policy is ByWorkload and Admission is Sequential should admit workloads sequentially even when quota is available [gangscheduling] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_gangscheduling_test.go:157 STEP: Creating Cluster Resources with enough quota for multiple jobs @ 02/25/26 13:29:23.714 STEP: Creating Namespace and LocalQueue @ 02/25/26 13:29:24.05 STEP: Creating LocalQueue local-queue in namespace sequential-ctmxr @ 02/25/26 13:29:24.218 STEP: Creating the first gang job (parallelism=2) with delayed pod readiness @ 02/25/26 13:29:24.381 STEP: Verifying first gang job workload is admitted @ 02/25/26 13:29:24.593 STEP: verifying workload created in namespace sequential-ctmxr and uid 1e7d8bff-c88d-4f8b-b109-c84ff44997f9 @ 02/25/26 13:29:24.593 STEP: Creating the second gang job (parallelism=2) while first gang job pods are not yet ready @ 02/25/26 13:29:24.922 STEP: Verifying second gang job is NOT admitted because sequential admission waits for first gang job pods to be ready @ 02/25/26 13:29:25.09 STEP: verifying workload is created in namespace sequential-ctmxr @ 02/25/26 13:29:25.09 STEP: Waiting for first gang job pods to complete @ 02/25/26 13:29:30.247 STEP: Verifying second gang job is now admitted after first gang job pods became ready @ 02/25/26 13:30:00.895 STEP: verifying workload created in namespace sequential-ctmxr and uid 2e7982cf-2070-49f5-a3ae-6cdf257a289e @ 02/25/26 13:30:00.896 STEP: Restoring initial Kueue configuration @ 02/25/26 13:30:01.219 STEP: Feching Kueue Instance @ 02/25/26 13:30:01.22 STEP: Updating Kueue config @ 02/25/26 13:30:01.703 STEP: Waiting for kueue-controller-manager resource version to change from 157968 @ 02/25/26 13:30:01.875 STEP: Checking if deployment replicas: 3 matches amount of ready replicas: 1 @ 02/25/26 13:30:12.823 STEP: Checking if deployment replicas: 2 matches amount of ready replicas: 2 @ 02/25/26 13:30:23.157 STEP: Waiting for webhook configurations to exist @ 02/25/26 13:30:23.158 STEP: Waiting for webhook to handle requests successfully @ 02/25/26 13:30:23.505 I0225 13:30:23.886924 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-x7df7 I0225 13:30:24.069348 45539 e2e_operator_test.go:1494] Webhook success 1/3 I0225 13:30:27.440373 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-v8xwg I0225 13:30:27.605003 45539 e2e_operator_test.go:1494] Webhook success 2/3 I0225 13:30:30.952888 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-5nnf6 I0225 13:30:31.125119 45539 e2e_operator_test.go:1491] Webhook stable after 3 consecutive successes I0225 13:30:31.301932 45539 warnings.go:110] "Warning: child pods are preserved by default when jobs are deleted; set propagationPolicy=Background to remove them or set propagationPolicy=Orphan to suppress this warning" I0225 13:30:31.480565 45539 warnings.go:110] "Warning: child pods are preserved by default when jobs are deleted; set propagationPolicy=Background to remove them or set propagationPolicy=Orphan to suppress this warning" STEP: Destroying LocalQueue sequential-ctmxr/local-queue @ 02/25/26 13:30:31.48 STEP: Deleting namespace sequential-ctmxr @ 02/25/26 13:30:31.97 STEP: Destroying ClusterQueue cluster-queue-rsc47 @ 02/25/26 13:30:42.639 STEP: Destroying ResourceFlavor resource-flavor-n5w7x @ 02/25/26 13:32:26.381 • [183.171 seconds] ------------------------------ Gangscheduling when Policy is ByWorkload and Admission is Parallel should admit workloads in parallel without waiting for pods to be ready [gangscheduling] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_gangscheduling_test.go:273 STEP: Saving initial Kueue configuration @ 02/25/26 13:32:26.875 STEP: Configuring Kueue with gangScheduling: policy=ByWorkload, admission=Parallel @ 02/25/26 13:32:27.035 STEP: Feching Kueue Instance @ 02/25/26 13:32:27.036 STEP: Updating Kueue config @ 02/25/26 13:32:27.51 STEP: Waiting for kueue-controller-manager resource version to change from 159371 @ 02/25/26 13:32:27.687 STEP: Checking if deployment replicas: 3 matches amount of ready replicas: 1 @ 02/25/26 13:32:38.356 STEP: Checking if deployment replicas: 2 matches amount of ready replicas: 2 @ 02/25/26 13:32:48.665 STEP: Waiting for webhook configurations to exist @ 02/25/26 13:32:48.665 STEP: Waiting for webhook to handle requests successfully @ 02/25/26 13:32:48.989 I0225 13:32:49.325432 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-b827t I0225 13:32:49.489006 45539 e2e_operator_test.go:1494] Webhook success 1/3 I0225 13:32:52.834500 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-jwsxt I0225 13:32:52.996913 45539 e2e_operator_test.go:1494] Webhook success 2/3 I0225 13:32:56.341060 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-gpjfw I0225 13:32:56.509208 45539 e2e_operator_test.go:1491] Webhook stable after 3 consecutive successes STEP: Waiting for Kueue configuration to be applied @ 02/25/26 13:32:56.509 STEP: Creating Cluster Resources with enough quota for multiple jobs @ 02/25/26 13:32:56.661 STEP: Creating Namespace and LocalQueue @ 02/25/26 13:32:57.003 STEP: Creating LocalQueue local-queue in namespace parallel-hzfkb @ 02/25/26 13:32:57.185 STEP: Creating the first gang job (parallelism=2) with delayed pod readiness @ 02/25/26 13:32:57.352 STEP: Verifying first gang job workload is admitted @ 02/25/26 13:32:57.73 STEP: verifying workload created in namespace parallel-hzfkb and uid bfd08d41-8f89-4460-bb9c-09a8d0c57b68 @ 02/25/26 13:32:57.731 STEP: Creating the second gang job (parallelism=2) while first gang job pods are not yet ready @ 02/25/26 13:32:58.07 STEP: Verifying second gang job is admitted immediately despite first gang job pods not being ready (parallel admission) @ 02/25/26 13:32:58.244 STEP: verifying workload created in namespace parallel-hzfkb and uid e47363cb-db78-4ca1-b6fb-1a74f3692b63 @ 02/25/26 13:32:58.245 STEP: Verifying both gang jobs are admitted together without waiting for pod readiness @ 02/25/26 13:32:58.567 STEP: Restoring initial Kueue configuration @ 02/25/26 13:32:58.722 STEP: Feching Kueue Instance @ 02/25/26 13:32:58.722 STEP: Updating Kueue config @ 02/25/26 13:32:59.2 STEP: Waiting for kueue-controller-manager resource version to change from 160809 @ 02/25/26 13:32:59.369 STEP: Checking if deployment replicas: 3 matches amount of ready replicas: 1 @ 02/25/26 13:33:10.026 STEP: Checking if deployment replicas: 2 matches amount of ready replicas: 2 @ 02/25/26 13:33:20.337 STEP: Waiting for webhook configurations to exist @ 02/25/26 13:33:20.338 STEP: Waiting for webhook to handle requests successfully @ 02/25/26 13:33:20.677 I0225 13:33:21.038560 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-r47qh I0225 13:33:21.204859 45539 e2e_operator_test.go:1494] Webhook success 1/3 I0225 13:33:24.544721 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-s7qbk I0225 13:33:24.708572 45539 e2e_operator_test.go:1494] Webhook success 2/3 I0225 13:33:28.052861 45539 e2e_operator_test.go:1482] Webhook test successful, cleaning up test job: webhook-test-2958s I0225 13:33:28.224553 45539 e2e_operator_test.go:1491] Webhook stable after 3 consecutive successes I0225 13:33:28.396831 45539 warnings.go:110] "Warning: child pods are preserved by default when jobs are deleted; set propagationPolicy=Background to remove them or set propagationPolicy=Orphan to suppress this warning" I0225 13:33:28.564875 45539 warnings.go:110] "Warning: child pods are preserved by default when jobs are deleted; set propagationPolicy=Background to remove them or set propagationPolicy=Orphan to suppress this warning" STEP: Destroying LocalQueue parallel-hzfkb/local-queue @ 02/25/26 13:33:28.565 STEP: Deleting namespace parallel-hzfkb @ 02/25/26 13:33:29.065 STEP: Destroying ClusterQueue cluster-queue-b9jfw @ 02/25/26 13:33:39.717 STEP: Destroying ResourceFlavor resource-flavor-pgwss @ 02/25/26 13:35:28.771 • [182.357 seconds] ------------------------------ Kueue Metrics Metrics Infrastructure should have metrics service created [metrics] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_metrics_test.go:156 STEP: Ensuring monitoring namespace exists @ 02/25/26 13:35:29.251 STEP: Copying metrics certificate secret to monitoring namespace @ 02/25/26 13:35:29.42 STEP: Creating curl pod in monitoring namespace @ 02/25/26 13:35:29.954 STEP: Waiting for curl pod to be ready @ 02/25/26 13:35:30.151 • [11.399 seconds] ------------------------------ Kueue Metrics Metrics Infrastructure should have metrics TLS secret created [metrics] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_metrics_test.go:165 • [0.165 seconds] ------------------------------ Kueue Metrics Metrics Infrastructure should have ServiceMonitor created [metrics] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_metrics_test.go:174 • [0.175 seconds] ------------------------------ Kueue Metrics Metrics Infrastructure should have metrics network policy with correct namespace selectors [metrics] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_metrics_test.go:187 • [0.166 seconds] ------------------------------ Kueue Metrics Metrics Endpoint should return valid metrics over HTTPS [metrics] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_metrics_test.go:223 STEP: Executing curl command in monitoring pod @ 02/25/26 13:35:41.155 STEP: Verifying metrics format @ 02/25/26 13:35:42.843 Metrics output (first 500 chars): # HELP certwatcher_read_certificate_errors_total Total number of certificate read errors # TYPE certwatcher_read_certificate_errors_total counter certwatcher_read_certificate_errors_total 0 # HELP certwatcher_read_certificate_total Total number of certificate reads # TYPE certwatcher_read_certificate_total counter certwatcher_read_certificate_total 32 # HELP controller_runtime_active_workers Number of currently used workers per controller # TYPE controller_runtime_active_workers gauge controller • [1.688 seconds] ------------------------------ Kueue Metrics Metrics Endpoint should expose kueue-specific metrics [metrics] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_metrics_test.go:247 STEP: Executing curl command in monitoring pod @ 02/25/26 13:35:42.843 STEP: Checking for kueue build info metric @ 02/25/26 13:35:44.464 STEP: Checking for controller-runtime metrics @ 02/25/26 13:35:44.464 • [1.621 seconds] ------------------------------ Kueue Metrics Cross-namespace Access should allow metrics scraping from monitoring namespace [metrics] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_metrics_test.go:271 STEP: Verifying monitoring pod can reach metrics from operator namespace @ 02/25/26 13:35:44.465 • [1.598 seconds] ------------------------------ Kueue Metrics Cross-namespace Access should block metrics scraping from non-monitoring namespace [metrics] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_metrics_test.go:283 STEP: Creating a test namespace without monitoring labels @ 02/25/26 13:35:46.063 STEP: Copying metrics certificate secret to test namespace @ 02/25/26 13:35:46.234 STEP: Creating curl pod in test namespace @ 02/25/26 13:35:46.602 STEP: Waiting for test pod to be ready @ 02/25/26 13:35:46.772 STEP: Verifying test pod CANNOT reach metrics (should timeout or be forbidden) @ 02/25/26 13:35:57.108 STEP: Cleaning up test namespace @ 02/25/26 13:35:58.174 • [12.604 seconds] ------------------------------ Summarizing 1 Failure: [FAIL] Kueue Operator when enable webhook via opt-in namespaces [It] should manage jobs only in labeled namespaces [operator] /Users/anahas/Github/forks/kueue-operator/kueue-operator/test/e2e/e2e_operator_test.go:559 Ran 44 of 53 Specs in 2556.851 seconds FAIL! -- 43 Passed | 1 Failed | 1 Flaked | 0 Pending | 9 Skipped --- FAIL: TestE2E (2556.85s) FAIL You're using deprecated Ginkgo functionality: ============================================= --ginkgo.slow-spec-threshold is deprecated --slow-spec-threshold has been deprecated and will be removed in a future version of Ginkgo. This feature has proved to be more noisy than useful. You can use --poll-progress-after, instead, to get more actionable feedback about potentially slow specs and understand where they might be getting stuck. To silence deprecations that can be silenced set the following environment variable: ACK_GINKGO_DEPRECATIONS=2.27.2 Ginkgo ran 1 suite in 42m42.7388865s Test Suite Failed make: *** [test-e2e] Error 1