-
Bug
-
Resolution: Duplicate
-
Undefined
-
None
-
4.14.z
-
None
-
Quality / Stability / Reliability
-
False
-
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
Our Hosted Cluster is having trouble reaching the ignition endpoint. The NodePool shows ReconcileError and the HO logs show the same errors as well. Here's the NodePool apiVersion: hypershift.openshift.io/v1beta1 kind: NodePool metadata: annotations: hypershift.openshift.io/nodePoolCurrentConfig: 62484d3e hypershift.openshift.io/nodePoolCurrentConfigVersion: d7a2fb4d hypershift.openshift.io/nodePoolPlatformMachineTemplate: nodepool-dev-gpc-zero-dc-rdu2-1-c880c958 creationTimestamp: "2025-03-28T17:13:11Z" finalizers: - hypershift.openshift.io/finalizer generation: 3 name: nodepool-dev-gpc-zero-dc-rdu2-1 namespace: dev-gpc-zero-dc-rdu2 ownerReferences: - apiVersion: hypershift.openshift.io/v1beta1 kind: HostedCluster name: dev-gpc-zero-dc-rdu2 uid: ef4c1b36-90b6-49ad-809d-88bafd6e6605 resourceVersion: "37304733" uid: 52611b50-2009-49b0-a9dc-adb54217a9a8 spec: arch: amd64 clusterName: dev-gpc-zero-dc-rdu2 management: autoRepair: false replace: rollingUpdate: maxSurge: 1 maxUnavailable: 0 strategy: RollingUpdate upgradeType: InPlace platform: agent: agentLabelSelector: {} type: Agent release: image: quay.io/openshift-release-dev/ocp-release:4.14.49-multi replicas: 3 status: conditions: - lastTransitionTime: "2025-03-28T17:13:13Z" observedGeneration: 3 reason: AsExpected status: "False" type: AutoscalingEnabled - lastTransitionTime: "2025-03-28T17:13:13Z" observedGeneration: 3 reason: AsExpected status: "True" type: UpdateManagementEnabled - lastTransitionTime: "2025-03-28T17:17:59Z" message: 'Using release image: quay.io/openshift-release-dev/ocp-release:4.14.49-multi' observedGeneration: 3 reason: AsExpected status: "True" type: ValidReleaseImage - lastTransitionTime: "2025-04-01T18:23:54Z" observedGeneration: 3 reason: AsExpected status: "True" type: ValidMachineConfig - lastTransitionTime: "2025-04-01T19:10:23Z" message: Payload generated successfully observedGeneration: 3 reason: AsExpected status: "True" type: ValidGeneratedPayload - lastTransitionTime: "2025-03-28T17:18:02Z" observedGeneration: 3 reason: ignitionNotReached status: "False" type: ReachedIgnitionEndpoint - lastTransitionTime: "2025-03-28T17:18:02Z" observedGeneration: 3 reason: AsExpected status: "True" type: ValidTuningConfig - lastTransitionTime: "2025-03-28T17:18:02Z" message: Reconciliation active on resource observedGeneration: 3 reason: ReconciliationActive status: "True" type: ReconciliationActive - lastTransitionTime: "2025-03-28T19:36:38Z" message: All is well observedGeneration: 3 reason: AsExpected status: "True" type: AllMachinesReady - lastTransitionTime: "2025-04-01T19:57:19Z" message: All is well observedGeneration: 3 reason: AsExpected status: "True" type: AllNodesHealthy - lastTransitionTime: "2025-03-28T19:36:39Z" observedGeneration: 3 reason: AsExpected status: "True" type: Ready - lastTransitionTime: "2025-03-28T17:18:03Z" observedGeneration: 3 reason: AsExpected status: "False" type: AutorepairEnabled - lastTransitionTime: "2025-04-01T18:27:19Z" observedGeneration: 3 reason: AsExpected status: "True" type: ValidArchPlatform - lastTransitionTime: "2025-04-01T19:57:51Z" observedGeneration: 3 reason: AsExpected status: "False" type: UpdatingConfig - lastTransitionTime: "2025-04-01T18:27:19Z" observedGeneration: 3 reason: AsExpected status: "False" type: UpdatingVersion - lastTransitionTime: "2025-04-01T18:27:21Z" observedGeneration: 3 reason: AsExpected status: "False" type: UpdatingPlatformMachineTemplate replicas: 3 version: 4.14.49 And the logs of the hypershift-operator Defaulted container "operator" out of: operator, init-environment (init) Defaulted container "operator" out of: operator, init-environment (init) {"level":"info","ts":"2025-04-02T15:42:09Z","msg":"removing secret as it does not match the expected set of names","controller":"secret","controllerGroup":"","controllerKind":"Secret","Secret":{"name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2"},"namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2","name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","reconcileID":"0accfadd-1249-429b-bf1a-cf01dba19946","secret":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2/token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","nodePool":"dev-gpc-zero-dc-rdu2/nodepool-dev-gpc-zero-dc-rdu2-1","options":["token-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d","user-data-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d"],"valid":false} {"level":"info","ts":"2025-04-02T15:42:12Z","msg":"removing secret as it does not match the expected set of names","controller":"secret","controllerGroup":"","controllerKind":"Secret","Secret":{"name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2"},"namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2","name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","reconcileID":"f1cc76d5-7508-4384-bc23-69ec5fe4e018","secret":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2/token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","nodePool":"dev-gpc-zero-dc-rdu2/nodepool-dev-gpc-zero-dc-rdu2-1","options":["token-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d","user-data-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d"],"valid":false} {"level":"info","ts":"2025-04-02T15:42:13Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"} {"level":"info","ts":"2025-04-02T15:42:13Z","msg":"removing secret as it does not match the expected set of names","controller":"secret","controllerGroup":"","controllerKind":"Secret","Secret":{"name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2"},"namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2","name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","reconcileID":"03d843f8-631b-403d-a6e2-a7924725a11f","secret":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2/token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","nodePool":"dev-gpc-zero-dc-rdu2/nodepool-dev-gpc-zero-dc-rdu2-1","options":["token-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d","user-data-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d"],"valid":false} {"level":"info","ts":"2025-04-02T15:42:14Z","msg":"removing secret as it does not match the expected set of names","controller":"secret","controllerGroup":"","controllerKind":"Secret","Secret":{"name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-f46f5ae4","namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2"},"namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2","name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-f46f5ae4","reconcileID":"279561ab-646b-4aac-a143-0097960ee621","secret":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2/token-nodepool-dev-gpc-zero-dc-rdu2-1-f46f5ae4","nodePool":"dev-gpc-zero-dc-rdu2/nodepool-dev-gpc-zero-dc-rdu2-1","options":["token-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d","user-data-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d"],"valid":false} {"level":"info","ts":"2025-04-02T15:42:14Z","msg":"removing secret as it does not match the expected set of names","controller":"secret","controllerGroup":"","controllerKind":"Secret","Secret":{"name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2"},"namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2","name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","reconcileID":"83d02462-d3f7-41b0-a026-8b5ceddfaed0","secret":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2/token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","nodePool":"dev-gpc-zero-dc-rdu2/nodepool-dev-gpc-zero-dc-rdu2-1","options":["token-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d","user-data-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d"],"valid":false} {"level":"info","ts":"2025-04-02T15:42:16Z","msg":"removing secret as it does not match the expected set of names","controller":"secret","controllerGroup":"","controllerKind":"Secret","Secret":{"name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-f46f5ae4","namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2"},"namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2","name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-f46f5ae4","reconcileID":"8a3ba627-1cd6-4acc-87f0-c5d73c9c7701","secret":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2/token-nodepool-dev-gpc-zero-dc-rdu2-1-f46f5ae4","nodePool":"dev-gpc-zero-dc-rdu2/nodepool-dev-gpc-zero-dc-rdu2-1","options":["token-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d","user-data-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d"],"valid":false} {"level":"info","ts":"2025-04-02T15:42:17Z","msg":"removing secret as it does not match the expected set of names","controller":"secret","controllerGroup":"","controllerKind":"Secret","Secret":{"name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-609bcfb1","namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2"},"namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2","name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-609bcfb1","reconcileID":"113831bb-7bc3-4d9e-b53f-6f9073ca86a6","secret":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2/token-nodepool-dev-gpc-zero-dc-rdu2-1-609bcfb1","nodePool":"dev-gpc-zero-dc-rdu2/nodepool-dev-gpc-zero-dc-rdu2-1","options":["token-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d","user-data-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d"],"valid":false} {"level":"error","ts":"2025-04-02T15:42:19Z","msg":"Reconciler error","controller":"secret","controllerGroup":"","controllerKind":"Secret","Secret":{"name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-609bcfb1","namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2"},"namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2","name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-609bcfb1","reconcileID":"225fcd0f-ce96-4b5a-8e57-a046dd3ec47c","error":"failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain image configuration for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: cannot retrieve image configuration for manifest sha256:b57ccde580b7e2d5383d006da07acebc099ad181bb877c4c705d3cdf0a535ee9 in manifest list sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: received unexpected HTTP status: 502 Bad Gateway","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:224"} {"level":"info","ts":"2025-04-02T15:42:20Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"} {"level":"info","ts":"2025-04-02T15:43:40Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"} {"level":"info","ts":"2025-04-02T15:43:40Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"} {"level":"info","ts":"2025-04-02T15:43:41Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"} {"level":"info","ts":"2025-04-02T15:43:56Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"} {"level":"info","ts":"2025-04-02T15:44:11Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"} {"level":"info","ts":"2025-04-02T15:44:26Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"} {"level":"info","ts":"2025-04-02T15:44:40Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"} {"level":"info","ts":"2025-04-02T15:44:40Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"} {"level":"info","ts":"2025-04-02T15:44:41Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"} {"level":"info","ts":"2025-04-02T15:44:56Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"}
Version-Release number of selected component (if applicable):
Using HyperShift image http://quay.io/acm-d/rhtap-hypershift-operator@sha256:ae7db7ff02251e32609f445905530b81adb991faa600aa5ee6a0e792e1ef05a0 Using MCE multicluster-engine.v2.4.8 HostedCluster running 4.14.49
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
I expected to see the NodePool not in a Reconcile Loop and the HO not to have those errors
Additional info:
> oc describe np nodepool-dev-gpc-zero-dc-rdu2-1 Name: nodepool-dev-gpc-zero-dc-rdu2-1 Namespace: dev-gpc-zero-dc-rdu2 Labels: <none> Annotations: hypershift.openshift.io/nodePoolCurrentConfig: 62484d3e hypershift.openshift.io/nodePoolCurrentConfigVersion: d7a2fb4d hypershift.openshift.io/nodePoolPlatformMachineTemplate: nodepool-dev-gpc-zero-dc-rdu2-1-c880c958 API Version: hypershift.openshift.io/v1beta1 Kind: NodePool Metadata: Creation Timestamp: 2025-03-28T17:13:11Z Finalizers: hypershift.openshift.io/finalizer Generation: 3 Owner References: API Version: hypershift.openshift.io/v1beta1 Kind: HostedCluster Name: dev-gpc-zero-dc-rdu2 UID: ef4c1b36-90b6-49ad-809d-88bafd6e6605 Resource Version: 37304733 UID: 52611b50-2009-49b0-a9dc-adb54217a9a8 Spec: Arch: amd64 Cluster Name: dev-gpc-zero-dc-rdu2 Management: Auto Repair: false Replace: Rolling Update: Max Surge: 1 Max Unavailable: 0 Strategy: RollingUpdate Upgrade Type: InPlace Platform: Agent: Agent Label Selector: Type: Agent Release: Image: quay.io/openshift-release-dev/ocp-release:4.14.49-multi Replicas: 3 Status: Conditions: Last Transition Time: 2025-03-28T17:13:13Z Observed Generation: 3 Reason: AsExpected Status: False Type: AutoscalingEnabled Last Transition Time: 2025-03-28T17:13:13Z Observed Generation: 3 Reason: AsExpected Status: True Type: UpdateManagementEnabled Last Transition Time: 2025-03-28T17:17:59Z Message: Using release image: quay.io/openshift-release-dev/ocp-release:4.14.49-multi Observed Generation: 3 Reason: AsExpected Status: True Type: ValidReleaseImage Last Transition Time: 2025-04-01T18:23:54Z Observed Generation: 3 Reason: AsExpected Status: True Type: ValidMachineConfig Last Transition Time: 2025-04-01T19:10:23Z Message: Payload generated successfully Observed Generation: 3 Reason: AsExpected Status: True Type: ValidGeneratedPayload Last Transition Time: 2025-03-28T17:18:02Z Observed Generation: 3 Reason: ignitionNotReached Status: False Type: ReachedIgnitionEndpoint Last Transition Time: 2025-03-28T17:18:02Z Observed Generation: 3 Reason: AsExpected Status: True Type: ValidTuningConfig Last Transition Time: 2025-03-28T17:18:02Z Message: Reconciliation active on resource Observed Generation: 3 Reason: ReconciliationActive Status: True Type: ReconciliationActive Last Transition Time: 2025-03-28T19:36:38Z Message: All is well Observed Generation: 3 Reason: AsExpected Status: True Type: AllMachinesReady Last Transition Time: 2025-04-01T19:57:19Z Message: All is well Observed Generation: 3 Reason: AsExpected Status: True Type: AllNodesHealthy Last Transition Time: 2025-03-28T19:36:39Z Observed Generation: 3 Reason: AsExpected Status: True Type: Ready Last Transition Time: 2025-03-28T17:18:03Z Observed Generation: 3 Reason: AsExpected Status: False Type: AutorepairEnabled Last Transition Time: 2025-04-01T18:27:19Z Observed Generation: 3 Reason: AsExpected Status: True Type: ValidArchPlatform Last Transition Time: 2025-04-01T19:57:51Z Observed Generation: 3 Reason: AsExpected Status: False Type: UpdatingConfig Last Transition Time: 2025-04-01T18:27:19Z Observed Generation: 3 Reason: AsExpected Status: False Type: UpdatingVersion Last Transition Time: 2025-04-01T18:27:21Z Observed Generation: 3 Reason: AsExpected Status: False Type: UpdatingPlatformMachineTemplate Replicas: 3 Version: 4.14.49 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning ReconcileError 123m nodepool-controller error getting token: failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to create repository client for https://quay.io: endpoint "https://quay.io" does not support v2 API (got 502 Bad Gateway) Warning ReconcileError 119m nodepool-controller error getting token: failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #4 from manifest list: received unexpected HTTP status: 502 Bad Gateway Warning ReconcileError 108m (x2 over 118m) nodepool-controller error getting token: failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: received unexpected HTTP status: 502 Bad Gateway Warning ReconcileError 108m nodepool-controller error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to create repository client for https://quay.io: endpoint "https://quay.io" does not support v2 API (got 502 Bad Gateway) Warning ReconcileError 108m nodepool-controller failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #4 from manifest list: received unexpected HTTP status: 502 Bad Gateway Warning ReconcileError 94m nodepool-controller error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #2 from manifest list: received unexpected HTTP status: 502 Bad Gateway Warning ReconcileError 92m (x2 over 4h) nodepool-controller error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #4 from manifest list: received unexpected HTTP status: 502 Bad Gateway Warning ReconcileError 87m nodepool-controller failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to create repository client for https://quay.io: Get "https://quay.io/v2/": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Warning ReconcileError 87m (x2 over 4h18m) nodepool-controller error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #3 from manifest list: received unexpected HTTP status: 502 Bad Gateway Warning ReconcileError 87m nodepool-controller failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #1 from manifest list: received unexpected HTTP status: 502 Bad Gateway Warning ReconcileError 81m (x3 over 87m) nodepool-controller error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: Get "https://quay.io/v2/openshift-release-dev/ocp-v4.0-art-dev/manifests/sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb": received unexpected HTTP status: 502 Bad Gateway Warning ReconcileError 79m (x2 over 95m) nodepool-controller error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain image configuration for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: cannot retrieve image configuration for manifest sha256:b57ccde580b7e2d5383d006da07acebc099ad181bb877c4c705d3cdf0a535ee9 in manifest list sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: received unexpected HTTP status: 502 Bad Gateway Warning ReconcileError 78m (x6 over 82m) nodepool-controller (combined from similar events): error getting token: failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to create repository client for https://quay.io: Get "https://quay.io/v2/": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Warning ReconcileError 58m (x2 over 58m) nodepool-controller error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain image configuration for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: cannot retrieve image configuration for manifest sha256:b57ccde580b7e2d5383d006da07acebc099ad181bb877c4c705d3cdf0a535ee9 in manifest list sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: content integrity error: the blob retrieved with digest sha256:a4261a9368560c023ff3e66869a0f4cf6c50d8ca73fe4cefeca9458ce08c9ee6 does not match the digest calculated from the content sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 Warning ReconcileError 58m nodepool-controller failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #1 from manifest list: error unmarshalling content: unexpected end of JSON input Warning ReconcileError 58m nodepool-controller error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #4 from manifest list: error unmarshalling content: unexpected end of JSON input Warning ReconcileError 58m (x2 over 58m) nodepool-controller error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: error unmarshalling content: unexpected end of JSON input Warning ReconcileError 58m (x2 over 58m) nodepool-controller error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #3 from manifest list: error unmarshalling content: unexpected end of JSON input Warning ReconcileError 58m nodepool-controller error getting token: failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain image configuration for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: cannot retrieve image configuration for manifest sha256:b57ccde580b7e2d5383d006da07acebc099ad181bb877c4c705d3cdf0a535ee9 in manifest list sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: content integrity error: the blob retrieved with digest sha256:a4261a9368560c023ff3e66869a0f4cf6c50d8ca73fe4cefeca9458ce08c9ee6 does not match the digest calculated from the content sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 Warning ReconcileError 57m (x3 over 59m) nodepool-controller error getting token: failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #1 from manifest list: error unmarshalling content: unexpected end of JSON input Warning ReconcileError 57m nodepool-controller failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #2 from manifest list: error unmarshalling content: unexpected end of JSON input Warning ReconcileError 57m nodepool-controller failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #3 from manifest list: error unmarshalling content: unexpected end of JSON input Warning ReconcileError 29m nodepool-controller error getting token: failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain image configuration for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: cannot retrieve image configuration for manifest sha256:b57ccde580b7e2d5383d006da07acebc099ad181bb877c4c705d3cdf0a535ee9 in manifest list sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: received unexpected HTTP status: 502 Bad Gateway Warning ReconcileError 14m (x2 over 20h) nodepool-controller error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #1 from manifest list: received unexpected HTTP status: 502 Bad Gateway Warning ReconcileError 13m nodepool-controller failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #2 from manifest list: received unexpected HTTP status: 502 Bad Gateway Warning ReconcileError 12m (x3 over 7h33m) nodepool-controller failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #3 from manifest list: received unexpected HTTP status: 502 Bad Gateway Warning ReconcileError 8m32s (x2 over 86m) nodepool-controller error getting token: failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #3 from manifest list: received unexpected HTTP status: 502 Bad Gateway Warning ReconcileError 8m27s (x3 over 19h) nodepool-controller error getting token: failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #2 from manifest list: received unexpected HTTP status: 502 Bad Gateway