-
Bug
-
Resolution: Duplicate
-
Undefined
-
None
-
4.14.z
-
None
-
Quality / Stability / Reliability
-
False
-
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
Our Hosted Cluster is having trouble reaching the ignition endpoint. The NodePool shows ReconcileError and the HO logs show the same errors as well. Here's the NodePool
apiVersion: hypershift.openshift.io/v1beta1
kind: NodePool
metadata:
annotations:
hypershift.openshift.io/nodePoolCurrentConfig: 62484d3e
hypershift.openshift.io/nodePoolCurrentConfigVersion: d7a2fb4d
hypershift.openshift.io/nodePoolPlatformMachineTemplate: nodepool-dev-gpc-zero-dc-rdu2-1-c880c958
creationTimestamp: "2025-03-28T17:13:11Z"
finalizers:
- hypershift.openshift.io/finalizer
generation: 3
name: nodepool-dev-gpc-zero-dc-rdu2-1
namespace: dev-gpc-zero-dc-rdu2
ownerReferences:
- apiVersion: hypershift.openshift.io/v1beta1
kind: HostedCluster
name: dev-gpc-zero-dc-rdu2
uid: ef4c1b36-90b6-49ad-809d-88bafd6e6605
resourceVersion: "37304733"
uid: 52611b50-2009-49b0-a9dc-adb54217a9a8
spec:
arch: amd64
clusterName: dev-gpc-zero-dc-rdu2
management:
autoRepair: false
replace:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
strategy: RollingUpdate
upgradeType: InPlace
platform:
agent:
agentLabelSelector: {}
type: Agent
release:
image: quay.io/openshift-release-dev/ocp-release:4.14.49-multi
replicas: 3
status:
conditions:
- lastTransitionTime: "2025-03-28T17:13:13Z"
observedGeneration: 3
reason: AsExpected
status: "False"
type: AutoscalingEnabled
- lastTransitionTime: "2025-03-28T17:13:13Z"
observedGeneration: 3
reason: AsExpected
status: "True"
type: UpdateManagementEnabled
- lastTransitionTime: "2025-03-28T17:17:59Z"
message: 'Using release image: quay.io/openshift-release-dev/ocp-release:4.14.49-multi'
observedGeneration: 3
reason: AsExpected
status: "True"
type: ValidReleaseImage
- lastTransitionTime: "2025-04-01T18:23:54Z"
observedGeneration: 3
reason: AsExpected
status: "True"
type: ValidMachineConfig
- lastTransitionTime: "2025-04-01T19:10:23Z"
message: Payload generated successfully
observedGeneration: 3
reason: AsExpected
status: "True"
type: ValidGeneratedPayload
- lastTransitionTime: "2025-03-28T17:18:02Z"
observedGeneration: 3
reason: ignitionNotReached
status: "False"
type: ReachedIgnitionEndpoint
- lastTransitionTime: "2025-03-28T17:18:02Z"
observedGeneration: 3
reason: AsExpected
status: "True"
type: ValidTuningConfig
- lastTransitionTime: "2025-03-28T17:18:02Z"
message: Reconciliation active on resource
observedGeneration: 3
reason: ReconciliationActive
status: "True"
type: ReconciliationActive
- lastTransitionTime: "2025-03-28T19:36:38Z"
message: All is well
observedGeneration: 3
reason: AsExpected
status: "True"
type: AllMachinesReady
- lastTransitionTime: "2025-04-01T19:57:19Z"
message: All is well
observedGeneration: 3
reason: AsExpected
status: "True"
type: AllNodesHealthy
- lastTransitionTime: "2025-03-28T19:36:39Z"
observedGeneration: 3
reason: AsExpected
status: "True"
type: Ready
- lastTransitionTime: "2025-03-28T17:18:03Z"
observedGeneration: 3
reason: AsExpected
status: "False"
type: AutorepairEnabled
- lastTransitionTime: "2025-04-01T18:27:19Z"
observedGeneration: 3
reason: AsExpected
status: "True"
type: ValidArchPlatform
- lastTransitionTime: "2025-04-01T19:57:51Z"
observedGeneration: 3
reason: AsExpected
status: "False"
type: UpdatingConfig
- lastTransitionTime: "2025-04-01T18:27:19Z"
observedGeneration: 3
reason: AsExpected
status: "False"
type: UpdatingVersion
- lastTransitionTime: "2025-04-01T18:27:21Z"
observedGeneration: 3
reason: AsExpected
status: "False"
type: UpdatingPlatformMachineTemplate
replicas: 3
version: 4.14.49
And the logs of the hypershift-operator
Defaulted container "operator" out of: operator, init-environment (init)
Defaulted container "operator" out of: operator, init-environment (init)
{"level":"info","ts":"2025-04-02T15:42:09Z","msg":"removing secret as it does not match the expected set of names","controller":"secret","controllerGroup":"","controllerKind":"Secret","Secret":{"name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2"},"namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2","name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","reconcileID":"0accfadd-1249-429b-bf1a-cf01dba19946","secret":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2/token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","nodePool":"dev-gpc-zero-dc-rdu2/nodepool-dev-gpc-zero-dc-rdu2-1","options":["token-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d","user-data-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d"],"valid":false}
{"level":"info","ts":"2025-04-02T15:42:12Z","msg":"removing secret as it does not match the expected set of names","controller":"secret","controllerGroup":"","controllerKind":"Secret","Secret":{"name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2"},"namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2","name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","reconcileID":"f1cc76d5-7508-4384-bc23-69ec5fe4e018","secret":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2/token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","nodePool":"dev-gpc-zero-dc-rdu2/nodepool-dev-gpc-zero-dc-rdu2-1","options":["token-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d","user-data-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d"],"valid":false}
{"level":"info","ts":"2025-04-02T15:42:13Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"}
{"level":"info","ts":"2025-04-02T15:42:13Z","msg":"removing secret as it does not match the expected set of names","controller":"secret","controllerGroup":"","controllerKind":"Secret","Secret":{"name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2"},"namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2","name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","reconcileID":"03d843f8-631b-403d-a6e2-a7924725a11f","secret":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2/token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","nodePool":"dev-gpc-zero-dc-rdu2/nodepool-dev-gpc-zero-dc-rdu2-1","options":["token-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d","user-data-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d"],"valid":false}
{"level":"info","ts":"2025-04-02T15:42:14Z","msg":"removing secret as it does not match the expected set of names","controller":"secret","controllerGroup":"","controllerKind":"Secret","Secret":{"name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-f46f5ae4","namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2"},"namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2","name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-f46f5ae4","reconcileID":"279561ab-646b-4aac-a143-0097960ee621","secret":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2/token-nodepool-dev-gpc-zero-dc-rdu2-1-f46f5ae4","nodePool":"dev-gpc-zero-dc-rdu2/nodepool-dev-gpc-zero-dc-rdu2-1","options":["token-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d","user-data-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d"],"valid":false}
{"level":"info","ts":"2025-04-02T15:42:14Z","msg":"removing secret as it does not match the expected set of names","controller":"secret","controllerGroup":"","controllerKind":"Secret","Secret":{"name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2"},"namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2","name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","reconcileID":"83d02462-d3f7-41b0-a026-8b5ceddfaed0","secret":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2/token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","nodePool":"dev-gpc-zero-dc-rdu2/nodepool-dev-gpc-zero-dc-rdu2-1","options":["token-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d","user-data-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d"],"valid":false}
{"level":"info","ts":"2025-04-02T15:42:16Z","msg":"removing secret as it does not match the expected set of names","controller":"secret","controllerGroup":"","controllerKind":"Secret","Secret":{"name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-f46f5ae4","namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2"},"namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2","name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-f46f5ae4","reconcileID":"8a3ba627-1cd6-4acc-87f0-c5d73c9c7701","secret":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2/token-nodepool-dev-gpc-zero-dc-rdu2-1-f46f5ae4","nodePool":"dev-gpc-zero-dc-rdu2/nodepool-dev-gpc-zero-dc-rdu2-1","options":["token-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d","user-data-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d"],"valid":false}
{"level":"info","ts":"2025-04-02T15:42:17Z","msg":"removing secret as it does not match the expected set of names","controller":"secret","controllerGroup":"","controllerKind":"Secret","Secret":{"name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-609bcfb1","namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2"},"namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2","name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-609bcfb1","reconcileID":"113831bb-7bc3-4d9e-b53f-6f9073ca86a6","secret":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2/token-nodepool-dev-gpc-zero-dc-rdu2-1-609bcfb1","nodePool":"dev-gpc-zero-dc-rdu2/nodepool-dev-gpc-zero-dc-rdu2-1","options":["token-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d","user-data-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d"],"valid":false}
{"level":"error","ts":"2025-04-02T15:42:19Z","msg":"Reconciler error","controller":"secret","controllerGroup":"","controllerKind":"Secret","Secret":{"name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-609bcfb1","namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2"},"namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2","name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-609bcfb1","reconcileID":"225fcd0f-ce96-4b5a-8e57-a046dd3ec47c","error":"failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain image configuration for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: cannot retrieve image configuration for manifest sha256:b57ccde580b7e2d5383d006da07acebc099ad181bb877c4c705d3cdf0a535ee9 in manifest list sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: received unexpected HTTP status: 502 Bad Gateway","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:224"}
{"level":"info","ts":"2025-04-02T15:42:20Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"}
{"level":"info","ts":"2025-04-02T15:43:40Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"}
{"level":"info","ts":"2025-04-02T15:43:40Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"}
{"level":"info","ts":"2025-04-02T15:43:41Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"}
{"level":"info","ts":"2025-04-02T15:43:56Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"}
{"level":"info","ts":"2025-04-02T15:44:11Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"}
{"level":"info","ts":"2025-04-02T15:44:26Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"}
{"level":"info","ts":"2025-04-02T15:44:40Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"}
{"level":"info","ts":"2025-04-02T15:44:40Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"}
{"level":"info","ts":"2025-04-02T15:44:41Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"}
{"level":"info","ts":"2025-04-02T15:44:56Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"}
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1.
2.
3.
Actual results:
Expected results:
I expected to see the NodePool not in a Reconcile Loop and the HO not to have those errors
Additional info:
> oc describe np nodepool-dev-gpc-zero-dc-rdu2-1 Name: nodepool-dev-gpc-zero-dc-rdu2-1 Namespace: dev-gpc-zero-dc-rdu2 Labels: <none> Annotations: hypershift.openshift.io/nodePoolCurrentConfig: 62484d3e hypershift.openshift.io/nodePoolCurrentConfigVersion: d7a2fb4d hypershift.openshift.io/nodePoolPlatformMachineTemplate: nodepool-dev-gpc-zero-dc-rdu2-1-c880c958 API Version: hypershift.openshift.io/v1beta1 Kind: NodePool Metadata: Creation Timestamp: 2025-03-28T17:13:11Z Finalizers: hypershift.openshift.io/finalizer Generation: 3 Owner References: API Version: hypershift.openshift.io/v1beta1 Kind: HostedCluster Name: dev-gpc-zero-dc-rdu2 UID: ef4c1b36-90b6-49ad-809d-88bafd6e6605 Resource Version: 37304733 UID: 52611b50-2009-49b0-a9dc-adb54217a9a8 Spec: Arch: amd64 Cluster Name: dev-gpc-zero-dc-rdu2 Management: Auto Repair: false Replace: Rolling Update: Max Surge: 1 Max Unavailable: 0 Strategy: RollingUpdate Upgrade Type: InPlace Platform: Agent: Agent Label Selector: Type: Agent Release: Image: quay.io/openshift-release-dev/ocp-release:4.14.49-multi Replicas: 3 Status: Conditions: Last Transition Time: 2025-03-28T17:13:13Z Observed Generation: 3 Reason: AsExpected Status: False Type: AutoscalingEnabled Last Transition Time: 2025-03-28T17:13:13Z Observed Generation: 3 Reason: AsExpected Status: True Type: UpdateManagementEnabled Last Transition Time: 2025-03-28T17:17:59Z Message: Using release image: quay.io/openshift-release-dev/ocp-release:4.14.49-multi Observed Generation: 3 Reason: AsExpected Status: True Type: ValidReleaseImage Last Transition Time: 2025-04-01T18:23:54Z Observed Generation: 3 Reason: AsExpected Status: True Type: ValidMachineConfig Last Transition Time: 2025-04-01T19:10:23Z Message: Payload generated successfully Observed Generation: 3 Reason: AsExpected Status: True Type: ValidGeneratedPayload Last Transition Time: 2025-03-28T17:18:02Z Observed Generation: 3 Reason: ignitionNotReached Status: False Type: ReachedIgnitionEndpoint Last Transition Time: 2025-03-28T17:18:02Z Observed Generation: 3 Reason: AsExpected Status: True Type: ValidTuningConfig Last Transition Time: 2025-03-28T17:18:02Z Message: Reconciliation active on resource Observed Generation: 3 Reason: ReconciliationActive Status: True Type: ReconciliationActive Last Transition Time: 2025-03-28T19:36:38Z Message: All is well Observed Generation: 3 Reason: AsExpected Status: True Type: AllMachinesReady Last Transition Time: 2025-04-01T19:57:19Z Message: All is well Observed Generation: 3 Reason: AsExpected Status: True Type: AllNodesHealthy Last Transition Time: 2025-03-28T19:36:39Z Observed Generation: 3 Reason: AsExpected Status: True Type: Ready Last Transition Time: 2025-03-28T17:18:03Z Observed Generation: 3 Reason: AsExpected Status: False Type: AutorepairEnabled Last Transition Time: 2025-04-01T18:27:19Z Observed Generation: 3 Reason: AsExpected Status: True Type: ValidArchPlatform Last Transition Time: 2025-04-01T19:57:51Z Observed Generation: 3 Reason: AsExpected Status: False Type: UpdatingConfig Last Transition Time: 2025-04-01T18:27:19Z Observed Generation: 3 Reason: AsExpected Status: False Type: UpdatingVersion Last Transition Time: 2025-04-01T18:27:21Z Observed Generation: 3 Reason: AsExpected Status: False Type: UpdatingPlatformMachineTemplate Replicas: 3 Version: 4.14.49 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning ReconcileError 123m nodepool-controller error getting token: failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to create repository client for https://quay.io: endpoint "https://quay.io" does not support v2 API (got 502 Bad Gateway) Warning ReconcileError 119m nodepool-controller error getting token: failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #4 from manifest list: received unexpected HTTP status: 502 Bad Gateway Warning ReconcileError 108m (x2 over 118m) nodepool-controller error getting token: failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: received unexpected HTTP status: 502 Bad Gateway Warning ReconcileError 108m nodepool-controller error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to create repository client for https://quay.io: endpoint "https://quay.io" does not support v2 API (got 502 Bad Gateway) Warning ReconcileError 108m nodepool-controller failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #4 from manifest list: received unexpected HTTP status: 502 Bad Gateway Warning ReconcileError 94m nodepool-controller error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #2 from manifest list: received unexpected HTTP status: 502 Bad Gateway Warning ReconcileError 92m (x2 over 4h) nodepool-controller error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #4 from manifest list: received unexpected HTTP status: 502 Bad Gateway Warning ReconcileError 87m nodepool-controller failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to create repository client for https://quay.io: Get "https://quay.io/v2/": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Warning ReconcileError 87m (x2 over 4h18m) nodepool-controller error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #3 from manifest list: received unexpected HTTP status: 502 Bad Gateway Warning ReconcileError 87m nodepool-controller failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #1 from manifest list: received unexpected HTTP status: 502 Bad Gateway Warning ReconcileError 81m (x3 over 87m) nodepool-controller error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: Get "https://quay.io/v2/openshift-release-dev/ocp-v4.0-art-dev/manifests/sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb": received unexpected HTTP status: 502 Bad Gateway Warning ReconcileError 79m (x2 over 95m) nodepool-controller error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain image configuration for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: cannot retrieve image configuration for manifest sha256:b57ccde580b7e2d5383d006da07acebc099ad181bb877c4c705d3cdf0a535ee9 in manifest list sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: received unexpected HTTP status: 502 Bad Gateway Warning ReconcileError 78m (x6 over 82m) nodepool-controller (combined from similar events): error getting token: failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to create repository client for https://quay.io: Get "https://quay.io/v2/": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Warning ReconcileError 58m (x2 over 58m) nodepool-controller error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain image configuration for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: cannot retrieve image configuration for manifest sha256:b57ccde580b7e2d5383d006da07acebc099ad181bb877c4c705d3cdf0a535ee9 in manifest list sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: content integrity error: the blob retrieved with digest sha256:a4261a9368560c023ff3e66869a0f4cf6c50d8ca73fe4cefeca9458ce08c9ee6 does not match the digest calculated from the content sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 Warning ReconcileError 58m nodepool-controller failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #1 from manifest list: error unmarshalling content: unexpected end of JSON input Warning ReconcileError 58m nodepool-controller error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #4 from manifest list: error unmarshalling content: unexpected end of JSON input Warning ReconcileError 58m (x2 over 58m) nodepool-controller error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: error unmarshalling content: unexpected end of JSON input Warning ReconcileError 58m (x2 over 58m) nodepool-controller error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #3 from manifest list: error unmarshalling content: unexpected end of JSON input Warning ReconcileError 58m nodepool-controller error getting token: failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain image configuration for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: cannot retrieve image configuration for manifest sha256:b57ccde580b7e2d5383d006da07acebc099ad181bb877c4c705d3cdf0a535ee9 in manifest list sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: content integrity error: the blob retrieved with digest sha256:a4261a9368560c023ff3e66869a0f4cf6c50d8ca73fe4cefeca9458ce08c9ee6 does not match the digest calculated from the content sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 Warning ReconcileError 57m (x3 over 59m) nodepool-controller error getting token: failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #1 from manifest list: error unmarshalling content: unexpected end of JSON input Warning ReconcileError 57m nodepool-controller failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #2 from manifest list: error unmarshalling content: unexpected end of JSON input Warning ReconcileError 57m nodepool-controller failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #3 from manifest list: error unmarshalling content: unexpected end of JSON input Warning ReconcileError 29m nodepool-controller error getting token: failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain image configuration for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: cannot retrieve image configuration for manifest sha256:b57ccde580b7e2d5383d006da07acebc099ad181bb877c4c705d3cdf0a535ee9 in manifest list sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: received unexpected HTTP status: 502 Bad Gateway Warning ReconcileError 14m (x2 over 20h) nodepool-controller error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #1 from manifest list: received unexpected HTTP status: 502 Bad Gateway Warning ReconcileError 13m nodepool-controller failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #2 from manifest list: received unexpected HTTP status: 502 Bad Gateway Warning ReconcileError 12m (x3 over 7h33m) nodepool-controller failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #3 from manifest list: received unexpected HTTP status: 502 Bad Gateway Warning ReconcileError 8m32s (x2 over 86m) nodepool-controller error getting token: failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #3 from manifest list: received unexpected HTTP status: 502 Bad Gateway Warning ReconcileError 8m27s (x3 over 19h) nodepool-controller error getting token: failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #2 from manifest list: received unexpected HTTP status: 502 Bad Gateway