Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-54529

NodePool failure to reconcile due to error obtaining token and reaching ignitioin

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Undefined Undefined
    • None
    • 4.14.z
    • HyperShift / Agent
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

          Our Hosted Cluster is having trouble reaching the ignition endpoint. The NodePool shows ReconcileError and the HO logs show the same errors as well. Here's the NodePool
      apiVersion: hypershift.openshift.io/v1beta1 kind: NodePool metadata:   annotations:     hypershift.openshift.io/nodePoolCurrentConfig: 62484d3e     hypershift.openshift.io/nodePoolCurrentConfigVersion: d7a2fb4d     hypershift.openshift.io/nodePoolPlatformMachineTemplate: nodepool-dev-gpc-zero-dc-rdu2-1-c880c958   creationTimestamp: "2025-03-28T17:13:11Z"   finalizers:   - hypershift.openshift.io/finalizer   generation: 3   name: nodepool-dev-gpc-zero-dc-rdu2-1   namespace: dev-gpc-zero-dc-rdu2   ownerReferences:   - apiVersion: hypershift.openshift.io/v1beta1     kind: HostedCluster     name: dev-gpc-zero-dc-rdu2     uid: ef4c1b36-90b6-49ad-809d-88bafd6e6605   resourceVersion: "37304733"   uid: 52611b50-2009-49b0-a9dc-adb54217a9a8 spec:   arch: amd64   clusterName: dev-gpc-zero-dc-rdu2   management:     autoRepair: false     replace:       rollingUpdate:         maxSurge: 1         maxUnavailable: 0       strategy: RollingUpdate     upgradeType: InPlace   platform:     agent:       agentLabelSelector: {}     type: Agent   release:     image: quay.io/openshift-release-dev/ocp-release:4.14.49-multi   replicas: 3 status:   conditions:   - lastTransitionTime: "2025-03-28T17:13:13Z"     observedGeneration: 3     reason: AsExpected     status: "False"     type: AutoscalingEnabled   - lastTransitionTime: "2025-03-28T17:13:13Z"     observedGeneration: 3     reason: AsExpected     status: "True"     type: UpdateManagementEnabled   - lastTransitionTime: "2025-03-28T17:17:59Z"     message: 'Using release image: quay.io/openshift-release-dev/ocp-release:4.14.49-multi'     observedGeneration: 3     reason: AsExpected     status: "True"     type: ValidReleaseImage   - lastTransitionTime: "2025-04-01T18:23:54Z"     observedGeneration: 3     reason: AsExpected     status: "True"     type: ValidMachineConfig   - lastTransitionTime: "2025-04-01T19:10:23Z"     message: Payload generated successfully     observedGeneration: 3     reason: AsExpected     status: "True"     type: ValidGeneratedPayload   - lastTransitionTime: "2025-03-28T17:18:02Z"     observedGeneration: 3     reason: ignitionNotReached     status: "False"     type: ReachedIgnitionEndpoint   - lastTransitionTime: "2025-03-28T17:18:02Z"     observedGeneration: 3     reason: AsExpected     status: "True"     type: ValidTuningConfig   - lastTransitionTime: "2025-03-28T17:18:02Z"     message: Reconciliation active on resource     observedGeneration: 3     reason: ReconciliationActive     status: "True"     type: ReconciliationActive   - lastTransitionTime: "2025-03-28T19:36:38Z"     message: All is well     observedGeneration: 3     reason: AsExpected     status: "True"     type: AllMachinesReady   - lastTransitionTime: "2025-04-01T19:57:19Z"     message: All is well     observedGeneration: 3     reason: AsExpected     status: "True"     type: AllNodesHealthy   - lastTransitionTime: "2025-03-28T19:36:39Z"     observedGeneration: 3     reason: AsExpected     status: "True"     type: Ready   - lastTransitionTime: "2025-03-28T17:18:03Z"     observedGeneration: 3     reason: AsExpected     status: "False"     type: AutorepairEnabled   - lastTransitionTime: "2025-04-01T18:27:19Z"     observedGeneration: 3     reason: AsExpected     status: "True"     type: ValidArchPlatform   - lastTransitionTime: "2025-04-01T19:57:51Z"     observedGeneration: 3     reason: AsExpected     status: "False"     type: UpdatingConfig   - lastTransitionTime: "2025-04-01T18:27:19Z"     observedGeneration: 3     reason: AsExpected     status: "False"     type: UpdatingVersion   - lastTransitionTime: "2025-04-01T18:27:21Z"     observedGeneration: 3     reason: AsExpected     status: "False"     type: UpdatingPlatformMachineTemplate   replicas: 3   version: 4.14.49 
      And the logs of the hypershift-operator
      Defaulted container "operator" out of: operator, init-environment (init) Defaulted container "operator" out of: operator, init-environment (init) {"level":"info","ts":"2025-04-02T15:42:09Z","msg":"removing secret as it does not match the expected set of names","controller":"secret","controllerGroup":"","controllerKind":"Secret","Secret":{"name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2"},"namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2","name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","reconcileID":"0accfadd-1249-429b-bf1a-cf01dba19946","secret":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2/token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","nodePool":"dev-gpc-zero-dc-rdu2/nodepool-dev-gpc-zero-dc-rdu2-1","options":["token-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d","user-data-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d"],"valid":false} {"level":"info","ts":"2025-04-02T15:42:12Z","msg":"removing secret as it does not match the expected set of names","controller":"secret","controllerGroup":"","controllerKind":"Secret","Secret":{"name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2"},"namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2","name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","reconcileID":"f1cc76d5-7508-4384-bc23-69ec5fe4e018","secret":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2/token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","nodePool":"dev-gpc-zero-dc-rdu2/nodepool-dev-gpc-zero-dc-rdu2-1","options":["token-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d","user-data-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d"],"valid":false} {"level":"info","ts":"2025-04-02T15:42:13Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"} {"level":"info","ts":"2025-04-02T15:42:13Z","msg":"removing secret as it does not match the expected set of names","controller":"secret","controllerGroup":"","controllerKind":"Secret","Secret":{"name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2"},"namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2","name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","reconcileID":"03d843f8-631b-403d-a6e2-a7924725a11f","secret":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2/token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","nodePool":"dev-gpc-zero-dc-rdu2/nodepool-dev-gpc-zero-dc-rdu2-1","options":["token-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d","user-data-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d"],"valid":false} {"level":"info","ts":"2025-04-02T15:42:14Z","msg":"removing secret as it does not match the expected set of names","controller":"secret","controllerGroup":"","controllerKind":"Secret","Secret":{"name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-f46f5ae4","namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2"},"namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2","name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-f46f5ae4","reconcileID":"279561ab-646b-4aac-a143-0097960ee621","secret":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2/token-nodepool-dev-gpc-zero-dc-rdu2-1-f46f5ae4","nodePool":"dev-gpc-zero-dc-rdu2/nodepool-dev-gpc-zero-dc-rdu2-1","options":["token-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d","user-data-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d"],"valid":false} {"level":"info","ts":"2025-04-02T15:42:14Z","msg":"removing secret as it does not match the expected set of names","controller":"secret","controllerGroup":"","controllerKind":"Secret","Secret":{"name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2"},"namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2","name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","reconcileID":"83d02462-d3f7-41b0-a026-8b5ceddfaed0","secret":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2/token-nodepool-dev-gpc-zero-dc-rdu2-1-b15a1ead","nodePool":"dev-gpc-zero-dc-rdu2/nodepool-dev-gpc-zero-dc-rdu2-1","options":["token-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d","user-data-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d"],"valid":false} {"level":"info","ts":"2025-04-02T15:42:16Z","msg":"removing secret as it does not match the expected set of names","controller":"secret","controllerGroup":"","controllerKind":"Secret","Secret":{"name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-f46f5ae4","namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2"},"namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2","name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-f46f5ae4","reconcileID":"8a3ba627-1cd6-4acc-87f0-c5d73c9c7701","secret":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2/token-nodepool-dev-gpc-zero-dc-rdu2-1-f46f5ae4","nodePool":"dev-gpc-zero-dc-rdu2/nodepool-dev-gpc-zero-dc-rdu2-1","options":["token-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d","user-data-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d"],"valid":false} {"level":"info","ts":"2025-04-02T15:42:17Z","msg":"removing secret as it does not match the expected set of names","controller":"secret","controllerGroup":"","controllerKind":"Secret","Secret":{"name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-609bcfb1","namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2"},"namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2","name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-609bcfb1","reconcileID":"113831bb-7bc3-4d9e-b53f-6f9073ca86a6","secret":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2/token-nodepool-dev-gpc-zero-dc-rdu2-1-609bcfb1","nodePool":"dev-gpc-zero-dc-rdu2/nodepool-dev-gpc-zero-dc-rdu2-1","options":["token-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d","user-data-nodepool-dev-gpc-zero-dc-rdu2-1-d7a2fb4d"],"valid":false} {"level":"error","ts":"2025-04-02T15:42:19Z","msg":"Reconciler error","controller":"secret","controllerGroup":"","controllerKind":"Secret","Secret":{"name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-609bcfb1","namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2"},"namespace":"dev-gpc-zero-dc-rdu2-dev-gpc-zero-dc-rdu2","name":"token-nodepool-dev-gpc-zero-dc-rdu2-1-609bcfb1","reconcileID":"225fcd0f-ce96-4b5a-8e57-a046dd3ec47c","error":"failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain image configuration for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: cannot retrieve image configuration for manifest sha256:b57ccde580b7e2d5383d006da07acebc099ad181bb877c4c705d3cdf0a535ee9 in manifest list sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: received unexpected HTTP status: 502 Bad Gateway","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:224"} {"level":"info","ts":"2025-04-02T15:42:20Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"} {"level":"info","ts":"2025-04-02T15:43:40Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"} {"level":"info","ts":"2025-04-02T15:43:40Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"} {"level":"info","ts":"2025-04-02T15:43:41Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"} {"level":"info","ts":"2025-04-02T15:43:56Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"} {"level":"info","ts":"2025-04-02T15:44:11Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"} {"level":"info","ts":"2025-04-02T15:44:26Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"} {"level":"info","ts":"2025-04-02T15:44:40Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"} {"level":"info","ts":"2025-04-02T15:44:40Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"} {"level":"info","ts":"2025-04-02T15:44:41Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"} {"level":"info","ts":"2025-04-02T15:44:56Z","msg":"cannot retrieve the number of vCPUs for nodepool-dev-gpc-zero-dc-rdu2-1 node pool as its platform is not supported (supported platforms: AWS)"}

      Version-Release number of selected component (if applicable):

      Using HyperShift image http://quay.io/acm-d/rhtap-hypershift-operator@sha256:ae7db7ff02251e32609f445905530b81adb991faa600aa5ee6a0e792e1ef05a0  
      Using MCE multicluster-engine.v2.4.8
      HostedCluster running 4.14.49

      How reproducible:

          

      Steps to Reproduce:

          1.
          2.
          3.
          

      Actual results:

          

      Expected results:

          I expected to see the NodePool not in a Reconcile Loop and the HO not to have those errors 

      Additional info:

          > oc describe np nodepool-dev-gpc-zero-dc-rdu2-1
         Name:         nodepool-dev-gpc-zero-dc-rdu2-1 Namespace:    dev-gpc-zero-dc-rdu2 Labels:       <none> Annotations:  hypershift.openshift.io/nodePoolCurrentConfig: 62484d3e               hypershift.openshift.io/nodePoolCurrentConfigVersion: d7a2fb4d               hypershift.openshift.io/nodePoolPlatformMachineTemplate: nodepool-dev-gpc-zero-dc-rdu2-1-c880c958 API Version:  hypershift.openshift.io/v1beta1 Kind:         NodePool Metadata:   Creation Timestamp:  2025-03-28T17:13:11Z   Finalizers:     hypershift.openshift.io/finalizer   Generation:  3   Owner References:     API Version:     hypershift.openshift.io/v1beta1     Kind:            HostedCluster     Name:            dev-gpc-zero-dc-rdu2     UID:             ef4c1b36-90b6-49ad-809d-88bafd6e6605   Resource Version:  37304733   UID:               52611b50-2009-49b0-a9dc-adb54217a9a8 Spec:   Arch:          amd64   Cluster Name:  dev-gpc-zero-dc-rdu2   Management:     Auto Repair:  false     Replace:       Rolling Update:         Max Surge:        1         Max Unavailable:  0       Strategy:           RollingUpdate     Upgrade Type:         InPlace   Platform:     Agent:       Agent Label Selector:     Type:  Agent   Release:     Image:   quay.io/openshift-release-dev/ocp-release:4.14.49-multi   Replicas:  3 Status:   Conditions:     Last Transition Time:  2025-03-28T17:13:13Z     Observed Generation:   3     Reason:                AsExpected     Status:                False     Type:                  AutoscalingEnabled     Last Transition Time:  2025-03-28T17:13:13Z     Observed Generation:   3     Reason:                AsExpected     Status:                True     Type:                  UpdateManagementEnabled     Last Transition Time:  2025-03-28T17:17:59Z     Message:               Using release image: quay.io/openshift-release-dev/ocp-release:4.14.49-multi     Observed Generation:   3     Reason:                AsExpected     Status:                True     Type:                  ValidReleaseImage     Last Transition Time:  2025-04-01T18:23:54Z     Observed Generation:   3     Reason:                AsExpected     Status:                True     Type:                  ValidMachineConfig     Last Transition Time:  2025-04-01T19:10:23Z     Message:               Payload generated successfully     Observed Generation:   3     Reason:                AsExpected     Status:                True     Type:                  ValidGeneratedPayload     Last Transition Time:  2025-03-28T17:18:02Z     Observed Generation:   3     Reason:                ignitionNotReached     Status:                False     Type:                  ReachedIgnitionEndpoint     Last Transition Time:  2025-03-28T17:18:02Z     Observed Generation:   3     Reason:                AsExpected     Status:                True     Type:                  ValidTuningConfig     Last Transition Time:  2025-03-28T17:18:02Z     Message:               Reconciliation active on resource     Observed Generation:   3     Reason:                ReconciliationActive     Status:                True     Type:                  ReconciliationActive     Last Transition Time:  2025-03-28T19:36:38Z     Message:               All is well     Observed Generation:   3     Reason:                AsExpected     Status:                True     Type:                  AllMachinesReady     Last Transition Time:  2025-04-01T19:57:19Z     Message:               All is well     Observed Generation:   3     Reason:                AsExpected     Status:                True     Type:                  AllNodesHealthy     Last Transition Time:  2025-03-28T19:36:39Z     Observed Generation:   3     Reason:                AsExpected     Status:                True     Type:                  Ready     Last Transition Time:  2025-03-28T17:18:03Z     Observed Generation:   3     Reason:                AsExpected     Status:                False     Type:                  AutorepairEnabled     Last Transition Time:  2025-04-01T18:27:19Z     Observed Generation:   3     Reason:                AsExpected     Status:                True     Type:                  ValidArchPlatform     Last Transition Time:  2025-04-01T19:57:51Z     Observed Generation:   3     Reason:                AsExpected     Status:                False     Type:                  UpdatingConfig     Last Transition Time:  2025-04-01T18:27:19Z     Observed Generation:   3     Reason:                AsExpected     Status:                False     Type:                  UpdatingVersion     Last Transition Time:  2025-04-01T18:27:21Z     Observed Generation:   3     Reason:                AsExpected     Status:                False     Type:                  UpdatingPlatformMachineTemplate   Replicas:                3   Version:                 4.14.49 Events:   Type     Reason          Age                  From                 Message   ----     ------          ----                 ----                 -------   Warning  ReconcileError  123m                 nodepool-controller  error getting token: failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to create repository client for https://quay.io: endpoint "https://quay.io" does not support v2 API (got 502 Bad Gateway)   Warning  ReconcileError  119m                 nodepool-controller  error getting token: failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #4 from manifest list: received unexpected HTTP status: 502 Bad Gateway   Warning  ReconcileError  108m (x2 over 118m)  nodepool-controller  error getting token: failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: received unexpected HTTP status: 502 Bad Gateway   Warning  ReconcileError  108m                 nodepool-controller  error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to create repository client for https://quay.io: endpoint "https://quay.io" does not support v2 API (got 502 Bad Gateway)   Warning  ReconcileError  108m                 nodepool-controller  failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #4 from manifest list: received unexpected HTTP status: 502 Bad Gateway   Warning  ReconcileError  94m                  nodepool-controller  error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #2 from manifest list: received unexpected HTTP status: 502 Bad Gateway   Warning  ReconcileError  92m (x2 over 4h)     nodepool-controller  error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #4 from manifest list: received unexpected HTTP status: 502 Bad Gateway   Warning  ReconcileError  87m                  nodepool-controller  failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to create repository client for https://quay.io: Get "https://quay.io/v2/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)   Warning  ReconcileError  87m (x2 over 4h18m)  nodepool-controller  error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #3 from manifest list: received unexpected HTTP status: 502 Bad Gateway   Warning  ReconcileError  87m                  nodepool-controller  failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #1 from manifest list: received unexpected HTTP status: 502 Bad Gateway   Warning  ReconcileError  81m (x3 over 87m)    nodepool-controller  error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: Get "https://quay.io/v2/openshift-release-dev/ocp-v4.0-art-dev/manifests/sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb": received unexpected HTTP status: 502 Bad Gateway   Warning  ReconcileError  79m (x2 over 95m)    nodepool-controller  error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain image configuration for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: cannot retrieve image configuration for manifest sha256:b57ccde580b7e2d5383d006da07acebc099ad181bb877c4c705d3cdf0a535ee9 in manifest list sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: received unexpected HTTP status: 502 Bad Gateway   Warning  ReconcileError  78m (x6 over 82m)    nodepool-controller  (combined from similar events): error getting token: failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to create repository client for https://quay.io: Get "https://quay.io/v2/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)   Warning  ReconcileError  58m (x2 over 58m)    nodepool-controller  error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain image configuration for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: cannot retrieve image configuration for manifest sha256:b57ccde580b7e2d5383d006da07acebc099ad181bb877c4c705d3cdf0a535ee9 in manifest list sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: content integrity error: the blob retrieved with digest sha256:a4261a9368560c023ff3e66869a0f4cf6c50d8ca73fe4cefeca9458ce08c9ee6 does not match the digest calculated from the content sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855   Warning  ReconcileError  58m                  nodepool-controller  failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #1 from manifest list: error unmarshalling content: unexpected end of JSON input   Warning  ReconcileError  58m                  nodepool-controller  error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #4 from manifest list: error unmarshalling content: unexpected end of JSON input   Warning  ReconcileError  58m (x2 over 58m)    nodepool-controller  error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: error unmarshalling content: unexpected end of JSON input   Warning  ReconcileError  58m (x2 over 58m)    nodepool-controller  error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #3 from manifest list: error unmarshalling content: unexpected end of JSON input   Warning  ReconcileError  58m                  nodepool-controller  error getting token: failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain image configuration for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: cannot retrieve image configuration for manifest sha256:b57ccde580b7e2d5383d006da07acebc099ad181bb877c4c705d3cdf0a535ee9 in manifest list sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: content integrity error: the blob retrieved with digest sha256:a4261a9368560c023ff3e66869a0f4cf6c50d8ca73fe4cefeca9458ce08c9ee6 does not match the digest calculated from the content sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855   Warning  ReconcileError  57m (x3 over 59m)    nodepool-controller  error getting token: failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #1 from manifest list: error unmarshalling content: unexpected end of JSON input   Warning  ReconcileError  57m                  nodepool-controller  failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #2 from manifest list: error unmarshalling content: unexpected end of JSON input   Warning  ReconcileError  57m                  nodepool-controller  failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #3 from manifest list: error unmarshalling content: unexpected end of JSON input   Warning  ReconcileError  29m                  nodepool-controller  error getting token: failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain image configuration for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: cannot retrieve image configuration for manifest sha256:b57ccde580b7e2d5383d006da07acebc099ad181bb877c4c705d3cdf0a535ee9 in manifest list sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: received unexpected HTTP status: 502 Bad Gateway   Warning  ReconcileError  14m (x2 over 20h)    nodepool-controller  error getting token: failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #1 from manifest list: received unexpected HTTP status: 502 Bad Gateway   Warning  ReconcileError  13m                  nodepool-controller  failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #2 from manifest list: received unexpected HTTP status: 502 Bad Gateway   Warning  ReconcileError  12m (x3 over 7h33m)  nodepool-controller  failed to generate HAProxy raw config: failed to check if we manage haproxy ignition config: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #3 from manifest list: received unexpected HTTP status: 502 Bad Gateway   Warning  ReconcileError  8m32s (x2 over 86m)  nodepool-controller  error getting token: failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #3 from manifest list: received unexpected HTTP status: 502 Bad Gateway   Warning  ReconcileError  8m27s (x3 over 19h)  nodepool-controller  error getting token: failed to detect CPO capabilities: failed to look up image metadata for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: failed to obtain root manifest for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb: unable to retrieve source image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a4e76f6fac44d3d55f8d3c57a7b7324b530714d1a6e2df843bca0f992043decb manifest #2 from manifest list: received unexpected HTTP status: 502 Bad Gateway 

              cchun@redhat.com Crystal Chun
              rhit_amohamed Ahmed Mohamed
              None
              None
              Elsa Passaro Elsa Passaro
              None
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: