Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-39447

[OCP 4.15] "error getting ignition payload: failed to download binaries"

XMLWordPrintable

    • Important
    • None
    • Hypershift Sprint 259
    • 1
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Hide
      {product-rosa} hosted control planes (HCP) and {product-title} clusters will fail to add new nodes in MachinePool versions older than 4.15.23. As a result, some upgrades are blocked. To see what clusters are affected and the recommended workaround, see (link:https://access.redhat.com/articles/7085666[ROSA upgrade issue mitigation for HOSTEDCP-1941 ]). (link:https://issues.redhat.com/browse/OCPBUGS-39447[*OCPBUGS-39447*])
      ______________
      Fixes an issue where NodePools with version 4.15.22 or earlier with not scale or rollout with a HostedCluster version 4.15.23/4.16.4 or later
      Show
      {product-rosa} hosted control planes (HCP) and {product-title} clusters will fail to add new nodes in MachinePool versions older than 4.15.23. As a result, some upgrades are blocked. To see what clusters are affected and the recommended workaround, see (link: https://access.redhat.com/articles/7085666 [ROSA upgrade issue mitigation for HOSTEDCP-1941 ]). (link: https://issues.redhat.com/browse/OCPBUGS-39447 [* OCPBUGS-39447 *]) ______________ Fixes an issue where NodePools with version 4.15.22 or earlier with not scale or rollout with a HostedCluster version 4.15.23/4.16.4 or later
    • Known Issue
    • In Progress

      This is a clone of issue OCPBUGS-39419. The following is the description of the original issue:
      —
      This is a clone of issue OCPBUGS-38794. The following is the description of the original issue:
      —
      Description of problem:

      HCP cluster is being updated but the nodepool is stuck updating:
~~~
NAME                   CLUSTER   DESIRED NODES   CURRENT NODES   AUTOSCALING   AUTOREPAIR   VERSION   UPDATINGVERSION   UPDATINGCONFIG   MESSAGE
nodepool-dev-cluster   dev       2               2               False         False        4.15.22   True              True
~~~

      Version-Release number of selected component (if applicable):

      Hosting OCP cluster 4.15
HCP 4.15.23

      How reproducible:

      N/A

      Steps to Reproduce:

          1.
    2.
    3.

      Actual results:

      Nodepool stuck in upgrade

      Expected results:

      Upgrade success

      Additional info:

      I have found this error repeating continually in the ignition-server pods:
~~~
{"level":"error","ts":"2024-08-20T09:02:19Z","msg":"Reconciler error","controller":"secret","controllerGroup":"","controllerKind":"Secret","Secret":{"name":"token-nodepool-dev-cluster-3146da34","namespace":"dev-dev"},"namespace":"dev-dev","name":"token-nodepool-dev-cluster-3146da34","reconcileID":"ec1f0a7f-1657-4245-99ef-c984977ff0f8","error":"error getting ignition payload: failed to download binaries: failed to extract image file: failed to extract image file: file not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}

{"level":"info","ts":"2024-08-20T09:02:20Z","logger":"get-payload","msg":"discovered machine-config-operator image","image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f3b55cc8f88b9e6564fe6ad0bc431cd7270c0586a06d9b4a19ff2b518c461ede"}
{"level":"info","ts":"2024-08-20T09:02:20Z","logger":"get-payload","msg":"created working directory","dir":"/payloads/get-payload4089452863"}

{"level":"info","ts":"2024-08-20T09:02:28Z","logger":"get-payload","msg":"extracted image-references","time":"8s"}

{"level":"info","ts":"2024-08-20T09:02:38Z","logger":"get-payload","msg":"extracted templates","time":"10s"}
{"level":"info","ts":"2024-08-20T09:02:38Z","logger":"image-cache","msg":"retrieved cached file","imageRef":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f3b55cc8f88b9e6564fe6ad0bc431cd7270c0586a06d9b4a19ff2b518c461ede","file":"usr/lib/os-release"}
{"level":"info","ts":"2024-08-20T09:02:38Z","logger":"get-payload","msg":"read os-release","mcoRHELMajorVersion":"8","cpoRHELMajorVersion":"9"}
{"level":"info","ts":"2024-08-20T09:02:38Z","logger":"get-payload","msg":"copying file","src":"usr/bin/machine-config-operator.rhel9","dest":"/payloads/get-payload4089452863/bin/machine-config-operator"}
~~~

            sjenning Seth Jennings
            openshift-crt-jira-prow OpenShift Prow Bot
            He Liu He Liu
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: