Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-12875

SNO OCP upgrade from 4.12 to 4.13 failed due to node-tuning operator is not available - tuned pod stuck at Terminating

XMLWordPrintable

    • Important
    • No
    • Proposed
    • False
    • Hide

      None

      Show
      None
    • Hide
      3/22: Green as a tentative fix is posted
      3/14: telco reviewed
      Show
      3/22: Green as a tentative fix is posted 3/14: telco reviewed

      This is a clone of issue OCPBUGS-11920. The following is the description of the original issue:
      —
      This is a clone of issue OCPBUGS-11031. The following is the description of the original issue:
      —
      This is a clone of issue OCPBUGS-10217. The following is the description of the original issue:
      —
      Description of problem:

      OCP upgrade from 4.12.6 to 4.13 nightly failed due to tuned operator is not available. 


$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.12.7    True        True          155m    Unable to apply 4.13.0-0.nightly-2023-03-13-122248: the cluster operator node-tuning is not available


$ oc get co 
...
node-tuning                                4.13.0-0.nightly-2023-03-13-122248   False       True          False      126m    DaemonSet "tuned" has no available Pod(s)


Following events in Terminating tuned pod: 

$ oc get pods -n openshift-cluster-node-tuning-operator
NAME                                            READY   STATUS        RESTARTS   AGE
cluster-node-tuning-operator-5c458cc9d5-fp9zn   1/1     Running       0          136m
tuned-94tcq                                     0/1     Terminating   2          4h43m

$ oc describe pods -n openshift-cluster-node-tuning-operator tuned-94tcq
...
  Normal   Started            3h6m                 kubelet            Started container tuned
  Normal   Killing            126m                 kubelet            Stopping container tuned
  Warning  FailedPreStopHook  126m                 kubelet            Exec lifecycle hook ([/var/lib/tuned/bin/run stop]) for Container "tuned" in Pod "tuned-94tcq_openshift-cluster-node-tuning-operator(e12b341f-0683-4ccf-870c-bb3d24de073f)" failed - error: command '/var/lib/tuned/bin/run stop' exited with 1: openshift-tuned stop response: 
, message: "openshift-tuned stop response: \n"

      Version-Release number of selected component (if applicable):

      4.13

      How reproducible:

      Intermittent

      Steps to Reproduce:

      1. Upgrade SNO from OCP 4.12.7 to 4.13 nightly
2.
3.

      Actual results:

      Upgrade stuck at waiting for node-tuning operator to be available

$ oc get clusterversion NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS version   4.12.7    True        True          155m    Unable to apply 4.13.0-0.nightly-2023-03-13-122248: the cluster operator node-tuning is not available

      Expected results:

      Upgrade succeed

      Additional info:

      Workaround: manually reboot DUT

       

              jmencak Jiri Mencak
              openshift-crt-jira-prow OpenShift Prow Bot
              Liquan Cui Liquan Cui
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: