Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-22169

4.14 nightly blocked by node reboot test failure

XMLWordPrintable

    • No
    • Approved
    • False
    • Hide

      None

      Show
      None
    • N/A
    • Release Note Not Required

      Description of problem:
      4.14.0-0.nightly-2023-10-19-115510
      periodic-ci-openshift-release-master-ci-4.14-upgrade-from-stable-4.13-e2e-gcp-ovn-rt-upgrade shows 10 out of 10 failures with

      [sig-node] nodes should not go unready after being upgraded and go unready only once expand_more
      

      Node tuning operator logs show failures to apply profile updates

      I1019 14:17:52.085071       1 status.go:303] 5/6 Profiles failed to be applied
      

      This is showing on 4.15 as well

      4.15 nightly has been blocked by consistent failure of the following test:
      
      [sig-node] nodes should not go unready after being upgraded and go unready only once 
      
      Take this job as an example:
      https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.15-upgrade-from-stable-4.14-e2e-gcp-ovn-rt-upgrade/1714719680779784192
      
      It looks like all worker nodes are rebooted twice based on the spyglass chart. 
      
      Corresponding logs can be seen here:
      https://grafana-loki.ci.openshift.org/explore?orgId=1&left=%7B%22datasource%22:%22PCEB727DF2F34084E%22,%22queries%22:%5B%7B%22expr%22:%22%7Binvoker%3D%5C%22openshift-internal-ci%2Fperiodic-ci-openshift-release-master-ci-4.15-upgrade-from-stable-4.14-e2e-gcp-ovn-rt-upgrade%2F1714719680779784192%5C%22,%20namespace%3D%5C%22openshift-machine-config-operator%5C%22%7D%20%22,%22refId%22:%22A%22,%22editorMode%22:%22code%22,%22queryType%22:%22range%22%7D%5D,%22range%22:%7B%22from%22:%221697660996000%22,%22to%22:%221697661356000%22%7D%7D
      
      

      Version-Release number of selected component (if applicable):

       

      How reproducible:

       

      Steps to Reproduce:

      1.
      2.
      3.
      

      Actual results:

       

      Expected results:

       

      Additional info:

       

            jmencak Jiri Mencak
            kenzhang@redhat.com Ken Zhang
            Liquan Cui Liquan Cui
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved: