Uploaded image for project: 'MicroShift'
  1. MicroShift
  2. USHIFT-5864

Offline test ai-model-serving fails after upgrading crio to 1.33

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • 0
    • None
    • None
    • uShift Sprint 273, uShift Sprint 274, uShift Sprint 275, uShift Sprint 277, uShift Sprint 278
    • None
    • None
    • None

      Description of problem:

      After upgrading CRI-O to 1.33 on Jun 12th 2025, the nightly ai-model-serving offline test started failing:
      
      https://prow.ci.openshift.org/job-history/gs/test-platform-results/logs/periodic-ci-openshift-microshift-release-4.20-periodics-e2e-aws-tests-bootc-nightly
      
      In the test: after rebooting the host to verify if all works as intended, the openvino-resnet50-predictor Pod does not come up (all other Pods do). Describing the Pod shows following events:
      
      ----
        Warning  FailedCreatePodContainer  4m21s (x2 over 4m22s)  kubelet            unable to ensure pod container exists: failed to create container for [kubepods burstable pod87f08872-82f7-4e59-9a4d-2842dd1926bd] : unable to start unit "kubepods-burstable-pod87f08872_82f7_4e59_9a4d_2842dd1926bd.slice" (properties [{Name:Description Value:"libcontainer container kubepods-burstable-pod87f08872_82f7_4e59_9a4d_2842dd1926bd.slice"} {Name:Wants Value:["kubepods-burstable.slice"]} {Name:MemoryAccounting Value:true} {Name:CPUAccounting Value:true} {Name:IOAccounting Value:true} {Name:TasksAccounting Value:true} {Name:DefaultDependencies Value:false}]): Transaction for kubepods-burstable-pod87f08872_82f7_4e59_9a4d_2842dd1926bd.slice/start is destructive (shutdown.target has \'start\' job queued, but \'stop\' is included in transaction).
        Warning  FailedCreatePodContainer  4m10s                  kubelet            unable to ensure pod container exists: failed to create container for [kubepods burstable pod87f08872-82f7-4e59-9a4d-2842dd1926bd] : unable to start unit "kubepods-burstable-pod87f08872_82f7_4e59_9a4d_2842dd1926bd.slice" (properties [{Name:Description Value:"libcontainer container kubepods-burstable-pod87f08872_82f7_4e59_9a4d_2842dd1926bd.slice"} {Name:Wants Value:["kubepods-burstable.slice"]} {Name:MemoryAccounting Value:true} {Name:CPUAccounting Value:true} {Name:IOAccounting Value:true} {Name:TasksAccounting Value:true} {Name:DefaultDependencies Value:false}]): Transaction for kubepods-burstable-pod87f08872_82f7_4e59_9a4d_2842dd1926bd.slice/start is destructive (systemd-reboot.service has \'start\' job queued, but \'stop\' is included in transaction).
        Warning  NetworkNotReady           3m6s (x5 over 3m12s)   kubelet            network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/cni/net.d/. Has your network provider started?
        Normal   SandboxChanged            2s (x16 over 3m4s)     kubelet            Pod sandbox changed, it will be killed and re-created.'
      ------

      Version-Release number of selected component (if applicable):

       

      How reproducible:

      Almost always

      Steps to Reproduce:

      1. Run el96-src@ai-model-serving-offline scenario
      

      Actual results:

      After VM reboot, the openvino-resnet-predictor does not come up due to problems with kubepods-burst.slice, but it's probably something deeper.

      Expected results:

      Test should pass

      Additional info:

       

              ggiguash@redhat.com Gregory Giguashvili
              pmatusza@redhat.com Patryk Matuszak
              None
              None
              None
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: