Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-15387

The node's image is deleted when the node is an unexpected reboot.

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Minor Minor
    • None
    • 4.13.z
    • Node / CRI-O
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • No
    • None
    • None
    • None
    • None
    • Customer Facing
    • Red Hat OpenShift Container Platform
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      A customer reported that when a node on ESXi is rebooted with an ESXi machine, the images that existed before the reboot are lost after the reboot. The specific procedure is to restart ESXi and check with podman images after the node is booted.
      
      I would like to ask if this is the expected cleaning up of the developer.

      Version-Release number of selected component (if applicable):

      Customer's environment is 4.11 on vSphere.
      My environment is 4.13 on AWS and I reproduced the following steps

      How reproducible:

      I did not reproduce this by simply stopping the instance in my AWS verification environment. However, I was able to reproduce it by running echo c > /proc/sysrq-trigger to intentionally cause kernel panic and then reboot.

      Steps to Reproduce:

      1.Deploy an appropriate pod to a node, and then reduce the number of replicas or otherwise make the pod unscheduled. In this test, httpd was deployed.
      
      2.Check the images using podman images. Obtain other ls results as well. 
      #podman images --all
      REPOSITORY                                                        TAG         IMAGE ID      CREATED      SIZE
      image-registry.openshift-image-registry.svc:5000/openshift/httpd  <none>      4c2e3e6ed803  3 days ago   454 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      781a37f1fe2f  2 weeks ago  453 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      54541c943861  2 weeks ago  455 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      3976fb9d3385  2 weeks ago  549 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      deda63aac062  3 weeks ago  622 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      06c3508474ed  3 weeks ago  806 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      2e93e1fa02de  3 weeks ago  365 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      2bdce6613e65  3 weeks ago  508 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      5fd99b169b27  3 weeks ago  767 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      530aaab5f11b  3 weeks ago  916 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      e7d9a027abca  3 weeks ago  456 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      f17170cae8db  3 weeks ago  454 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      65df008b154e  3 weeks ago  484 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      43f894469415  3 weeks ago  384 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      c76f55db8fbb  3 weeks ago  347 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      8d66687d9781  3 weeks ago  404 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      11e33901d661  3 weeks ago  374 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      c5c53d84e7fe  3 weeks ago  454 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      1e91c9731ae4  3 weeks ago  417 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      5c5a5dd41dc1  3 weeks ago  401 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      06a2691e725c  3 weeks ago  446 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      d7841824be9b  3 weeks ago  380 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      d298c30a25d0  3 weeks ago  366 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      838ebd6278bd  3 weeks ago  431 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      cb9e81fdf506  3 weeks ago  437 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      2a2a90c629d8  3 weeks ago  342 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      9dede6d5713e  3 weeks ago  617 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      39f285a6e873  3 weeks ago  415 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      599e0f50ede7  3 weeks ago  590 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      7edeff8495a6  3 weeks ago  382 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      3657b7793308  3 weeks ago  368 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev                    <none>      f5189c73bf7e  3 weeks ago  331 MB
      
      3. Stopping after letting os panic.
      #echo c > /proc/sysrq-triggerStop 
      
      The node instance in the AWS console after executing the above.
      
      4. Booting up the node
      Confirm that the node has become NotReady and is also stopped in the AWS console article, after that start it up.5. Confirm information as in step 2.
      
      #podman images --all
      REPOSITORY                                      TAG         IMAGE ID      CREATED      SIZE
      quay.io/openshift-release-dev/ocp-v4.0-art-dev  <none>      deda63aac062  3 weeks ago  622 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev  <none>      06c3508474ed  3 weeks ago  806 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev  <none>      2e93e1fa02de  3 weeks ago  365 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev  <none>      2bdce6613e65  3 weeks ago  508 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev  <none>      5fd99b169b27  3 weeks ago  767 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev  <none>      530aaab5f11b  3 weeks ago  916 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev  <none>      e7d9a027abca  3 weeks ago  456 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev  <none>      f17170cae8db  3 weeks ago  454 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev  <none>      c76f55db8fbb  3 weeks ago  347 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev  <none>      1e91c9731ae4  3 weeks ago  417 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev  <none>      06a2691e725c  3 weeks ago  446 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev  <none>      d7841824be9b  3 weeks ago  380 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev  <none>      d298c30a25d0  3 weeks ago  366 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev  <none>      9dede6d5713e  3 weeks ago  617 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev  <none>      599e0f50ede7  3 weeks ago  590 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev  <none>      7edeff8495a6  3 weeks ago  382 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev  <none>      3657b7793308  3 weeks ago  368 MB
      quay.io/openshift-release-dev/ocp-v4.0-art-dev  <none>      f5189c73bf7e  3 weeks ago  331 MB

      Actual results:

      I attach on test1.tar including results.
      The following output is recorded before and after the reboot.
      
      # podman images --all
      # ls -lR /var/lib/containers/ --ignore="overlay"

      Expected results:

      I would like to know if this is the expected behavior, as it appears to be a non-gracefull shutdown that cleans up all images. If it is intended, it should output some logs meaning clean up.

      Additional info:

       

              aos-node@redhat.com Node Team Bot Account
              rhn-support-rnoma Ryoji Noma
              None
              None
              David Darrah David Darrah (Inactive)
              None
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: