Uploaded image for project: 'OpenShift Request For Enhancement'
  1. OpenShift Request For Enhancement
  2. RFE-5225

Improve handling of attached storage

XMLWordPrintable

    • Icon: Feature Request Feature Request
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • openshift-4.12, openshift-4.13, openshift-4.14
    • Node
    • False
    • None
    • False
    • Not Selected
    • 0
    • 0% 0%

      1. Proposed title of this feature request

      Improve handling of storage attached to pods
      2. What is the nature and description of the request?

      OpenShift (and Kubernetes) is generally bad at dealing with storage, and especially with storage that it owns and controls.

      The general Linux behavior is that, if you have dirty data in your page cache (ie, have saved a file but not fsynced it to disk), it must flush out that data before turning off. If the data has to be written out to a network filesystem, that means the filesystem (and the network!) must be accessible — if they aren't, Linux hangs the shutdown until it can write out its data. This is just basic data safety — if it can't write out its data, and it restarts, it has lost data which the user may care about.

      You can override this with the -f flag and tell it to ignore any failures.

      OpenShift could detect that it has inaccessible storage, and that it wants to do a restart despite that, though I'm not sure what the right heuristics would be. It could do a forced shutdown after some time delay (though again, the heuristics are hard). It could present an option for admins to do a forced shutdown (which seems like the easiest and safest). But these are all things that need to be handled by the some layer or human above the storage system and above Linux, because the storage system's job and Linux's job is to take data and make it safe, and this is a situation where it cannot do so without external action.

      Note that this problem could also be resolved by returning the storage system to operation, but frequently it comes up when there is a problem (with Ceph or elsewhere) or upgrade that is causing the admin to reboot the entire cluster. (In that specific case, it would be great if OpenShift turned off application pods and unmounted before it turned off the Ceph pods, or only turned off one Ceph host at a time because that should allow it to keep operating, but this is one of the ways that Kubernetes handles storage poorly — the idea that it manages the storage is pretty foreign to it. 

      3. Why does the customer need this? (List the business requirements here)

      Multiple customers already faced issues that a node could not be restarted due to issues with storage during node reboot. Draining the node should take care for also unattaching all attached storage in the correct order. In the current situation the node reboot is simply stuck. There is a Ceph Bugzilla open https://bugzilla.redhat.com/show_bug.cgi?id=2190503 which based on Ceph engineers feedback can't be solved from the storage providers end.

      This causes effort and uncertainty about a potential loss of data for customers.
      4. List any affected packages or components.

            gausingh@redhat.com Gaurav Singh
            rhn-support-afaulhab Anne Faulhaber
            Gaurav Singh
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: