Please fill in the following information:

Issue description

First request: Add a timeout to the lab python scripts and/or ansible-playbooks

Second request: Add checks to see if the storage pods are healthy (several techniques can be adopted to solve this issue)

What happened:

The student attempted to start an activity - it really does not matter which one because this could impact any activity - and the lab start script runs into perpetuity. The student only contacted me after 30 minutes for assistance. If the lab script timed out after 5 or 8 minutes, then the student would not have waited time waiting for something to finish which never would.

The problem:

The DataVolume was not created by the lab start script because the the PVC went into a pending state as the Ceph RBD provisioner could not dynamically provision the storage (as revealed by the project events). Nothing untoward was reported in the events of the openshift-storage project. Restarting the provisioner pods resolved the issue.

This is likely a bug in the product.

Steps to reproduce:

Workaround:

Restarting the provisioner pods or restarting the master01,02,03 and worker01,02 VMs and waiting for OCP to come back online.

Expected result: