Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-27283

E2E test: node failure and PVC remediation

XMLWordPrintable

    • Moderate

      Automate E2E test for PodRemediator remediation (node failure → PVC deleted → workload rescheduled)

      Description:

      Implement an automated end-to-end test that validates PodRemediator behaviour when a node fails. The test should: create a stateful workload (e.g. StatefulSet or single Pod) with a local PVC on a chosen worker node; simulate node failure (e.g. virsh destroy of the worker VM from a jumphost) so the node becomes NotReady; wait for NHC/SNR to detect the failure and for PodRemediator to reconcile; verify that PodRemediator deletes the PVC bound to that node; verify that the workload is recreated on another node (with a new PVC). Automation can be implemented as an Ansible playbook, shell script, or both, and should be runnable from the repo. Optionally, the test restores the node (e.g. virsh start) and performs cleanup. Document prerequisites (cluster with PodRemediator and NHC/SNR, jumphost with virsh and SSH to the node’s host, oc and kubeconfig or API access), timeouts, and any required environment variables or extra vars (e.g. domain name or prefix for virsh, oc path).

      Acceptance criteria:

      • The E2E scenario is runnable from the repo (e.g. script or playbook invoked from repo root or a test directory).
      • The test creates a stateful workload with a local PVC, simulates node failure, and asserts that the PVC is deleted and the workload is recreated on another node.
      • Timeouts (e.g. for node NotReady, PVC deletion, pod reschedule) are configurable or documented.
      • Prerequisites and how to run the test (including jumphost, oc, virsh domain/prefix) are documented.
      • Optional: test restores the failed node (e.g. virsh start) and documents cleanup steps.

              rhn-support-aromito Antonio Romito
              rhn-support-aromito Antonio Romito
              rhos-dfg-pidone
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: