# RFE 2 HLD: Orchestrated cross-AZ workload re-instantiation using replicated Cinder volumes

## Proposal (High-Level Design)

### A. Orchestration entry points

- Heat provides new resources and actions to drive quiesce, wait for replication states, final sync, group promote, server create or update in the target AZ, volume attach, and start
- OpenStackClient offers high level commands to run the same sequence outside Heat
- Ansible and Terraform patterns and examples that call Cinder and Nova primitives in the required order

### B. Cutover workflow

1. Validate preconditions: replication is active for all volumes in the group and instances are ready to quiesce
2. Quiesce application or stop instance as required by policy
3. Trigger final sync and wait for completion
4. Promote the replicated group atomically in the target AZ
5. Recreate or update servers in the target AZ with the desired flavor and networks
6. Attach promoted volumes to the new instances in the correct mapping order
7. Start instances and run post-start health checks
8. Optional cleanup of source replication relationships

### C. Heat resource model

- Resource to wait on replication state transitions with timeouts and failure reasons
- Action to request final sync and to promote a group with `atomic=true`
- Composite workflow resource that chains quiesce, promote, server build or update, attach, and start
- Outputs that return new server ids, volume ids, and cutover timestamps

### D. Idempotency and failure handling

- Each step is retriable with clear checkpoints
- Rollback guidance when a step fails, including how to reattach to the original source if promotion is not completed
- Events and stack outputs suitable for CI pipelines and change records

### E. Status and observability

- Unified progress view showing replication percent, current action, and estimated time remaining
- Emission of Heat events and OpenStackClient progress lines for each milestone

### F. Scope exclusions

- No use of Glance import or export
- No new driver replication features, this RFE consumes capabilities delivered by RHOSRFE-209

## CLI / UX Sketch (OpenStackClient)

```bash
# End-to-end cutover using a helper command
openstack workload cutover   --group db-tier-group   --target-az AZ3   --final-sync   --atomic   --server-spec server-spec.yaml

# Stepwise alternative
openstack workload quiesce --server <id>
openstack volume group replication final-sync db-tier-group --wait
openstack volume group replication promote db-tier-group --atomic --break --wait
openstack server create --flavor m4.large --image None --block-device-mapping from=<promoted-volume-ids> --availability-zone AZ3
openstack server start <new-server-id>
```

## Policy and Security

- Orchestration actions are tenant visible but policy controlled
- Enforce detach or quiesce before final sync and promote unless allowed by operator policy and driver capability
- Restrict target AZ choices to operator-permitted pairs

## Dependencies and Driver Notes

- Depends on RFE 1 for tenant APIs and scheduler capability
- Requires Nova and Neutron to accept server re-creation or update in the target AZ with existing networks and security groups
- Requires OpenStackClient and Heat client updates

## Backwards Compatibility

- Tenants can continue using existing manual runbooks
- New orchestration resources are additive and optional
