ACM 2.15.1, MCE 2.10.1 zStream Regression Testing

Overview

Comprehensive zStream regression testing for ACM 2.15.1 and MCE 2.10.1 across:

• Regression (Non-UI/UI)

• Upgrade (Y-stream and Z-stream)

• Staging

• Post-Publishing test paths

• Advisory/Pre-Production validation

---

Environment Configuration

Hub Cluster

• Platform: vSphere

• ACM Version: 2.15.1

• MCE Version: 2.10.1

• OCP Version: 4.21

• Purpose: ACM Hub deployment and management

Spoke Clusters

1. Hive Cluster

• • Platform: AWS/Azure/GCP (any supported provider)

• • OCP Version: 4.21 or 4.20

• • Purpose: Cluster lifecycle management and provisioning

1. IKS Cluster

• • Platform: IBM Kubernetes Service (IKS)

• • OCP/K8s Version: Latest supported

• • Purpose: Multi-cloud policy and governance testing

Environment Matrix Table

---

Testing Scope

Overview

Defines testing scope for each component in the zStream release.
Components are evaluated for E2E regression testing, Y-stream upgrade testing, and Z-stream upgrade testing based on bugs requiring retest.

Testing Scope Matrix

Legend:

• E2E (Non-UI): All components require comprehensive E2E regression testing

• Y-Upgrade: Required for all components with bugs (minor version upgrade testing)

• Z-Upgrade: Required for all components with bugs (patch version upgrade testing)

• QE Focal: Primary QE contact responsible for component testing

• Comments: Additional testing notes or special requirements

Testing Logic:

• ✅ Yes = Testing required

• Components with bugs → Both Y-Upgrade and Z-Upgrade testing required

---

Bug Retest Requirements

Overview

This zStream release includes 24 bug fixes requiring comprehensive retesting across multiple components and squads. Each bug must be retested to verify the fix in the target environment.

---

Bugs Organized by Component

Application Lifecycle (1 bug)

QE Focal: yupchang@redhat.com | Squad: @acm-qe-workload-mgmt

---

Cluster Lifecycle (1 bug)

QE Focal: dhuynh@redhat.com | Squad: @acm-qe-workload-mgmt

---

Console (9 bugs)

QE Focal: dhuynh@redhat.com | Squad: @acm-qe-workload-mgmt

---

HyperShift (1 bug)

QE Focal: dhuynh@redhat.com | Squad: @acm-qe-workload-mgmt

---

Server Foundation (1 bug)

QE Focal: slai@redhat.com  | Squad: @acm-qe-serverfoundation

GRC (1 bug)

QE Focal: dho@redhat.com  | Squad: @acm-qe-grc

Infrastructure Operator (1 bug)

QE Focal: Unassigned | Squad: Unassigned

---

Observability (9 bugs)

QE Focal: mzardab@redhat.com | Squad: @observatorium-support

---

SiteConfig Operator (3 bugs)

QE Focal: Unassigned | Squad: Unassigned

---

Container Native Virtualization (1 bug)

QE Focal: Unassigned | Squad: Unassigned

---

Squad Task Assignment

Automated task creation will be performed for each squad to retest their respective component bugs:

Note: Individual retest tasks will be created as child tasks under this Epic, with each task assigned to the appropriate squad owner for bug verification.
Total Bugs: 24 across all components

---

CVE/Vulnerability Issues

• Total CVE/Vulnerability Issues: 0

• No CVE/Vulnerability security issues in this zStream release

---

Test Paths

1. Regression Testing

1.1 MCE 2.10.1 Regression Testing

Test Focus:

• MCE component functionality validation

• Multi-cluster engine core features

• Backend functionality testing

• CLI operations and automation

Key Test Areas:

• Cluster lifecycle operations

• Multi-cluster engine infrastructure

• Cluster provisioning and management

• Infrastructure components validation

1.2 ACM 2.15.1 Regression Testing

Non-UI Testing

Jenkins Pipeline: e2e_non-ui_test_pipeline

Test Focus:

• API validation and integration

• Backend functionality testing

• CLI operations and automation

• Policy and governance non-UI workflows

Key Test Areas:

• Cluster lifecycle operations

• Policy compliance and governance

• Application lifecycle management

• Observability and monitoring

UI Testing

Jenkins Pipeline: e2e_ui_test_pipeline

Test Focus:

• Console functionality and user workflows

• UI component validation

• Browser compatibility testing

• User experience validation

Key Test Areas:

• Cluster management UI

• Policy creation and management

• Application deployment UI

• Observability dashboards

---

2. Upgrade Testing

Jenkins Pipeline: e2e_upgrade_pipeline

Test Focus:

• ACM 2.15.1 version upgrade paths validation (Y-stream and Z-stream)

• Data migration and preservation

• Backward compatibility testing

• Configuration retention during upgrade

Important: MCE upgrade testing is NOT included in this test path

Key Test Areas:

• ACM Y-stream upgrade testing (minor version upgrade)

• ACM Z-stream upgrade testing (patch version upgrade)

• Managed cluster upgrades and compatibility

• Policy migration and continuity

• Application continuity across upgrades

• Rollback and recovery procedures

---

3. Staging Testing

3.1 MCE 2.10.1 Staging Testing

Test Focus:

• MCE staging environment deployment validation

• MCE release candidate testing

• Integration testing with staging infrastructure

• Performance benchmarking in staging

Key Test Areas:

• MCE end-to-end staging environment validation

• Performance and scalability testing

• Security vulnerability scanning

• Pre-production readiness checks

3.2 ACM 2.15.1 Staging Testing

Test Focus:

• ACM staging environment deployment validation

• ACM release candidate testing

• Integration testing with staging infrastructure

• Performance benchmarking in staging

• Security scanning and validation

• Release readiness verification

Key Test Areas:

• End-to-end staging environment validation

• Performance and scalability testing

• Security vulnerability scanning

• Documentation accuracy verification

• Pre-production readiness checks

---

4. Post-Publishing Testing

4.1 MCE 2.10.1 Post-Publishing Testing

Test Focus:

• Published MCE release artifact validation

• Production deployment verification

• Customer scenario validation

• Support handoff validation

Key Test Areas:

• Official MCE release artifact verification

• Production-like environment testing

• Real-world customer use cases

• Documentation and support material validation

4.2 ACM 2.15.1 Post-Publishing Testing

Test Focus:

• Published ACM release artifact validation

• Production deployment verification

• Customer scenario validation

• Support handoff validation

• Hotfix and patch readiness

Key Test Areas:

• Official release artifact verification

• Production-like environment testing

• Real-world customer use cases

• Documentation and support material validation

• Emergency response readiness

---

5. Advisory and Pre-Production Testing

5.1 Prepare and Review Advisory

Test Focus:

• Advisory content preparation and validation

• Security errata and CVE documentation

• Release notes and documentation review

• Customer-facing communication validation

Key Test Areas:

• Advisory content accuracy and completeness

• CVE mapping and security documentation

• Release notes technical review

• Communication materials validation

• Compliance and regulatory requirements

5.2 Pre-Production Pushing Testing

Test Focus:

• Pre-production environment deployment validation

• Release artifact pushing and validation

• Production readiness verification

• Rollback procedures testing

Key Test Areas:

• Pre-production deployment validation

• Artifact repository and registry testing

• Production promotion process validation

• Emergency rollback procedures

• Production environment readiness checks

---

Test Execution Workflow

1. Identify and organize bugs for retest - Retrieve all bugs from ACM 2.15.1 and MCE 2.10.1 releases

1. Create squad-specific retest tasks - Use automated tooling to create individual retest tasks for each squad

1. Deploy test environment - Deploy ACM 2.15.1 on vSphere hub (OCP 4.21)

1. Provision spoke clusters - Provision spoke clusters (Hive + IKS)

1. Execute MCE bug retests - Squads verify MCE 2.10.1 bug fixes

1. Execute ACM bug retests - Squads verify ACM 2.15.1 bug fixes

1. Execute MCE Regression tests - Run MCE 2.10.1 regression test suite

1. Execute ACM Regression tests - Run ACM 2.15.1 regression test pipelines (Non-UI and UI)

1. Execute ACM Upgrade tests - Run ACM 2.15.1 upgrade test pipeline (Y-stream and Z-stream)

1. Execute MCE Staging tests - Run MCE 2.10.1 staging test suite

1. Execute ACM Staging tests - Run ACM 2.15.1 staging test suite

1. Execute MCE Post-Publishing tests - Run MCE 2.10.1 post-publishing test suite

1. Execute ACM Post-Publishing tests - Run ACM 2.15.1 post-publishing test suite

1. Prepare and review advisory - Prepare advisory content and documentation

1. Execute Pre-Production pushing testing - Validate pre-production deployment and artifact promotion

1. Document and report results - Document all test results and file issues for failures

---

Jenkins Job Links

• Non-UI Pipeline

• UI Pipeline

• Upgrade Pipeline