ACM 2.15.1 and MCE 2.10.1 zStream Regression Testing

Overview

Comprehensive zStream regression testing for ACM 2.15.1 and MCE 2.10.1 across Regression (Non-UI/UI), Upgrade (Y-stream and Z-stream), Staging, Post-Publishing test paths, and Advisory/Pre-Production validation.

Environment Configuration

Testing Scope

Automation Testing Scope Matrix

Legend:

• Fresh Install: All components require fresh installation testing
• Y-Upgrade: Minor version upgrade testing (required for components with bugs)
• Z-Upgrade: Patch version upgrade testing (required for components with bugs)
• QE Focal: Primary QE contact responsible for component testing
• Comments: Additional testing notes or special requirements

Testing Logic:

• ✅ Yes = Testing required
• ❌ No = Testing not required
• Components with bugs → Both Y-Upgrade and Z-Upgrade testing required
• Components without bugs → Only Fresh Install testing required

Bug Retest Requirements

Bugs Organized by Component

Issues Classified by Component

Total Issues: 31
Total Groups: 10

---

Application Lifecycle (1 issues)

Owner: rh-ee-yupchang, Slack Group: @acm-qe-workload-mgmt

---

Cluster Lifecycle (1 issues)

Owner: rhn-support-dhuynh, Slack Group: @acm-qe-workload-mgmt

---

Console (10 issues)

Owner: rhn-support-dhuynh, Slack Group: @acm-qe-workload-mgmt

---

Container Native Virtualization (1 issues)

Owner: Unassigned, Slack Group: Unassigned

---

GRC (1 issues)

Owner: drkho, Slack Group: @acm-qe-grc

---

HyperShift (1 issues)

Owner: rhn-support-dhuynh, Slack Group: @acm-qe-workload-mgmt

---

Infrastructure Operator (3 issues)

Owner: Unassigned, Slack Group: Unassigned

---

Observability (9 issues)

Owner: mzardab@redhat.com, Slack Group: @observatorium-support

---

Server Foundation (1 issues)

Owner: laisongls, Slack Group: @acm-qe-serverfoundation

---

SiteConfig Operator (3 issues)

Owner: Unassigned, Slack Group: Unassigned

---

CVE/Vulnerability Issues

Total CVE/Vulnerability Issues: 26

All CVE/Vulnerability issues are tracked separately and will be verified as part of the staging and post-publishing test paths. The majority of these are CVE-2025-58183 related fixes across multiple components.

Full CVE list available in Jira: Search CVE Issues

Test Paths

1. Regression Testing

Comprehensive regression testing to validate core functionality and bug fixes for both MCE and ACM components.

1.1 MCE 2.10.1 Regression Testing

Test Focus:

• MCE component functionality validation
• Multi-cluster engine core features
• Backend functionality testing
• CLI operations and automation

Key Test Areas:

• Cluster lifecycle operations
• Multi-cluster engine infrastructure
• Cluster provisioning and management
• Infrastructure components validation

---

1.2 ACM Regression Testing

Non-UI Testing

Jenkins Pipeline: e2e_non-ui_test_pipeline

Test Focus:

• API validation and integration
• Backend functionality testing
• CLI operations and automation
• Policy and governance non-UI workflows

Key Test Areas:

• Cluster lifecycle operations
• Policy compliance and governance
• Application lifecycle management
• Observability and monitoring

UI Testing

Jenkins Pipeline: e2e_ui_test_pipeline

Test Focus:

• Console functionality and user workflows
• UI component validation
• Browser compatibility testing
• User experience validation

Key Test Areas:

• Cluster management UI
• Policy creation and management
• Application deployment UI
• Observability dashboards

---

2. Upgrade Testing

Jenkins Pipeline: e2e_upgrade_pipeline

Test Focus:

• ACM version upgrade paths validation (Y-stream and Z-stream)
• Data migration and preservation
• Backward compatibility testing
• Configuration retention during upgrade

Important: MCE upgrade testing is NOT included in this test path.

Key Test Areas:

• ACM Y-stream upgrade testing (minor version upgrade)
• ACM Z-stream upgrade testing (patch version upgrade)
• Managed cluster upgrades and compatibility
• Policy migration and continuity
• Application continuity across upgrades
• Rollback and recovery procedures

---

3. Staging Testing

Test Focus:

• Pre-release validation in staging environment
• Integration testing with latest OCP versions
• Performance and scalability validation

Key Test Areas:

• End-to-end workflow validation
• Multi-cluster management at scale
• Integration with OpenShift ecosystem

---

4. Post-Publishing Testing

Test Focus:

• Verification after official release
• Installation and upgrade from published artifacts
• Documentation validation

Key Test Areas:

• Fresh install from published images
• Upgrade from previous versions
• Release notes verification

---

5. Advisory and Pre-Production Testing

Test Focus:

• Final validation before customer availability
• Security advisory verification
• Production readiness assessment

Key Test Areas:

• CVE fixes validation
• Security scanning results
• Production environment compatibility

Test Execution Workflow

1. Environment Setup

• Deploy ACM 2.15.1 on vSphere hub (OCP 4.21)
• Provision spoke clusters (Hive + IKS)
• Configure test infrastructure

2. Component Bug Retest

• Each component squad retests their assigned bugs
• Verify fixes are working as expected
• Document test results in Jira

3. Regression Testing (Path 1)

• Execute MCE 2.10.1 regression tests
• Execute ACM Non-UI test pipeline
• Execute ACM UI test pipeline

4. Upgrade Testing (Path 2)

• Execute Y-stream upgrade tests
• Execute Z-stream upgrade tests
• Validate data migration and continuity

5. Staging Testing (Path 3)

• Deploy to staging environment
• Execute end-to-end workflows
• Performance validation

6. Post-Publishing Testing (Path 4)

• Install from published artifacts
• Verify upgrade paths
• Documentation validation

7. Advisory Testing (Path 5)

• CVE fixes verification
• Security scanning validation
• Production readiness assessment

8. Test Completion

• Document all test results
• File issues for any failures
• Sign off on release quality

Acceptance Criteria

• [ ] All component bugs retested and verified
• [ ] All Non-UI tests pass on the test environment
• [ ] All UI tests pass with supported browsers
• [ ] Upgrade path validated from previous ACM version (both Y and Z stream)
• [ ] Multi-cloud spoke clusters tested (Hive + IKS)
• [ ] Staging environment testing completed
• [ ] Post-publishing tests completed successfully
• [ ] All CVE/Vulnerability fixes verified
• [ ] All critical bugs identified and documented
• [ ] Test results documented in Jira

Dependencies

• vSphere environment for hub cluster deployment
• Cloud provider access for spoke cluster provisioning
• Jenkins pipeline access and permissions
• ACM 2.15.1 and MCE 2.10.1 release artifacts
• Staging environment availability
• Published container images and operators

Jenkins Job Links

• Non-UI Pipeline
• UI Pipeline
• Upgrade Pipeline