-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
1.12.4, 1.19.0
-
5
-
False
-
-
False
-
-
-
GitOps Crimson Sprint 29
Description of Problem
* Upgrading the OpenShift GitOps Operator specifically from v1.12.3 to v1.12.4 causes the `service.beta.openshift.io/serving-cert-secret-name` annotation to be stripped from the Argo CD server Service.
This prevents the Service-CA ClusterOperator from rotating or managing the service certificate. While the console may work initially, it will eventually return an HTTP 503 Service Unavailable error once the certificate expires or the secret is modified. Because this state is persisted in the Service object, the issue remains even if the operator is subsequently upgraded to later versions (e.g., v1.19).
Additional Info
* Root Cause: The Service-CA ClusterOperator tracks self-signed certs via the `serving-cert-secret-name` annotation. A regression in the v1.12.4 reconciliation logic removes this annotation if the instance was upgraded from v1.12.3.
* Diagnostic Steps: Check the Service for the missing annotation while the pod is healthy:
`$ oc get svc -n <argocd-namespace> <argocd-instance>-server -ojson | jq '.metadata.annotations'`
(Result will show `signed-by` annotations but will be missing the `serving-cert-secret-name` key).
* Workaround: Manually re-annotate the service:
`oc annotate -n openshift-gitops svc/openshift-gitops-server service.beta.openshift.io/serving-cert-secret-name=argocd-server-tls`
Problem Reproduction
* The problem is reproduced by following the specific upgrade path of v1.12.3 -> v1.12.4. It does not appear to affect fresh 1.12.4 installs or upgrades from other versions.
Reproducibility
* Always (given the specific upgrade path).
Prerequisites/Environment
* Environment: OpenShift Container Platform (any version).
* Operator: OpenShift GitOps / Argo CD Operator.
* Version Path: Must have undergone an upgrade from v1.12.3 to v1.12.4. The bug persists in the environment through current versions (v1.19).
Steps to Reproduce
* 1. Deploy OpenShift GitOps Operator v1.12.3.
* 2. Verify `svc/openshift-gitops-server` has the `service.beta.openshift.io/serving-cert-secret-name` annotation.
* 3. Upgrade the Operator to v1.12.4.
* 4. Check the Service annotations again; observe the annotation has been removed.
* 5. (Optional) Upgrade further to v1.19; observe the annotation is still missing.
Expected Results
* The Operator should preserve the Service-CA annotations during and after the upgrade to ensure certificate continuity and console availability.
Actual Results
* The annotation is removed, eventually leading to:
* 1. Loss of certificate management by the Service-CA ClusterOperator.
* 2. 503 Service Unavailable / "Application is not available" errors in the browser when accessing the Argo CD UI.
Problem Analysis
- <Completed by engineering team as part of the triage/refinement process>
Root Cause
- <What is the root cause of the problem? Or, why is it not a bug?>
Workaround (If Possible)
- <Are there any workarounds we can provide to the customers?>
Fix Approaches
- <If we decide to fix this bug, how will we do it?>
Acceptance Criteria
- ...
Definition of Done
- Code Complete:
- All code has been written, reviewed, and approved.
- Tested:
- Unit tests have been written and passed.
- Ensure code coverage is not reduced with the changes.
- Integration tests have been automated.
- System tests have been conducted, and all critical bugs have been fixed.
- Tested and merged on OpenShift either upstream or downstream on a local build.
- Documentation:
- User documentation or release notes have been written (if applicable).
- Build:
- Code has been successfully built and integrated into the main repository / project.
- Midstream changes (if applicable) are done, reviewed, approved and merged.
- Review:
- Code has been peer-reviewed and meets coding standards.
- All acceptance criteria defined in the user story have been met.
- Tested by reviewer on OpenShift.
- Deployment:
- The feature has been deployed on OpenShift cluster for testing.