-
Bug
-
Resolution: Done
-
Critical
-
1.2
-
2
-
False
-
-
False
-
-
Known Issue
-
Proposed
-
-
-
RHDH Core Team 3257
Description of problem:
I had an already running operator and RHDH instance deployed in 1.1, then wanted to upgrade the operator. The upgrade of the operator itself succeeded, but I noticed that the existing RHDH DB was still there, and an additional DB was created.
Prerequisites (if any, like setup, operators/versions):
- ClusterBot OpenShift Cluster (tested with 4.15.0)
Steps to Reproduce
- 1- Install the RHDH Operator "next" catalog source (script from https://github.com/janus-idp/operator/blob/main/.rhdh/scripts/install-rhdh-catalog-source.sh):
$ ./install-rhdh-catalog-source.sh --next
- 2- From the OCP web console, install the previous version of the Operator (1.1) using a "Manual" update strategy:
- 3- Once the operator is installed, create a Developer Hub instance and wait for it to be completely up and running
- 4- Check the resources created by the operator. Output in the "Actual results" section below
- 5- Now from the OCP web console, upgrade the Operator to the latest version available (1.2):
- 6- Wait a few minutes until the Operator is upgraded and the existing RHDH instance is upgraded (i.e., new pods should be created).
Actual results:
A new PostgreSQL StatefulSet is created upon reconciliation by the 1.2 operator, and it seems the upgraded Backstage instance uses that new database host.
- This is the output of all resources created by the Operator in 1.1:
$ oc get all
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
NAME READY STATUS RESTARTS AGE
pod/backstage-my-developer-hub-776449d545-wtvfs 0/1 Init:0/1 0 23s
pod/backstage-psql-my-developer-hub-0 0/1 ContainerCreating 0 23s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/backstage-my-developer-hub ClusterIP 172.30.3.227 <none> 80/TCP 23s
service/backstage-psql-my-developer-hub ClusterIP 172.30.5.182 <none> 5432/TCP 24s
service/backstage-psql-my-developer-hub-hl ClusterIP 172.30.37.54 <none> 5432/TCP 23s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/backstage-my-developer-hub 0/1 1 0 23s
NAME DESIRED CURRENT READY AGE
replicaset.apps/backstage-my-developer-hub-776449d545 1 1 0 23s
NAME READY AGE
statefulset.apps/backstage-psql-my-developer-hub 0/1 25s
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
route.route.openshift.io/backstage-my-developer-hub backstage-my-developer-hub-my-ns.apps.ci-ln-hcsdct2-76ef8.origin-ci-int-aws.dev.rhcloud.com / backstage-my-developer-hub http-backend edge/Redirect None
$ oc get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
backstage-my-developer-hub-776449d545-98lz9-dynamic-plugins-root Bound pvc-64d360fc-ebd6-4e6c-b78a-2c754beaa44f 2Gi RWO gp3-csi 28s
data-backstage-psql-my-developer-hub-0 Bound pvc-b868273c-bc39-4619-8485-971828c35e2e 1Gi RWO gp3-csi 28s
- And this is the output of all resources once the Operator is upgraded to 1.2 and the existing RHDH instance is reconciled:
$ oc get all
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
NAME READY STATUS RESTARTS AGE
pod/backstage-db-my-developer-hub-0 1/1 Running 0 3m51s
pod/backstage-my-developer-hub-5794594784-p8q9z 1/1 Running 0 3m52s
pod/backstage-psql-my-developer-hub-0 1/1 Running 0 11m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/backstage-db-my-developer-hub ClusterIP 172.30.1.247 <none> 5432/TCP 3m51s
service/backstage-my-developer-hub ClusterIP 172.30.3.227 <none> 80/TCP 11m
service/backstage-psql-my-developer-hub ClusterIP 172.30.5.182 <none> 5432/TCP 11m
service/backstage-psql-my-developer-hub-hl ClusterIP 172.30.37.54 <none> 5432/TCP 11m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/backstage-my-developer-hub 1/1 1 1 11m
NAME DESIRED CURRENT READY AGE
replicaset.apps/backstage-my-developer-hub-5794594784 1 1 1 3m52s
replicaset.apps/backstage-my-developer-hub-776449d545 0 0 0 11m
NAME READY AGE
statefulset.apps/backstage-db-my-developer-hub 1/1 3m51s
statefulset.apps/backstage-psql-my-developer-hub 1/1 11m
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
route.route.openshift.io/backstage-my-developer-hub backstage-my-developer-hub-my-ns.apps.ci-ln-hcsdct2-76ef8.origin-ci-int-aws.dev.rhcloud.com / backstage-my-developer-hub http-backend edge/Redirect None
$ oc get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
backstage-my-developer-hub-5794594784-mgvbf-dynamic-plugins-root Bound pvc-b93886c5-fc98-405f-aa55-0f4ad7b7f8b0 2Gi RWO gp3-csi 20s
backstage-my-developer-hub-776449d545-98lz9-dynamic-plugins-root Bound pvc-64d360fc-ebd6-4e6c-b78a-2c754beaa44f 2Gi RWO gp3-csi 6m19s
data-backstage-db-my-developer-hub-0 Bound pvc-1ebc88d8-b499-4235-88a1-bd4984eae5b5 1Gi RWO gp3-csi 19s
data-backstage-psql-my-developer-hub-0 Bound pvc-b868273c-bc39-4619-8485-971828c35e2e 1Gi RWO gp3-csi 6m19s
The main application Deployment is correctly upgraded, but the concerning changes are:
- We have a new DB StatefulSet (backstage-db-my-developer-hub in this example):
NAME READY AGE statefulset.apps/backstage-db-my-developer-hub 1/1 3m51s statefulset.apps/backstage-psql-my-developer-hub 1/1 11m
- We have new PVC created for this StatefulSet (data-backstage-db-my-developer-hub-0 in this example):
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE backstage-my-developer-hub-5794594784-mgvbf-dynamic-plugins-root Bound pvc-b93886c5-fc98-405f-aa55-0f4ad7b7f8b0 2Gi RWO gp3-csi 20s backstage-my-developer-hub-776449d545-98lz9-dynamic-plugins-root Bound pvc-64d360fc-ebd6-4e6c-b78a-2c754beaa44f 2Gi RWO gp3-csi 6m19s data-backstage-db-my-developer-hub-0 Bound pvc-1ebc88d8-b499-4235-88a1-bd4984eae5b5 1Gi RWO gp3-csi 19s data-backstage-psql-my-developer-hub-0 Bound pvc-b868273c-bc39-4619-8485-971828c35e2e 1Gi RWO gp3-csi 6m19s
As a consequence, the upgraded application would use the new StatefulSet, causing it to start over with a new empty DB.
Expected results:
The upgraded application should use the existing database, as it may already contain data from the customer.
Reproducibility (Always/Intermittent/Only Once):
Always, when upgrading an existing operator-backed RHDH instance from 1.1 to 1.2
Build Details:
- Operator image (1.2): registry.redhat.io/rhdh/rhdh-rhel9-operator@sha256:d627521b3e9fe739b463c29886a030ba9701a5088c86bb144deeb24ddd7d5108
- RHDH image (1.2): registry.redhat.io/rhdh/rhdh-hub-rhel9@sha256:ac3d1f2d298f1966254da8821876b50cce2e44229acbd7e37a9e722c4a8486e6
Additional info (Such as Logs, Screenshots, etc):
-
- links to
