Loading...

Type: Bug
Resolution: Done
Priority: Critical
Fix Version/s: 1.2
Affects Version/s: 1.2
Component/s: Operator
Labels:
- doc-rn-1-2-fi

Story Points:
2
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Release Note Text:

Hide
This makes sure to provide a seamless upgrade of the Red Hat Developer Hub (RHDH) Operator from 1.1, by making sure that any existing database resources are used by the upgraded RHDH instances.
As a known issue, please note that if you had an existing operator-backed instance of RHDH, upgrading the operator to 1.2 would lead to the creation of additional resources like ConfigMaps or Services, and leave some resources unused in the namespace. It is recommended to manually clean up the deprecated resources after the upgrade:
- `configmap/<CustomResourceName>-auth-app-config`
- `configmap/<CustomResource>-dynamic-plugins`
- `service/backstage-psql-<CustomResourceName>-hl`

Show
This makes sure to provide a seamless upgrade of the Red Hat Developer Hub (RHDH) Operator from 1.1, by making sure that any existing database resources are used by the upgraded RHDH instances. As a known issue, please note that if you had an existing operator-backed instance of RHDH, upgrading the operator to 1.2 would lead to the creation of additional resources like ConfigMaps or Services, and leave some resources unused in the namespace. It is recommended to manually clean up the deprecated resources after the upgrade: - `configmap/<CustomResourceName>-auth-app-config` - `configmap/<CustomResource>-dynamic-plugins` - `service/backstage-psql-<CustomResourceName>-hl`
Release Note Type:
Known Issue
Release Note Status:
Proposed
Intelligence Requested:
Market:

Sprint:
RHDH Core Team 3257

SFDC Cases Links:
SFDC Cases Open:
SFDC Cases Counter:

Description of problem:

I had an already running operator and RHDH instance deployed in 1.1, then wanted to upgrade the operator. The upgrade of the operator itself succeeded, but I noticed that the existing RHDH DB was still there, and an additional DB was created.

Prerequisites (if any, like setup, operators/versions):

ClusterBot OpenShift Cluster (tested with 4.15.0)

Steps to Reproduce

1- Install the RHDH Operator "next" catalog source (script from https://github.com/janus-idp/operator/blob/main/.rhdh/scripts/install-rhdh-catalog-source.sh):

$ ./install-rhdh-catalog-source.sh --next

2- From the OCP web console, install the previous version of the Operator (1.1) using a "Manual" update strategy:

3- Once the operator is installed, create a Developer Hub instance and wait for it to be completely up and running
4- Check the resources created by the operator. Output in the "Actual results" section below
5- Now from the OCP web console, upgrade the Operator to the latest version available (1.2):

6- Wait a few minutes until the Operator is upgraded and the existing RHDH instance is upgraded (i.e., new pods should be created).

Actual results:

A new PostgreSQL StatefulSet is created upon reconciliation by the 1.2 operator, and it seems the upgraded Backstage instance uses that new database host.

This is the output of all resources created by the Operator in 1.1:

$ oc get all
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
NAME                                              READY   STATUS              RESTARTS   AGE
pod/backstage-my-developer-hub-776449d545-wtvfs   0/1     Init:0/1            0          23s
pod/backstage-psql-my-developer-hub-0             0/1     ContainerCreating   0          23s

NAME                                         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/backstage-my-developer-hub           ClusterIP   172.30.3.227   <none>        80/TCP     23s
service/backstage-psql-my-developer-hub      ClusterIP   172.30.5.182   <none>        5432/TCP   24s
service/backstage-psql-my-developer-hub-hl   ClusterIP   172.30.37.54   <none>        5432/TCP   23s

NAME                                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/backstage-my-developer-hub   0/1     1            0           23s

NAME                                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/backstage-my-developer-hub-776449d545   1         1         0       23s

NAME                                               READY   AGE
statefulset.apps/backstage-psql-my-developer-hub   0/1     25s

NAME                                                  HOST/PORT                                                                                     PATH   SERVICES                     PORT           TERMINATION     WILDCARD
route.route.openshift.io/backstage-my-developer-hub   backstage-my-developer-hub-my-ns.apps.ci-ln-hcsdct2-76ef8.origin-ci-int-aws.dev.rhcloud.com   /      backstage-my-developer-hub   http-backend   edge/Redirect   None 

$ oc get pvc                                                                                                       
NAME                                                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE                            
backstage-my-developer-hub-776449d545-98lz9-dynamic-plugins-root   Bound    pvc-64d360fc-ebd6-4e6c-b78a-2c754beaa44f   2Gi        RWO            gp3-csi        28s                            
data-backstage-psql-my-developer-hub-0                             Bound    pvc-b868273c-bc39-4619-8485-971828c35e2e   1Gi        RWO            gp3-csi        28s

And this is the output of all resources once the Operator is upgraded to 1.2 and the existing RHDH instance is reconciled:

$ oc get all
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
NAME                                              READY   STATUS    RESTARTS   AGE
pod/backstage-db-my-developer-hub-0               1/1     Running   0          3m51s
pod/backstage-my-developer-hub-5794594784-p8q9z   1/1     Running   0          3m52s
pod/backstage-psql-my-developer-hub-0             1/1     Running   0          11m

NAME                                         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/backstage-db-my-developer-hub        ClusterIP   172.30.1.247   <none>        5432/TCP   3m51s
service/backstage-my-developer-hub           ClusterIP   172.30.3.227   <none>        80/TCP     11m
service/backstage-psql-my-developer-hub      ClusterIP   172.30.5.182   <none>        5432/TCP   11m
service/backstage-psql-my-developer-hub-hl   ClusterIP   172.30.37.54   <none>        5432/TCP   11m

NAME                                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/backstage-my-developer-hub   1/1     1            1           11m

NAME                                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/backstage-my-developer-hub-5794594784   1         1         1       3m52s
replicaset.apps/backstage-my-developer-hub-776449d545   0         0         0       11m

NAME                                               READY   AGE
statefulset.apps/backstage-db-my-developer-hub     1/1     3m51s
statefulset.apps/backstage-psql-my-developer-hub   1/1     11m

NAME                                                  HOST/PORT                                                                                     PATH   SERVICES                     PORT           TERMINATION     WILDCARD
route.route.openshift.io/backstage-my-developer-hub   backstage-my-developer-hub-my-ns.apps.ci-ln-hcsdct2-76ef8.origin-ci-int-aws.dev.rhcloud.com   /      backstage-my-developer-hub   http-backend   edge/Redirect   None


$ oc get pvc                                                                                                
NAME                                                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
backstage-my-developer-hub-5794594784-mgvbf-dynamic-plugins-root   Bound    pvc-b93886c5-fc98-405f-aa55-0f4ad7b7f8b0   2Gi        RWO            gp3-csi        20s
backstage-my-developer-hub-776449d545-98lz9-dynamic-plugins-root   Bound    pvc-64d360fc-ebd6-4e6c-b78a-2c754beaa44f   2Gi        RWO            gp3-csi        6m19s
data-backstage-db-my-developer-hub-0                               Bound    pvc-1ebc88d8-b499-4235-88a1-bd4984eae5b5   1Gi        RWO            gp3-csi        19s
data-backstage-psql-my-developer-hub-0                             Bound    pvc-b868273c-bc39-4619-8485-971828c35e2e   1Gi        RWO            gp3-csi        6m19s

The main application Deployment is correctly upgraded, but the concerning changes are:

We have a new DB StatefulSet (backstage-db-my-developer-hub in this example):

NAME                                               READY   AGE
statefulset.apps/backstage-db-my-developer-hub     1/1     3m51s
statefulset.apps/backstage-psql-my-developer-hub   1/1     11m

We have new PVC created for this StatefulSet (data-backstage-db-my-developer-hub-0 in this example):

NAME                                                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
backstage-my-developer-hub-5794594784-mgvbf-dynamic-plugins-root   Bound    pvc-b93886c5-fc98-405f-aa55-0f4ad7b7f8b0   2Gi        RWO            gp3-csi        20s
backstage-my-developer-hub-776449d545-98lz9-dynamic-plugins-root   Bound    pvc-64d360fc-ebd6-4e6c-b78a-2c754beaa44f   2Gi        RWO            gp3-csi        6m19s
data-backstage-db-my-developer-hub-0                               Bound    pvc-1ebc88d8-b499-4235-88a1-bd4984eae5b5   1Gi        RWO            gp3-csi        19s
data-backstage-psql-my-developer-hub-0                             Bound    pvc-b868273c-bc39-4619-8485-971828c35e2e   1Gi        RWO            gp3-csi        6m19s

As a consequence, the upgraded application would use the new StatefulSet, causing it to start over with a new empty DB.

Expected results:

The upgraded application should use the existing database, as it may already contain data from the customer.

Reproducibility (Always/Intermittent/Only Once):

Always, when upgrading an existing operator-backed RHDH instance from 1.1 to 1.2

Build Details:

Operator image (1.2): registry.redhat.io/rhdh/rhdh-rhel9-operator@sha256:d627521b3e9fe739b463c29886a030ba9701a5088c86bb144deeb24ddd7d5108
RHDH image (1.2): registry.redhat.io/rhdh/rhdh-hub-rhel9@sha256:ac3d1f2d298f1966254da8821876b50cce2e44229acbd7e37a9e722c4a8486e6