Description of problem:

RHDHPLAN-615 introduced a new `spec.deployment.kind` field in the v1alpha5 API, so that users can specify the RHDH workload type: Deployment (default) or StatefulSet (new).
The problem is that if users update their CR in the cluster by switching the deployment kind from one type to another, they will be left with resources created for both types in the cluster.

For example, if users are updating an existing Backstage CR (that did not previously have this new spec.deployment.kind field set, or had it set to Deployment) by switching to StatefulSet, they would need to manually delete the previous Deployment resource that was created by the RHDH Operator.
This is because the Operator would now be managing a StatefulSet resource in the cluster. So the previous Deployment would remain orphan.
Worse, I noticed that the traffic may still be sent to the previous Deployment pods, which can create confusion.

Prerequisites (if any, like setup, operators/versions):

RHDH Operator 1.9+

Steps to Reproduce

1. Create a CR without this field set (default behavior would be Deployment)

apiVersion: rhdh.redhat.com/v1alpha5
kind: Backstage
metadata: 
  name: rhdh-bs1

2. Wait until successful reconciliation

3. Confirm the list of Deployments and StatefulSets

❯ kubectl get deploy                          
NAME                 READY   UP-TO-DATE   AVAILABLE   AGE
backstage-rhdh-bs1   1/1     1            1           2m41s

❯ kubectl get statefulset
NAME                      READY   AGE
backstage-psql-rhdh-bs1   1/1     41s

4. Now set spec.deployment.kind to StatefulSet

apiVersion: rhdh.redhat.com/v1alpha5
kind: Backstage
metadata: 
  name: rhdh-bs1
spec: 
  deployment: 
    kind: StatefulSet

5. Wait until successful reconciliation

6. Now check the list of Deployments and StatefulSets. We can also see pods for both the RHDH Deployment and StatefulSet

❯ kubectl get deploy                          
NAME                 READY   UP-TO-DATE   AVAILABLE   AGE
backstage-rhdh-bs1   1/1     1            1           4m41s

❯ kubectl get statefulset
NAME                      READY   AGE
backstage-psql-rhdh-bs1   1/1     4m14s
backstage-rhdh-bs1        1/1     2m34s

❯ kubectl get pods
NAME                                 READY   STATUS    RESTARTS   AGE
backstage-psql-rhdh-bs1-0            1/1     Running   0          7m24s
backstage-rhdh-bs1-0                 1/1     Running   0          4m3s
backstage-rhdh-bs1-689574bff-zhgpb   1/1     Running   0          7m24s

Actual results:

The previous Deployment resource is still present in the cluster, even if the workload type has been changed to StatefulSet.
This may be an issue, as the Route traffic may be sent to the old Deployment pods (because of the backing Service selectors). I noticed this after making a change to the application configuration.

To fix this, users would need to delete the previous resource from the cluster. In the example, I had to manually delete the previous Deployment with:

kubectl delete deployment -l 'app.kubernetes.io/instance=rhdh-bs1'

Expected results:

To avoid issues like this, the Operator could automatically delete the previous left-over resource. But this might introduce some application downtime (which users should probably expect after changing the workload type).

Reproducibility (Always/Intermittent/Only Once):

Always

Build Details:

Additional info (Such as Logs, Screenshots, etc):