Description of Problem

Customer has a two-staged Argo CD setup, meaning there is a central Argo CD instance managing multiple namespaces which in turn have other Argo CD instances running. In this setup we are seeing issues with RoleBindings.

Specifically the customer has a namespace-scoped ArgoCD in the namespace "cicd-mgmt" created through an "ArgoCD" kind with the name "argocd".

The customer then creates an additional namespace called "customer-gitops" with the label "argocd.argoproj.io/managed-by: cicd-mgmt". This creates multiple rolebindings in the customer-gitops namepsace for th ArgoCD in cicd-mgmt, e.g. with the name "argocd-argocd-application-controller", with the subject to the ServiceAccount for the application controller in the cicd-mgmt namespace.

If they now create in the "customer-gitops" namespace another namespace-scoped ArgoCD instance through the ArgoCD kind with the name "argocd" these rolebindings are overwritten, e.g. the rolebinding with the name "argocd-argocd-application-controller" gets the ServiceAccount in customer-gitops as the subject.

This has the effect, that the "cicd-mgmt" ArgoCD looses access to the namespace it is supposed to manage. The workaround is to name these two ArgoCDs differently, then another RoleBinding name is chosen.

HOWEVER, what we additionally observe in this situation, is that as soon as we create the ArgoCD in customer-gitops, we do get additional rolebindings (e.g. for redis) not only for the local ArgoCD (called argocd), but also for the remote ArgoCD called mgmt. Ending up with more rights than necessary for components of the mgmt ArgoCD in the customer-gitops namepsace.

This likely also indicates that the operator mixes up the two different setups of rolebindings completely and might require a general overhaul.

Additional Info

Likely this is because argocd uses the same rolebinding naming convention in two different code paths (one for manage rolebindings of remote argocds and one for setting up the local argocd). We also observe that these rolebindings are flipping, likely due to the reason that the different codepaths constantly re-concile.

We can observe that our approach works if we for example name the ArgoCD in cicd-mgmt something like mgmt. Then the rolebinding for this ArgoCD is called mgmt-argocd-application-controller in the customer-gitops namespace and all access stays the same.

Problem Reproduction

1. Create a new namespace called "cicd-mgmt"
2. Create an "ArgoCD" object in that namespace named "argocd"
3. Create a new namespace called "customer-gitops" with the label "argocd.argoproj.io/managed-by: cicd-mgmt"
4. In the "customer-gitops" namespace, create an "ArgoCD" object  named "argocd"
5. Observe the content of the "argocd-argocd-application-controller" RoleBinding

Reproducibility

• Always

Prerequisites/Environment

• OpenShift Container Platform 4.16
• GitOps 1.15.1

Expected Results

• "argocd-argocd-application-controller" RoleBinding always keeps the same "subject"

Actual Results

• "argocd-argocd-application-controller" RoleBinding subject is changing between the two Argo CD instances

Workaround (If Possible)

• Naming the Argo CD instances differently, but there is a follow-up issue, be above