Description of Problem

Because of missing permission in application controller cluster role, we see this error after enabling impersonation and deploying the application

serviceaccounts "guestbook-deployer" is forbidden: User "system:serviceaccount:openshift-gitops:openshift-gitops-argocd-application-controller" cannot impersonate resource "serviceaccounts" in API group "" in the namespace "guestbook"  

Additional Info

• IBM team was not able to reproduce the issue on P and Z archs initially. Even with missing  impersonate permission in application controller cluster role, this error was not seen during sync. However, when we tried this on a fresh P cluster, we were able to see the issue

Problem Reproduction

Enable the Application sync impersonation feature in argocd-cm

oc patch argocd/openshift-gitops -n openshift-gitops --type=merge -p='{"spec": {"extraConfig" : {"application.sync.impersonation.enabled":"true"}}}'

Create an AppProject called guestbook-proj

argocd proj create guestbook-proj -d https://kubernetes.default.svc,guestbook -s https://github.com/argoproj/argocd-example
-apps.git

Whitelist all cluster resources in the AppProject, so that the Application can auto create namespaces if required.

argocd proj allow-cluster-resource guestbook-proj '*' '*'

Add destination service account configuration for guestbook ns as below

argocd proj add-destination-service-account guestbook-proj https://kubernetes.default.svc guestbook guestbook-deployer

Create an argo application guestbook associated with AppProject guestbook-proj

argocd app create guestbook  \
    --repo https://github.com/argoproj/argocd-example-apps \
    --path guestbook \
    --project guestbook-proj \
    --dest-server  https://kubernetes.default.svc \
    --dest-namespace guestbook \
    --directory-recurse \
    --sync-policy automated \
    --sync-option ServerSideApply=true \
    --sync-option CreateNamespace=true

Check if the application fails to sync as the service account is not created yet.

oc get application guestbook -n openshift-gitops -o yaml

Reproducibility

• Intermittent (It had succeeded initially on x86, IBM team was unable to see this issue)

Prerequisites/Environment

• OCP x86

Steps to Reproduce

• See Problem Reproduction

Expected Results

message: 'Namespace auto creation failed: namespaces "guestbook" is forbidden:
          User "system:serviceaccount:guestbook:guestbook-deployer" cannot get resource
          "namespaces" in API group "" in the namespace "guestbook"' 

Actual Results

serviceaccounts "guestbook-deployer" is forbidden: User "system:serviceaccount:openshift-gitops:openshift-gitops-argocd-application-controller" cannot impersonate resource "serviceaccounts" in API group "" in the namespace "guestbook"  

Problem Analysis

• <Completed by engineering team as part of the triage/refinement process>

Root Cause

• <What is the root cause of the problem? Or, why is it not a bug?>

Workaround (If Possible)

Step 1: Create a new ClusterRole with impersonate permission

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: impersonation
rules:
  - verbs:
      - impersonate
    apiGroups:
      - ''
    resources:
      - serviceaccounts

Step 2: Add the new ClusterRole to Argo CD Application Controller service account using a new ClusterRoleBinding

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: impersonation-crb
subjects:
  - kind: ServiceAccount
    name: openshift-gitops-argocd-application-controller
    namespace: openshift-gitops
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: impersonation 

Fix Approaches

• <If we decide to fix this bug, how will we do it?>

Acceptance Criteria

• ...

Definition of Done

• Code Complete:
  • All code has been written, reviewed, and approved.
• Tested:
  • Unit tests have been written and passed.
  • Ensure code coverage is not reduced with the changes.
  • Integration tests have been automated.
  • System tests have been conducted, and all critical bugs have been fixed.
  • Tested and merged on OpenShift either upstream or downstream on a local build.
• Documentation:
  • User documentation or release notes have been written (if applicable).
• Build:
  • Code has been successfully built and integrated into the main repository / project.
  • Midstream changes (if applicable) are done, reviewed, approved and merged.
• Review:
  • Code has been peer-reviewed and meets coding standards.
  • All acceptance criteria defined in the user story have been met.
  • Tested by reviewer on OpenShift.
• Deployment:
  • The feature has been deployed on OpenShift cluster for testing.