-
Bug
-
Resolution: Done
-
Major
-
1.15.0, 1.16.0, 1.17.0, 1.18.0
-
GitOps Scarlet Sprint 17, GitOps Scarlet Sprint 18, GitOps Scarlet Sprint 19
Description of Problem
Customer is trying to deploy the `ibm-common-service operator` and `ibm-integration-platform-navigator` operators using ArgoCD application.
With this application, they are trying deploy the 2 namespaces, catalogsources, ImageDigestMirrorSet and operator subscriptions and custom resources.
Rollout of the resources largely work, except that ArgoCD fails to claim proper privileges to administrate the CRDs provided by the operators and we get a below errors :
one or more objects failed to apply, reason: error when patching "/dev/shm/660316825": commonservices.operator.ibm.com "common-service" is forbidden: User "system:serviceaccount:openshift-gitops:openshift-gitops-argocd-application-controller" cannot patch resource "commonservices" in API group "operator.ibm.com" in the namespace "openshift-operators",platformnavigators.integration.ibm.com is forbidden: User "system:serviceaccount:openshift-gitops:openshift-gitops-argocd-application-controller"
platformnavigators.integration.ibm.com is forbidden: User "system:serviceaccount:openshift-gitops:openshift-gitops-argocd-application-controller" cannot create resource "platformnavigators" in API group "integration.ibm.com" in the namespace "cp4i"
The respective namespaces has a `argocd.argoproj.io/managed-by: openshift-gitops` label added.
Still the respective `openshift-gitops-argocd-application-controller` role does not have permission for these respective resources in API group "operator.ibm.com" and "integration.ibm.com".
Additional Info
Customer has below hypothesis on this :
As per customer, the `argocd.argoproj.io/managed-by` label is only honored on changes to namespace meta data, but not when new CRDs are introduced, which is causing permission issue.
Customer is suspecting that ArgoCD does not recognize these CRDs and does not update the role, Once the namespace got updated with any change it recognize the CRDs and update the role to have a proper permissions on it.
Customer presume that the ArgoCD component responsible for updating the namespace roles, is listening for events related to namespace updates in order to honor the argocd.argoproj.io/managed-by labels.
They suspect that the same operator is either not listening for events related to new CRDs or that such events are improperly routed to this operator.
ArgoCD/helm prioritizes namespace creation before most other resources, so naturally we can't expect that the initial argocd.argoproj.io/managed-by label causes permissions to be included for a CRD which hasn't been installed on the cluster yet. Instead we need to rely on ArgoCD discovering the discrepancy and adding the missing permissions at a later point in time.
Workaround (If Possible)
Customer confirmed that, As a workaround when they manually update/change namespace metadata, like simply updating namespace with just an annotation `foo=bar` after the CRDs successfully have been installed on the cluster.
With this update to namespace, The respective role also gets updated with proper permission for the respective api resources and the api group.