Description of problem:

On a hub cluster, openshift-gitops is setup based on the push model (where ACM creates the Application on the ManagedCluster using ManifestWork).

An ApplicationSet (named: app-busybox-cephfs-1) is initially placed on ManagedCluster rackm14, later the PlacementDecision (app-busybox-cephfs-1-placement-decision-1) is updated to deploy the Application to rackm03.

What is noted is as follows:

• The Application resource for rackm03 exists (app-busybox-cephfs-1-rackm03)
• There is no Application resource for rackm14
• The application ManifestWork for both ManagedClusters exist
    - app-busybox-cephfs-1-rackm03-fe0ac (namespace: rackm03)
    - app-busybox-cephfs-1-rackm14-48b8c (namespace: rackm14)

The ManifestWork app-busybox-cephfs-1-rackm14-48b8c was expected to be cleaned up as a result of the Application resource not being present for the same ManagedCluster.

Version-Release number of selected component (if applicable):

ACM 2.13 on OCP 4.18

How reproducible:

Happened once, but similar instances have been seen in the past (quite one off)

Steps to Reproduce:

1. Create an ApplicationSet with gitops setup in the push model
2. Set it up such that it is initially placed only on one ManagedCluster
3. Switch the placement decision for the related Placement for the AppSet to another cluster
4. Ensure original cluster does not have the required Application (or ensure on the hub that the ManifestWork for the Application on the older cluster namespace is deleted)
5. Repeat a few time, with some wait in between for the Application to be rolled out fully, and it is possible that the issue may appear

Actual results:

Application (and its ManifestWork) is not removed from a cluster where it was orginally placed (by a PlacementDecision)

Expected results:

Application (and its ManifestWork) should be garbage collected if its PlacementDecision changes

Additional info:

As this was observed in the ODF-DR testing environment, one point of note is that the Placement for the ApplicationSet is reconciled by ODF-DR controllers and not ACM controllers.

Additional notes:

• Hub must-gather can be found here: http://rhsqe-repo.lab.eng.blr.redhat.com/ocs4qe/amrita/bz/racks/DFBUGS-2057/
• A similar operation was performed for ApplicationSet (app-busybox-rbd-3), where the ManifestWork for the related Application on rackm14 was deleted, once the Application instance for the same was deleted
• It can be seen that the controller (argocd-pull-integration-controller-manager container:argocd-pull-integration-controller-manager) is detecting the Application resource as missing, but the ManifestWork for the same still remains
  • Log: http://rhsqe-repo.lab.eng.blr.redhat.com/ocs4qe/amrita/bz/racks/DFBUGS-2057/hub/03-04-2025_20-45-33-api-bm-m14-mydomain-com/acm-mg/inspect.local.7524865032251751690/namespaces/open-cluster-management/pods/multicluster-integrations-9ffd44fc-mhjqs/argocd-pull-integration-controller-manager/argocd-pull-integration-controller-manager/logs/current.log
  • 2025-04-03T08:44:39.600557960Z 2025-04-03T08:44:39Z    ERROR    unable to fetch Application    {"controller": "application", "controllerGroup": "argoproj.io", "controllerKind": "Application", "Application": {"name":"app-busybox-cephfs-1-rackm14","namespace":"openshift-gitops"}

    , "namespace": "openshift-gitops", "name": "app-busybox-cephfs-1-rackm14", "reconcileID": "abf4850c-3ad3-49bd-9663-8889b6e35803", "error": "applications.argoproj.io \"app-busybox-cephfs-1-rackm14\" not found"}