Description of problem:

Argo CD Application controller is stuck Syncing applications

Prerequisites (if any, like setup, operators/versions):

ArgoCD Version 1.7.0

Steps to Reproduce

1. Create a cluster and applications using ACM
2. Remove the cluster and its secret from GitOps
3. The applications remains
4. Application controller is stuck Syncing applications

Actual results:

The cluster is removed but applications running on it don't

Expected results:

The applications get also removed

Reproducibility (Always/Intermittent/Only Once):

Always

Build Details:

Additional info (Such as Logs, Screenshots, etc):

E0223 04:04:00.653127       1 diff.go:697]  "msg"="Failed to convert from unstructured into Secret" "error"="error decoding from json: illegal base64 data at input byte 12" 

$ fgrep -c "error decoding from json" openshift-gitops-application-controller-0-argocd-application-controller.log 
2606

time="2023-02-23T04:04:00Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: git-events)" application=openshift-gitops/cluster1-events
time="2023-02-23T04:04:00Z" level=info msg="getRepoObjs stats" application=openshift-gitops/cluster-events build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=5 unmarshal_ms=4 version_ms=0
E0223 04:04:00.653127       1 diff.go:697]  "msg"="Failed to convert from unstructured into Secret" "error"="error decoding from json: illegal base64 data at input byte 12" 
E0223 04:04:00.690523       1 diff.go:697]  "msg"="Failed to convert from unstructured into Secret" "error"="error decoding from json: illegal base64 data at input byte 12" 
time="2023-02-23T04:04:00Z" level=info msg="Update successful" application=openshift-gitops/cluster1-events

time="2023-02-23T13:21:54Z" level=info msg="Alloc=2372508 TotalAlloc=16985355093 Sys=8671797 NumGC=10571 Goroutines=9713"
time="2023-02-23T13:31:54Z" level=info msg="Alloc=2385076 TotalAlloc=16985483219 Sys=8671797 NumGC=10576 Goroutines=9713"
time="2023-02-23T13:41:54Z" level=info msg="Alloc=2390510 TotalAlloc=16985603015 Sys=8671797 NumGC=10581 Goroutines=9712"
time="2023-02-23T13:51:54Z" level=info msg="Alloc=2382914 TotalAlloc=16985716008 Sys=8671797 NumGC=10586 Goroutines=9712"
time="2023-02-23T14:01:54Z" level=info msg="Alloc=2379606 TotalAlloc=16985826893 Sys=8671797 NumGC=10591 Goroutines=9712"
time="2023-02-23T14:11:54Z" level=info msg="Alloc=2377274 TotalAlloc=16985936872 Sys=8671797 NumGC=10596 Goroutines=9700"

time="2023-02-23T13:05:51Z" level=error msg="Failed to reload open api schema" error="error getting openapi resources: Get \"https://api.git.subdomain.domain.com:6443/openapi/v2?timeout=32s\": dial tcp 10.110
.197.138:6443: i/o timeout" server="https://api.git.subdomain.domain.com:6443"
time="2023-02-23T13:05:51Z" level=info msg="Updated sync status: OutOfSync -> Unknown" application=git-openshift-generic dest-namespace=openshift-config dest-server="https://api.git.subdomain.domain.com:644
3" reason=ResourceUpdated type=Normal
time="2023-02-23T13:05:51Z" level=info msg="Updated sync status: OutOfSync -> Unknown" application=git-oauth dest-namespace=openshift-authentication dest-server="https://api.git.subdomain.domain.com:6443" r
eason=ResourceUpdated type=Normal

time="2023-02-23T15:41:54Z" level=info msg="Alloc=2402617 TotalAlloc=16986861125 Sys=8671797 NumGC=10640 Goroutines=9620"