Cluster Autoscaler not scaling down nodes which seem to qualify for scale-down [NEEDINFO]

Description of problem:

The `clusterautoscaler` in his cluster to scale up and down the nodes automatically as per the requirement and load.
However, it has been noticed that when removing test load Pods, nodes do not scale down.

Version-Release number of selected component (if applicable):
OpenShift Version: 4.10.20

How reproducible:
Easily reproducible: https://docs.openshift.com/container-platform/4.10/rest_api/autoscale_apis/clusterautoscaler-autoscaling-openshift-io-v1.html#specification

Expected results:

When there is no load, I waited a very long time for the ClusterAutoscaler to scale down, but this never occurs. The ClusterAutoscaler controller Pod's logs keep saying no nodes are eligible to scale down despite the nodes very quite idle.

Additional info:

This Bugzilla is looking similar:
https://bugzilla.redhat.com/show_bug.cgi?id=2053343

Interestingly, I could not find the OLM and redhat-operator pod scheduled on any node other than the master.
Kindly have a look at the attached file for pod details.