Description of problem:

mismatch of virt-handler daemonset state and pods state after making master schedulable and unschedulable

0) Check the Schedulable labels.
oc get nodes -l "kubevirt.io/schedulable"
NAME STATUS ROLES AGE VERSION
c01-sb48a-mg575-worker-0-hst2w Ready worker 4d23h v1.21.8+ee73ea2
c01-sb48a-mg575-worker-0-qxfr2 Ready worker 4d23h v1.21.8+ee73ea2
c01-sb48a-mg575-worker-0-wxp85 Ready worker 4d23h v1.21.8+ee73ea2

1) Mark the master nodes as Schedulable == "true"

]$ oc patch schedulers.config.openshift.io cluster --type='merge' --patch='{"spec": {"mastersSchedulable": true}}'

2) Notice that the virt-handler pod are now running on the Master nodes as well.
$ oc get ds -n openshift-cnv
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
...
virt-handler 6 6 6 6 6 kubernetes.io/os=linux 4d22h

oc get nodes -l "kubevirt.io/schedulable=true"
NAME STATUS ROLES AGE VERSION
c01-sb48a-mg575-master-0 Ready master,worker 4d23h v1.21.8+ee73ea2
c01-sb48a-mg575-master-1 Ready master,worker 4d23h v1.21.8+ee73ea2
c01-sb48a-mg575-master-2 Ready master,worker 4d23h v1.21.8+ee73ea2
c01-sb48a-mg575-worker-0-hst2w Ready worker 4d23h v1.21.8+ee73ea2
c01-sb48a-mg575-worker-0-qxfr2 Ready worker 4d23h v1.21.8+ee73ea2
c01-sb48a-mg575-worker-0-wxp85 Ready worker 4d23h v1.21.8+ee73ea2

3) Mark the master nodes as Schedulable == "false"
]$ oc patch schedulers.config.openshift.io cluster --type='merge' --patch='{"spec": {"mastersSchedulable": false}}'

4) The Daemonset has got updated.
$ oc get ds -n openshift-cnv
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
...
virt-handler 3 3 3 3 3 kubernetes.io/os=linux 4d22h

5) Notice that the virt-handler pod are STILL running on the Master nodes as well.
Though the virt-handler daemonset count says Current count as "3".

$ oc get nodes -l "kubevirt.io/schedulable=true"
NAME STATUS ROLES AGE VERSION
c01-sb48a-mg575-master-0 Ready master 4d23h v1.21.8+ee73ea2
c01-sb48a-mg575-master-1 Ready master 4d23h v1.21.8+ee73ea2
c01-sb48a-mg575-master-2 Ready master 4d23h v1.21.8+ee73ea2
c01-sb48a-mg575-worker-0-hst2w Ready worker 4d23h v1.21.8+ee73ea2
c01-sb48a-mg575-worker-0-qxfr2 Ready worker 4d23h v1.21.8+ee73ea2
c01-sb48a-mg575-worker-0-wxp85 Ready worker 4d23h v1.21.8+ee73ea2

Version-Release number of selected component (if applicable):
CNV-4.8 , 4.10

How reproducible:

Steps to Reproduce:
1. Make Masters Schedulable and Unschedulable
2.
3.

Actual results:
virt-handler daemonset count and virt-handler pods actual count mismatch

Expected results:
1) virt-handler daemonset count and virt-handler pods count match
2) expectation is when I check for "kubevirt.io/schedulable=true"
it should only exist on nodes, VMs are currently able to be scheduled on.

Additional info: