When running a large delete of namespaces, containing pods that have been annotated to use whereabouts managed IP addresses some of the managed IPs are not released when the pods are deleted 
    

    1. Create an /21 Ip pool
    2. Create 341 namespaces with 6 pods per namespace to fill up the IP pool (2046 total)
    3. Delete all the namespaces by label selector similar to oc delete ns -l kube-burner-job=ipam-swizzle
In an effort to get a smaller reproducer, this issue can be seen when 50 namespaces of 6 pods per namespace are deleted. 
    

a large amount of addresses (>2000) remained in the IPpool when the pods they were allocated to have been deleted.
    

The ip pool shows 0 allocations
    

This test was run with kube-burner-ocp, with a QPS/Burst setting of 20/20
This test was left in the state overnight. At some point in time during the night the reconciler did its job to delete the ip pool.
I am not sure that it is intentional or not, but the IP pool is empty, but the "over lapping range ip reservations" are not.