We used jgroups to replicate session data. When we use JRockit mission control to profile the appplication on a heavy load (about 5K JGroups unicast message send and 5K JGroups unicast message receive in one second). The JRockit shows the TimeScheduler2._run method is hot method, and call ConcurrentSkipList.size() method long time.

After investigation, the root cause is the tasks.keySet().removeAll(keys); The keySet method returns a ConcurrentSkipList.KeySet instance and it extends from AbstractSet, the ConcurrentSkipList.KeySet did not override the removeAll method, then the removeAll will use the AbstractSet.removeAll.

In AbstraceSet.removeAll method, the opendJDK implementation will check the size() of the current set and the parameter. Then it will call the ConcurrentSkipListMap.size() method. The ConcurrentSkipListMap.size will traverse the whole Map to count the entrys in the map. So if the skipListMap is big, it will be a performance issue.

So I think in the TimeScheduler._run, it should not use removeAll, and just remove the key one by one.