https://bugs.launchpad.net/watcher/+bug/2086710

 
in the newton release a background job scheduler was added to the Decision Engine.

https://github.com/openstack/watcher/commit/06c6c4691b103bf0b3fd3304a1a45fb22aedad50

to facilitate this the apscheduler lib was introduced as a depency to watcher.
apscheduler has a lost of capability but does not officially support eventlet.

since its introduction to watcher it has mostly worked partly by accident.
over the year as oslo, apscheduler and eventlet have evolved and adapted to newer python
release watcher has continued to use apscheduler even though that is not technically supported.

with the move to python 3.12 it became apparent that the background jobs executed on the apscheduler
BackgroundScheduler instances were accellign shared global state from a non-monkeypatched native thread.

that results in greenthread sometimes calling into objects that are using un monkey patched code.

for example oslo.db uses time.sleep to yield executions.
when that oslo.db function is first imported from a non patched thread if its invoked after that in the main thread it will block.

this can by this exception "RuntimeError: do not call blocking functions from the mainloop" here https://paste.opendev.org/show/bGPgfURx1cZYOsgmtDyw/

this has been repdocuded in ci as part of moving the ci jobs to ubutnu 24.04 and python 3.12

https://review.opendev.org/c/openstack/watcher/+/932963/comments/f54005d7_b0f831bb

to address this issue we need to ensure that the background thread used to schedule background task is properly monkey patched.