After looking into the errors for the builds some test threw 504 exceptions when hitting the endpoint '/v1/internal/rpc/tally/snapshots/18939578'. After further investigation the following splunk query reveled the dead lock errors.  

The follow MR may be linked to the issue.
Here is the report portal report that shows the tests throwing this exception. 

Here is a timestamp of when one of the tests threw an error '2025-10-16 11:44:38'