Oct-01-2019, 08:01 AM
So, a bit of necromancy, but it turns out that requests.Session() can build multiple thread pools with one for each domain you connect to, recycling pools if the number of domains exceeds the limit. Threads will fail to return results if you try to open more than the pool max size, but I was able to set the number of pools equal to the number of domains, spawn a thread for each domain, then have each domain thread spawn pool_max_size child threads. The final results were much faster than anything I've ever seen go through this network, so I was quite pleased. Unfortunately, I don't think I'll be allowed to share code as this was for work, but I hope this helps anyone who faces a similar issue in the future.