Hi,
I’d like to create a queue of tasks and share those tasks between two computers. I thought of just splitting the tasks evenly between the two computers but the completion time for each task can vary significantly.
Can anyone recommend a possible solution I could learn about?
Oh, I’m not sure if it matters but each task is just a string of parameters passed to an external program.
What about having a thread that repeatedly waits until one of the computers is ready to accept a task, then sends the first task of the queue to that computer?
Or a message queue that you can write to and the machines can read from when they're ready to pick up work. I've used Rabbit MQ for this at work; other technologies are available.
Hi,
I've gone with RabbitMQ - it's awesome and I had it setup within an hour using the following resources as a guide:
Installation
Implementation
Thanks for the advice guys!
Also runs in a Docker container ;).
I’m not familiar with Docker container… I’ll google that too
Docker is a very, very useful piece of software.
I guess you can use aioconnectors, which is a very easy to set up broker based on asyncio :
https://pypi.org/project/aioconnectors/
You have the send_message_await_response method which sends a message and waits asynchronously for a response from your remote application : your remote application could respond when the processing is done, which could enable you to manage your local load balancing accordingly.
(Jun-22-2020, 12:16 PM)freeman Wrote: [ -> ]I guess you can use aioconnectors, which is a very easy to set up broker based on asyncio : https://pypi.org/project/aioconnectors/
You have the send_message_await_response method which sends a message and waits asynchronously for a response from your remote application : your remote application could respond when the processing is done, which could enable you to manage your local load balancing accordingly.
short demo :
https://youtu.be/ZSoomtVcGxE