Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
task queue
#1
Hi,
I’d like to create a queue of tasks and share those tasks between two computers. I thought of just splitting the tasks evenly between the two computers but the completion time for each task can vary significantly.
Can anyone recommend a possible solution I could learn about?
Oh, I’m not sure if it matters but each task is just a string of parameters passed to an external program.
Reply
#2
What about having a thread that repeatedly waits until one of the computers is ready to accept a task, then sends the first task of the queue to that computer?
Reply
#3
Or a message queue that you can write to and the machines can read from when they're ready to pick up work. I've used Rabbit MQ for this at work; other technologies are available.
Reply
#4
Hi,
I've gone with RabbitMQ - it's awesome and I had it setup within an hour using the following resources as a guide:
Installation
Implementation

Thanks for the advice guys!
Reply
#5
Also runs in a Docker container ;).
Reply
#6
I’m not familiar with Docker container… I’ll google that too
Reply
#7
Docker is a very, very useful piece of software.
Reply
#8
I guess you can use aioconnectors, which is a very easy to set up broker based on asyncio : https://pypi.org/project/aioconnectors/
You have the send_message_await_response method which sends a message and waits asynchronously for a response from your remote application : your remote application could respond when the processing is done, which could enable you to manage your local load balancing accordingly.
Reply
#9
(Jun-22-2020, 12:16 PM)freeman Wrote: I guess you can use aioconnectors, which is a very easy to set up broker based on asyncio : https://pypi.org/project/aioconnectors/
You have the send_message_await_response method which sends a message and waits asynchronously for a response from your remote application : your remote application could respond when the processing is done, which could enable you to manage your local load balancing accordingly.

short demo :
https://youtu.be/ZSoomtVcGxE
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  count certain task in task manager[solved] kucingkembar 2 1,083 Aug-29-2022, 05:57 PM
Last Post: kucingkembar
  Schedule a task and render/ use the result of the task in any given time klllmmm 2 2,034 May-04-2021, 10:17 AM
Last Post: klllmmm
  How to create a task/import a task(task scheduler) using python Tyrel 7 3,628 Feb-11-2021, 11:45 AM
Last Post: Tyrel
  Queue in Pygame constantin01 1 3,640 Jan-07-2020, 04:02 PM
Last Post: metulburr
  Queue maxsize mr_byte31 2 4,502 Sep-03-2019, 07:02 PM
Last Post: mr_byte31
  Queue.Queue() would not reduce capacity after get() yuan8421 9 10,965 Jan-02-2018, 09:38 PM
Last Post: Windspar
  Threading and Queue nexusfactor 5 4,221 Oct-16-2017, 04:14 PM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020