+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html)
+--- Thread: How to schedule independent processes (/thread-15549.html)
How to schedule independent processes - vaeVictis - Jan-21-2019
Hi all.
I'm coding a little routine whose job I am going to explain in a very simplified fashion.
I have a list of "hours" and "tags", on a txt file.
At "some hour" I need to check a web page and see whether the "some tag" is present or not.
The search for the "some tag" starts at "some hour".
If the "some tag" is found, the search finishes. It also finishes after a timeout I could impose.
(Obviously, when I find it I must do some stuff with that "some tag")
I am using selenium for retrieving information and have not problem with this aspect.
The problem (to me) is: there might be more than one search which start at the same hour.
I could not use sched module "priority" because the several search must start in parallel (respecting obviously the treshold of cores number).
I can not risk that a search lasts long and the second one in priority starts when the first finishes.
I could not figure out how to schedule several different processes I read from a file.
Any help would be appreciated.
RE: How to schedule independent processes - Larz60+ - Jan-21-2019
There are many scheduling packages available (also cron if on linux), see: https://pypi.org/search/?q=scheduler
RE: How to schedule independent processes - vaeVictis - Jan-22-2019
Thanks for your quick answer :)
The problem is not with the scheduling module but with the process module.
Or maybe I misunderstood and you are suggesting to use a scheduling mudule which directly implement a way to schedule really independent jobs? Is this your point, Larz60+?
RE: How to schedule independent processes - Larz60+ - Jan-22-2019
Quote:Or maybe I misunderstood and you are suggesting to use a scheduling mudule which directly implement a way to schedule really independent jobs?Yes
RE: How to schedule independent processes - vaeVictis - Jan-22-2019
Ok, thanks.
I'll take into account your suggestion. I did not know all that usefull stuff.
By the way, I'm considering if it would be better to rethink the problem, make a single list with all the "some tag", iterate over it, and eventually pop out the found element.
I'll keep you updated.
RE: How to schedule independent processes - Larz60+ - Jan-22-2019
Quote:rethink the problemAlways a good idea, even after it's working well!