starting a process every 10 seconds

[Skaperen]

i want to start a command every 10 seconds and capture its stdout output as it is produced. each command could be done in as short a time as 9 seconds or as long a time as 19 seconds. the exact command is ['ping','-c','10',address]. i want to be sure the next command process is started exactly 10 seconds after the previous command process was started, regardless when any processes quit. this means there could be some time that no processes are running and there could be some that that two processes are running. i also want to leave open the possibility that it could run even longer than 19 seconds, so there could be many processes running in extreme cases. the loop starting these processes will run indefinitely, possibly for several months. i also want to not leave any processes in a <defunct> state for more than a few seconds, certainly not collecting a bunch of them.

i could go about doing this with a bunch of module os functions to do it just like i have done this before in the C language. but i would like to find a more pythonic way of achieving this. i have tried some function in module multiprocessing and in module subprocess, but these have always run into problems like a growing pile of defunct processes or the whole thing just being hung and stop running. i would run the ping command indefinitely except that it drifts in time regardless. the ping rate is effectively a few milliseconds long than 1 second. by starting an N second ping every N seconds i can keep the average rate held to exactly one second with only a small amount of jitter for more accurate long-scale analysis in real time.

[squenson]

If you are running on a Linux platform, I strongly recommend that you create a cron job, find some explanations here: https://en.wikipedia.org/wiki/Cron

Your cron job will run at the exact same frequency, independently of the time of execution of the task.

[Skaperen]

this needs to be run under the python script that starts it and reads the real time results.

[Larz60+]

use python sched, see: https://docs.python.org/3/library/sched.html

[snippsat]

Python job scheduling for humans have used it several times work fine and easy to use.
Here a demo with pinging and use of Thread so all work together.

from subprocess import check_output
import subprocess
from threading import Thread
import time
import schedule

def ping():
    out = check_output(['ping', '-n', '10', 'google.com'], timeout=30)
    print(out.decode('utf-8').strip())

def run_threaded(ping):
    job_thread = Thread(target=ping)
    job_thread.start()

schedule.every(10).seconds.do(run_threaded, ping)
schedule.every(5).seconds.do(run_threaded, ping)
schedule.every(4).seconds.do(run_threaded, ping)

while 1:
    schedule.run_pending()
    time.sleep(1)

[Skaperen]

notice that sched will call a function. will that asynchronously start a new thread to run the function in or fork a new process?

---

the big issue is not figuring out when to start the command process, but how to manage doing so while the script is reading the responses from all the running processes (which can share the write end of a single pipe, so there is only one pipe to read).

---

an ideal solution would be to be able to make everything into events that can be waited for in an event loop:
1. data becoming available for reading in the pipe
2. time to start the next ping commamd.
3. child process exit indicating need to call os.waitpid()
then a simple loop can wait for each event and act on it.

---

if i were not reading from the pipe and not needing to call waitpid() after each process exits, a simple loop could get the current time, calculate how long to wait until the next period of a 10 second cycle, sleep for that long by calling time.sleep(), and starting the process in the background. the problem is that i need to be reading the pipe() and waiting for a process to exit, at the same time, while also waiting for when to start a new process. does Python have an event handler that can handle these kinds of events with an event wait function that returns when the event happens with full information about the event so the caller can know exactly which event happened? note that there is no underlying system event handler in either Unix or Windows, so this will be non-trivial to implement.

[Larz60+]

There's a good (and free) book which does an excellent job on syncronization:
The Little Book of Semaphores
chapters about

• Synchronization
  
• Semaphores
  

then breaks synchronization into 5 categories:

• Basic syncronization patterns
  
• Classical syncronization problems
  
• Less Classical syncronization problems
  
• Not-so-Classical syncronization problems
  
• Not remotely Classical problems
  
• Examples in C
  
• Examples in Python
  
• Cleanup Python
  
• Cleanup POSIX
  

[Skaperen]

ok, i downloaded the book in PDF format, which is convenient for me. but i am still thinking about the events project. i originally was planning to do this in C and then in Pike. so now i want to do it in Python. i can see how this could be greatly advantaged by semaphores.

the big idea is that all the waiting is merged into one method, events.next() which waits until the next event, whatever that is. there will need to be a variety of event classes, methods to interface with each class, and a means to create and add on new event classes. one built-in event class will be event.timer which defines events that will happen at specific times (and dates), absolute (at 18:00:00 today if not past it, already) and relative (10 seconds from now, or at the next hour). input and output events will also be available. input events can be defined as data is available to read or the data has already been read (and is provided by an appropriate interface). signals will make up another class of events. handling events of child process state changes (stop, continue, exit) will also be included. more may be added either as built-in events or as add-on event classes (the instances events.register() themselves as an event handler).