Python Forum

Full Version: how does GIL impact threads?
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
how does GIL impact the running of threads? i know i can have each running thread start a command in a separate process since i have tested that. i want to run N such processes for a system with N CPUs. when the command process ends, the thread will end. maybe i can have it put a small object on an instance of queue.SimpleQueue and have the main thread waiting for that. but i'm still not sure how GIL can impact this.
Global Interpreter Lock
You can share data between threads but not between processes. For each separate process, I think a new Python starts.
> I think a new Python starts.

or does it just replicate a copy of the already running Python? that's what would happen when the system level fork() is called. but it can reload Python and run it from the beginning. normally a fresh Python expects to start a script from the beginning. but is there a way to start a new Python in the middle of a script?

what i have experimented with, so far, is a thread calling subprocess.call(), which runs (and waits for) a process with a specific command. in cases where running some executable in a process is the goal, this seems to be a fine way to go. but cases where continuing to run the same script (maybe in an import module) is a different issue. that replicates everything and the child process would still have threads and the GIL would still be present. maybe it is not safe to fork a process from a multithread situation. doing so in C is not safe, but that is system level threads where there can literally be another thread running in another core when the fork() syscall happens. but, in Python, the GIL is keeping that from being the case.

i need to run the multiprocess case, i think, because i need multiple tasks running botocore, which apparently is not multithread ready (mulithready). i think i need to stay totally away from threads to do processes that keep a copy of Python running (e.g. never call any exec??() syscall).