Post your whole code. Otherwise it's hard to explain what is going on.
Usually the pitfall is serialization. The processes don't have
shared memory (with 3.8 we get the support). The data is serialized
with pickle and send through a Pipe to the other process.
The higher the amount of processes, the more communication overhead you get.
If you do things like this:
Big calculations can consume also much CPU time and even if you use numpy,
then it can use more than one core in parallel without the GIL.
This works, because numpy is implemented in C and in C you can disable the GIL,
but then you're on your own.
If you work with io-related stuff, multiprocessing is not the right approach.
In this case threading or asyncio is better. Thread works fine with IO, but
fail with CPU-bound tasks.
If you want to do scientific parallel computations, look for Dask.
Usually the pitfall is serialization. The processes don't have
shared memory (with 3.8 we get the support). The data is serialized
with pickle and send through a Pipe to the other process.
The higher the amount of processes, the more communication overhead you get.
If you do things like this:
while True:
passI will consume the most cpu-time. In other words, you get 100% load on one core.Big calculations can consume also much CPU time and even if you use numpy,
then it can use more than one core in parallel without the GIL.
This works, because numpy is implemented in C and in C you can disable the GIL,
but then you're on your own.
If you work with io-related stuff, multiprocessing is not the right approach.
In this case threading or asyncio is better. Thread works fine with IO, but
fail with CPU-bound tasks.
If you want to do scientific parallel computations, look for Dask.
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
All humans together. We don't need politicians!