Python Forum
CPU load at approx. 30% when multiprocessing
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
CPU load at approx. 30% when multiprocessing
#1
Hi,
I am having trouble understanding the issue with my code. (This is just a part of my project with changed names to make it the code clearer)

def eval(val_list, value1):
   prod = partial(boolean_func, value1= value1)
   boolean_list = pool.map(func = prod, iterable = val_list, chunksize = 50)
val_list is a list of roughly 250 items und boolean_func checks if a item of val_list and value1 match to a specific "rule"
boolean_list later in the code is iterated over to work with based on the evaluation.

When I run my code the CPU load is rather evenly distributed between the cores to reach a maximal CPU load of approx. 30% on each core. How can I increase the CPU usage?
Because if I compare my code to the following loop:

boolean_list = []
for item in val_list:
   boolean_list.append(boolean_func(item, value1))
The loop runs faster than my code although it doesn't use multiprocessing.
What am I missing?

Thank you
Reply
#2
Hi,

first, multiprocessing is no guarantee that something runs faster. It is very likely to run faster on CPU bound problems, but not always.

The time per process per CPU is set by the underlaying OS. Thus, if you want your processes to have more CPU-time, you need to give them a higher priority from the OS side.

Second, multiprocessing creates some overhead setting up the processes, thus of the overall process run relatively short, multiprocessing may be slower due to that.

Third, the chunksize may be unfavorably chosen.

What are the actual run times of our code?

By the way: eval is a bad name for a function, as it overrides the build-in function eval?. Which may have very "funny" side effects. Better change the name of your function.

Regards, noisefloor
Reply
#3
Thank you for the quick response.
Is it possible to set the priority of my processes higher or is it an internal os mechanism which I can not control?
If I set the chunksize lower to like 1 or 10 then the run takes much longer compared to a higher chunksize.
I am asking myself if going through a list and matching its content to specific rules and assigning each matching process to a boolean value is a task that is worth multiprocessing. If my list gets longer the run also takes longer which doesn't seem to make sense. I thought the more you have to compute the better does multiprocessing perform.

Edit: run times: sequential: approx. 12 seconds
Parallel : approx. 14 seconds
Reply
#4
Hi,

Quote: Is it possible to set the priority of my processes higher or is it an internal os mechanism which I can not control?
You can, but I think not from inside Python - at least I'm not aware how. The way to go depends on the underlying OS. On Linux, you can set process priorities with nice.

In general, I think your problem is suitable for multiprocessing. You have a list of independent data and want to process each data set - which works fine with multiprocessing.

However, for a runtime of ~10 seconds only, it's questionable if you really get any big gains in runtime on multiprocessing. If you want to run faster, I would go here for PyPy or probably the JIT of Numba. Whereas pypy is the easier why: just run your code with pypy instead the the CPython interpreter.

Regards, noisefloor
Reply
#5
Post your whole code. Otherwise it's hard to explain what is going on.
Usually the pitfall is serialization. The processes don't have
shared memory (with 3.8 we get the support). The data is serialized
with pickle and send through a Pipe to the other process.

The higher the amount of processes, the more communication overhead you get.

If you do things like this:

while True:
    pass
I will consume the most cpu-time. In other words, you get 100% load on one core.
Big calculations can consume also much CPU time and even if you use numpy,
then it can use more than one core in parallel without the GIL.
This works, because numpy is implemented in C and in C you can disable the GIL,
but then you're on your own.


If you work with io-related stuff, multiprocessing is not the right approach.
In this case threading or asyncio is better. Thread works fine with IO, but
fail with CPU-bound tasks.

If you want to do scientific parallel computations, look for Dask.
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply


Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020