If you use multiprocessing, you must know that all messages from one process to the other have to be pickled by python. This costs time. So if you have a small amount of data, which needs much time to calculate, you're good with multiprocessing.
If the task has too much data, you'll lose speed because of communication overhead.
Are you using numpy and pandas? If it's the case, look for https://dask.org/
BTW: For a prallel download task (only io) a single process is enough. You should use asyncio for this task.
If the task has too much data, you'll lose speed because of communication overhead.
Are you using numpy and pandas? If it's the case, look for https://dask.org/
BTW: For a prallel download task (only io) a single process is enough. You should use asyncio for this task.
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
All humans together. We don't need politicians!