Parallel processing and distributed computing with Python

[GarlicScience]

What are the most popular approaches for parallel processing and distributed computing using Python.

1. Does it worth to use Python for such tasks?
2. What are the most general and efficient frameworks/libraries/packages for them?

I'd like so much to hear a complex answer, which describes state of things in this subject.

[heiner55]

See also https://wiki.python.org/moin/ParallelProcessing

[wavic]

It depends on what are you doing. Data processing, heavy math calculations or something else. You could use all the cores of a PC, some asynchronous execution ( doing something else while the core waits for a data from a disk storage, RAM, network ) or even spread the calculations on many PCs. If you are more specific we could point to some approaches and libraries. There are lots of modules you can use for concurrent programming.

[GarlicScience]

Thanks for the answer!
Well, I mean more heavy math calculations and data processing, especially machine learning / deep learning computations and algorithms.

[wavic]

You could start with built-in concurrent.futures.ProcessPoolExecutor which is a wrapper around multiprocessing module. If heavy computations are involved I will recommend gmpy2 library. I've used it for a while. Perhaps Scikit-learn for the machine learning part. I am saying perhaps because I've never done machine learning before. I think installing this module you will get Numpy and SciPy as well. And I think this is enough to start.

---

Quick google search for this Deep Learning thing and seems Keras is on top of the list.

[snippsat]

Well, I mean more heavy math calculations and data processing, especially machine learning / deep learning computations and algorithms.

There are a lot going on in that field,as Python is maybe most used now in that field. 

Dask
Scales up: Runs resiliently on clusters with 1000s of cores

Dask.distributed
It extends both the concurrent.futures and dask APIs to moderate sized clusters.

TensorFlow
The flexible architecture allows you to deploy computation to one or more CPUs or GPUs in a desktop,
server, or mobile device with a single API.

Both of this library can take advantages of also GPU.