best parallelisation strategy on python - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: General Coding Help (https://python-forum.io/forum-8.html) +--- Thread: best parallelisation strategy on python (/thread-9615.html) |
best parallelisation strategy on python - simona - Apr-19-2018 I have a code fortran 90 that is parallelised with MPI. I would like to traslate it in python, but I am not sure on the parallelisation strategy and libraries. I work on clusters, with each node with 5GB memory and 12 processors or 24 processors (depending on the cluster I am using). Ideally I would like to split the computation on several nodes. Let me explain what this code does: It read ~100GB data, they are divided in hdf5 files of ~25GB each. The code should read the data, go through it and then select a fraction of the data, ~1GB and then some CPU intensive work on it, and repeat this process many times, say 1000 times, then write the results to a single final file. I was thinking that the CPU intensive part would be written as a shared object in C. Do you have suggestions about which library to use? RE: best parallelisation strategy on python - Larz60+ - Apr-19-2018 This video is from a couple of years ago, but still (in my opinion) the best for concurrency: https://www.youtube.com/watch?v=MCs5OvhV9S4 slides, programs, etc here: http://www.dabeaz.com/usenix2009/concurrent/index.html |