When will the GIL be removed?

[consuli]

Thanks for the Web-scraping-part-2 link.

At the current state of development PyPy software transactional memory version is only available for Python 2.7 and only for Linux. http://pypy.org/download.html

However, the PyPy technical framework for me seems to be more path-breaking than the one of current CPython.

The PyPy project has developed a tool chain that analyzes RPython code and translates it into C code, which is then compiled to produce a native interpreter. It also allows for pluggable garbage collectors as well as optionally enabling Stackless Python features. Finally, it includes a just-in-time (JIT) generator which builds a just-in-time compiler into the interpreter, given a few annotations in the interpreter source code. The generated JIT compiler is a tracing JIT.[5]

As a numerical Python programmer I will be especially interested in the JIT Compiler.

Is Standard PyPy compatible with the Anaconda distro? For both linux and windows?

[consuli]

using the package concurrent.futures or similar, posted by one of the mods. Would be nice to be linked again.

Here is the link Web-scraping-part-2

Can I use concurrent.futures away from asynchronous computing for parallel computing, too? I am referencing to the construct concurrent.futures.ProcessPoolExecutor.

[snippsat]

I am referencing to the construct concurrent.futures.ProcessPoolExecutor.

ProcessPoolExecutor is Multiprocessing and ThreadPoolExecutor is Threading.

ProcessPoolExecutor(Multiprocessing) is parallel computing it work bye side-stepping the GIL.
So it will use all core on your computer.
What can be tricky if there need to be a shared state between processes.
I had need no need for share state in my demo,so then is pretty straight forward to use.

[snippsat]

Is Standard PyPy compatible with the Anaconda distro? For both linux and windows?

All that is written python can be run bye PyPy.
Problem can be libraries that is written with a mix of Python and C/C++...ect.
There is work done be converting some of the most popular like NumPyPy.

Do get an overview of PyPy David Beazley has a fun video looking into to this large prosject.

[consuli]

Many thanks, snippsat!

I've been able to reproduce the concurrent.futures.ProcessPoolExecutor example from the python manual.

import concurrent.futures
import math

PRIMES = [
    112272535095293,
    112582705942171,
    112272535095293,
    115280095190773,
    115797848077099,
    1099726899285419]

def is_prime(n):
   if n % 2 == 0:
       return False

   sqrt_n = int(math.floor(math.sqrt(n)))
   for i in range(3, sqrt_n + 1, 2):
       if n % i == 0:
           return False
   return True

def main():
   with concurrent.futures.ProcessPoolExecutor(max_workers= 2) as executor:
       for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
           print('%d is prime: %s' % (number, prime))

if __name__ == '__main__':
   main()

However, concurrent futures computing on 2 cores as shown above is NOT significantly faster than serial processing.

for i in range(0, len(PRIMES)):
print (PRIMES[i], is_prime(PRIMES[i]))

Thus, the big head from calling a process (which is only necessary because of the GIL) makes parallel computing in Python unattractive until the GIL is removed.

Quod erat demonstrandum.

[consuli]

Maybe there are other numerical python programmers in the same situation as I. Meanwhile I have figured out, that Cython offers (in comparison to pure C++) a relatively easy but performant opportunity, to program parallel without GIL and process big head.

For instance see
https://python.g-node.org/python-summer ... cython.pdf
http://www.perrygeo.com/parallelizing-n ... d-mpi.html
http://docs.cython.org/en/latest/src/use...elism.html
[url=http://www.perrygeo.com/parallelizing-numpy-array-loops-with-cython-and-mpi.html][/url]

Consuli

[micseydel]

I know this is an old-ish thread but thought this was worth posting:
https://github.com/google/grumpy/blob/master/README.md
https://opensource.googleblog.com/2017/0...ython.html

[nilamo]

Transcompiling to go is an interesting choice. But, it's Google, so I guess they wanted to use their language. I think it would have been MoreBetter™ to try to compile to llvm mir. But, whatever I guess, right? :)

[micseydel]

Yeah, the particular strategy they used is somewhat strange... but it seemed relevant here since they nix'd the GIL and were explicit that they wanted their Python to be more parallel.