Python Forum
When will the GIL be removed?
Thread Rating:
  • 1 Vote(s) - 1 Average
  • 1
  • 2
  • 3
  • 4
  • 5
When will the GIL be removed?
#11
Thanks for the Web-scraping-part-2 link. Smile

At the current state of development PyPy software transactional memory version is only available for Python 2.7 and only for Linux. http://pypy.org/download.html

However, the PyPy technical framework for me seems to be more path-breaking than the one of current CPython.

https://en.wikipedia.org/wiki/PyPy Wrote:The PyPy project has developed a tool chain that analyzes RPython code and translates it into C code, which is then compiled to produce a native interpreter. It also allows for pluggable garbage collectors as well as optionally enabling Stackless Python features. Finally, it includes a just-in-time (JIT) generator which builds a just-in-time compiler into the interpreter, given a few annotations in the interpreter source code. The generated JIT compiler is a tracing JIT.[5]

As a numerical Python programmer I will be especially interested in the JIT Compiler.

Is Standard PyPy compatible with the Anaconda distro? For both linux and windows?
Reply
#12
(Nov-29-2016, 05:57 PM)snippsat Wrote:
(Nov-27-2016, 11:27 AM)consuli Wrote: using the package concurrent.futures or similar, posted by one of the mods. Would be nice to be linked again.
Here is the link Web-scraping-part-2

Can I use concurrent.futures away from asynchronous computing for parallel computing, too? I am referencing to the construct concurrent.futures.ProcessPoolExecutor.
Reply
#13
(Dec-04-2016, 06:15 PM)consuli Wrote: I am referencing to the construct concurrent.futures.ProcessPoolExecutor.
ProcessPoolExecutor is Multiprocessing and ThreadPoolExecutor is Threading.

ProcessPoolExecutor(Multiprocessing) is parallel computing it work bye side-stepping the GIL.
So it will use all core on your computer.
What can be tricky if there need to be a shared state between processes.
I had need no need for share state in my demo,so then is pretty straight forward to use.
Reply
#14
(Nov-29-2016, 07:00 PM)consuli Wrote: Is Standard PyPy compatible with the Anaconda distro? For both linux and windows?
All that is written python can be run bye PyPy.
Problem can be libraries that is written with a mix of Python and C/C++...ect.
There is work done be converting some of the most popular like NumPyPy.

Do get an overview of PyPy David Beazley has a fun video looking into to this large prosject.
Reply
#15
Many thanks, snippsat!

I've been able to reproduce the concurrent.futures.ProcessPoolExecutor example from the python manual.

import concurrent.futures
import math

PRIMES = [
    112272535095293,
    112582705942171,
    112272535095293,
    115280095190773,
    115797848077099,
    1099726899285419]

def is_prime(n):
   if n % 2 == 0:
       return False

   sqrt_n = int(math.floor(math.sqrt(n)))
   for i in range(3, sqrt_n + 1, 2):
       if n % i == 0:
           return False
   return True

def main():
   with concurrent.futures.ProcessPoolExecutor(max_workers= 2) as executor:
       for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
           print('%d is prime: %s' % (number, prime))

if __name__ == '__main__':
   main()
However, concurrent futures computing on 2 cores as shown above is NOT significantly faster than serial processing.

for i in range(0, len(PRIMES)):
print (PRIMES[i], is_prime(PRIMES[i]))
Thus, the big head from calling a process (which is only necessary because of the GIL) makes parallel computing in Python unattractive until the GIL is removed.

Quod erat demonstrandum.
Reply
#16
Maybe there are other numerical python programmers in the same situation as I. Meanwhile I have figured out, that Cython offers (in comparison to pure C++) a relatively easy but performant opportunity, to program parallel without GIL and process big head.

For instance see
https://python.g-node.org/python-summer ... cython.pdf
http://www.perrygeo.com/parallelizing-n ... d-mpi.html
http://docs.cython.org/en/latest/src/use...elism.html
[url=http://www.perrygeo.com/parallelizing-numpy-array-loops-with-cython-and-mpi.html][/url]

Consuli
Reply
#17
I know this is an old-ish thread but thought this was worth posting:
https://github.com/google/grumpy/blob/master/README.md
https://opensource.googleblog.com/2017/0...ython.html
Reply
#18
Transcompiling to go is an interesting choice. But, it's Google, so I guess they wanted to use their language. I think it would have been MoreBetter™ to try to compile to llvm mir. But, whatever I guess, right? :)
Reply
#19
Yeah, the particular strategy they used is somewhat strange... but it seemed relevant here since they nix'd the GIL and were explicit that they wanted their Python to be more parallel.
Reply


Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020