Python Forum

This is from Dave Beazly's Generators: The Final Frontier here. Part 3, the example inline_recursive.py

When I run inline_recursive.py in Idle, by copying and pasting the parts into Idle, it works and prints lots of:

print('Tick:', n)

But I get this error when I run inline_recursive.py in bash, right after the first tick is printed:

pedro@pedro-HP:~/myPython/yield/tutorial2014$ ./inline_recursive.py
Tick: 0

Error:
exception calling callback for <Future at 0x75e01f9e0be0 state=finished returned NoneType>
Traceback (most recent call last):
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 342, in _invoke_callbacks
    callback(self)
  File "/home/pedro/myPython/yield/tutorial2014/./inline_recursive.py", line 26, in _wakeup
    self.step(None, exc)
  File "/home/pedro/myPython/yield/tutorial2014/./inline_recursive.py", line 14, in step
    fut = self._gen.throw(exc)
  File "/home/pedro/myPython/yield/tutorial2014/./inline_recursive.py", line 24, in _wakeup
    self.step(result, None)
  File "/home/pedro/myPython/yield/tutorial2014/./inline_recursive.py", line 16, in step
    fut = self._gen.send(value)
  File "/home/pedro/myPython/yield/tutorial2014/./inline_recursive.py", line 44, in recursive
    Task(recursive(n+1)).step()
  File "/home/pedro/myPython/yield/tutorial2014/./inline_recursive.py", line 16, in step
    fut = self._gen.send(value)
  File "/home/pedro/myPython/yield/tutorial2014/./inline_recursive.py", line 42, in recursive
    yield pool.submit(time.sleep, 0.001)
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 169, in submit
    raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown

Like I said, it works in the Idle shell.

#! /usr/bin/python3

# inline_recursive.py
#
# Bizarre inline recursive example

class Task:
    def __init__(self, gen):
        self._gen = gen

    def step(self, value=None, exc=None):
        try:
            if exc:
                fut = self._gen.throw(exc)
            else:
                fut = self._gen.send(value)
            fut.add_done_callback(self._wakeup)
        except StopIteration as exc:
            pass

    def _wakeup(self, fut):
        try:
            result = fut.result()
            self.step(result, None)
        except Exception as exc:
            self.step(None, exc)

# Example
if __name__ == '__main__':
    from concurrent.futures import ThreadPoolExecutor
    import time

    pool = ThreadPoolExecutor(max_workers=8)

    """ Error
    File "/usr/lib/python3.10/concurrent/futures/thread.py", line 169, in submit
        raise RuntimeError('cannot schedule new futures after '
    RuntimeError: cannot schedule new futures after interpreter shutdown
    """
    def recursive(n):
        # this submit causes a problem
        yield pool.submit(time.sleep, 0.001)
        print('Tick:', n)
        Task(recursive(n+1)).step()

    Task(recursive(0)).step()

I found, on the other hand, examples that work with .ProcessPoolExecutor(), like below, will not work in Idle, but work in bash!

#! /usr/bin/python3
import concurrent.futures

def worker(task):
    result = task * 2
    print(f"Task {task}: Result = {result}\n")
    return result

if __name__ == "__main__":
    tasks = [1, 2, 3, 4, 5]
    # will not work in the shell

    with concurrent.futures.ProcessPoolExecutor() as executor:
        results = executor.map(worker, tasks)
        print("Results:", list(results))

Any tips please?

you need to run the python interpreter: python3 ./inline_recursive.py

Sorry my comments were wrong, I mixed up inline_recursive.py and concurrent_futures_example3.py

inline_recursive.py works in Idle, concurrent_futures_example3.py, the second example above, works in bash but freezes Idle.

I have this, "shebang" I believe it is called, at the top of inline_recursive.py and inline_recursive.py runs in bash but throws the error:

Quote:#! /usr/bin/python3

But concurrent_futures_example3.py does not run in Idle

list(results) freezes Idle and I can only restart the shell can get Idle going again!

results is:

Quote:# results <generator object _chain_from_iterable_of_lists at 0x76ef7d02eb20>

bash output from concurrent_futures_example3.py , no problem:

Output:
pedro@pedro-HP:~/myPython/yield/tutorial2014$ ./concurrent_futures_example3.py
Task 1: Result = 2

Task 2: Result = 4

Task 3: Result = 6

Task 4: Result = 8

Task 5: Result = 10

Results: [2, 4, 6, 8, 10]
pedro@pedro-HP:~/myPython/yield/tutorial2014$ 

Should never ever run parallel/asynchronous code in Idle shell or Editor,
always from command line so thread/process don't interfere with what Idle/Editors use.

The reason why it works in IDLE and not in command line/bash might be related to how IDLE manages its environment versus how a bash session does.
It's possible that IDLE keeps the interpreter alive longer than the bash session does, which might allow the ThreadPoolExecutor to complete its tasks before being shut down.

I get the same error and work in IDLE,i would skip this as mention bye David this is a Bizarre💀 example.
This code may work fine on a older slower computer.
Could maybe look into and try to rewrite code,but i would just skip this.

You can look into post if want example of concurrent.futures.
so a example that dos a real world task and can see speedup as you get 200 images download fast.

Thanks, I will skip the offending example!