Jun-10-2019, 09:45 PM
The Python Documentation shows a good example: threadpoolexecutor-example
If you use this as example and replace the
If you use this as example and replace the
load_url
function with read_file
it could work like this:import time from random import randint, choice from concurrent.futures import ( ThreadPoolExecutor, as_completed, ) def computation(desc): print('Slow task', desc) pause = randint(1,10) time.sleep(pause) return desc def read_file(file): # simulation of big file computation('reading ' + file) if choice([True, False]): exc = choice([ZeroDivisionError('Not really'), ValueError('Wrong input'), TypeError('Wrong operation')]) raise exc with open(file) as fd: return fd.read() ############ files = ('Chesterton.txt', 'Alice.txt', 'Chesterton.txt', 'Alice.txt', 'Chesterton.txt', 'Alice.txt') ############ # We can use a with statement to ensure threads are cleaned up promptly with ThreadPoolExecutor(max_workers=5) as executor: # Start the load operations and mark each future with its file future_to_text = {executor.submit(read_file, file): file for file in files} # <- this is a dict # alternative # future_to_text = {} # for file in files: # future = executor.submit(read_file, file) # future_to_text[future] = file # # they keys are the futures, the values are the filenames for future in as_completed(future_to_text): # iterating over a dict directly, returns the key, in this case the futures # as_completed returns the future, as it's as_completed, it may out of order file = future_to_text[future] # text is the future try: data = future.result() except Exception as exc: print(f'{file!r} generated an exception: {exc!r}') else: print(f'{file!r} is {len(data)} chars long')
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
All humans together. We don't need politicians!