Python Forum
ThreadPoolExecutor read file to list
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
ThreadPoolExecutor read file to list
#3
The Python Documentation shows a good example: threadpoolexecutor-example
If you use this as example and replace the load_url function with read_file it could work like this:

import time
from random import randint, choice
from concurrent.futures import (
    ThreadPoolExecutor,
    as_completed,
    )


def computation(desc):
    print('Slow task', desc)
    pause = randint(1,10)
    time.sleep(pause)
    return desc


def read_file(file):
    # simulation of big file
    computation('reading ' + file)
    if choice([True, False]):
        exc = choice([ZeroDivisionError('Not really'), ValueError('Wrong input'), TypeError('Wrong operation')])
        raise exc
    with open(file) as fd:
        return fd.read()

############
files = ('Chesterton.txt', 'Alice.txt', 'Chesterton.txt', 'Alice.txt', 'Chesterton.txt', 'Alice.txt')
############

# We can use a with statement to ensure threads are cleaned up promptly
with ThreadPoolExecutor(max_workers=5) as executor:
    # Start the load operations and mark each future with its file
    future_to_text = {executor.submit(read_file, file): file for file in files} # <- this is a dict
    # alternative
    # future_to_text = {}
    # for file in files:
    #     future = executor.submit(read_file, file)
    #     future_to_text[future] = file
    #
    # they keys are the futures, the values are the filenames
    for future in as_completed(future_to_text):
        # iterating over a dict directly, returns the key, in this case the futures
        # as_completed returns the future, as it's as_completed, it may out of order
        file = future_to_text[future]
        # text is the future
        try:
            data = future.result()
        except Exception as exc:
            print(f'{file!r} generated an exception: {exc!r}')
        else:
            print(f'{file!r} is {len(data)} chars long')
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply


Messages In This Thread
ThreadPoolExecutor read file to list - by DaLiPy - Jun-10-2019, 07:59 PM
RE: ThreadPoolExecutor read file to list - by Yoriz - Jun-10-2019, 08:35 PM
RE: ThreadPoolExecutor read file to list - by DeaD_EyE - Jun-10-2019, 09:45 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Connecting to Remote Server to read contents of a file ChaitanyaSharma 1 295 May-03-2024, 07:23 AM
Last Post: Pedroski55
  PyYAML read list of int zisco 2 388 Apr-02-2024, 12:36 PM
Last Post: zisco
  Recommended way to read/create PDF file? Winfried 3 2,986 Nov-26-2023, 07:51 AM
Last Post: Pedroski55
  python Read each xlsx file and write it into csv with pipe delimiter mg24 4 1,629 Nov-09-2023, 10:56 AM
Last Post: mg24
  How to read module/class from list of strings? popular_dog 1 536 Oct-04-2023, 03:08 PM
Last Post: deanhystad
  read file txt on my pc to telegram bot api Tupa 0 1,188 Jul-06-2023, 01:52 AM
Last Post: Tupa
  parse/read from file seperated by dots giovanne 5 1,175 Jun-26-2023, 12:26 PM
Last Post: DeaD_EyE
  Formatting a date time string read from a csv file DosAtPython 5 1,436 Jun-19-2023, 02:12 PM
Last Post: DosAtPython
  How do I read and write a binary file in Python? blackears 6 7,356 Jun-06-2023, 06:37 PM
Last Post: rajeshgk
  Read csv file with inconsistent delimiter gracenz 2 1,254 Mar-27-2023, 08:59 PM
Last Post: deanhystad

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020