Python Forum
ThreadPoolExecutor read file to list
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
ThreadPoolExecutor read file to list
#4
(Jun-10-2019, 08:35 PM)Yoriz Wrote: The first argument passed to the method submit should be a callable function.
Thanks for the replies.

The solution from Yoriz made the script working and bring back the results. However it does not multithread this way. The lines are executed in sequence.
I guess the function is the key here. Should this be a multiprocess or lies the solution in a piece of the code from Dead_EyE?

with ThreadPoolExecutor(max_workers=5) as executor:
    # Start the load operations and mark each future with its file
    future_to_text = {executor.submit(read_file, file): file for file in files} # <- this is a dict

# Could the solution be in getting the files in a dict and then submit them like above? Would they be multithreaded then?
What am I missing here?

My current half working code:
def _ListyReader1(xfile):
    ArtLev = list(open(xfile, encoding='ANSI'))
    print('len(reader): ', len(ArtLev))
    return ArtLev

with ThreadPoolExecutor(max_workers=4) as e:
    a = e.submit(_ListyReader1, ArtLev_filename)
    print(len(a.result()))
    ArtLev = a.result()

    b = e.submit(_ListyReader1, ArtPrijs_filename)
    print(len(b.result()))
    ArtPrijs = b.result()



end = time.time()
print('Inlezen klaar in: ', end - start)
print(len(ArtLev))
print(len(ArtPrijs))
Results are:
len(reader): 1281115
1281115
len(reader): 1281115
1281115
Inlezen klaar in: 17.544724225997925
1281115
1281115
Reply


Messages In This Thread
ThreadPoolExecutor read file to list - by DaLiPy - Jun-10-2019, 07:59 PM
RE: ThreadPoolExecutor read file to list - by Yoriz - Jun-10-2019, 08:35 PM
RE: ThreadPoolExecutor read file to list - by DaLiPy - Jun-11-2019, 05:55 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Connecting to Remote Server to read contents of a file ChaitanyaSharma 1 171 May-03-2024, 07:23 AM
Last Post: Pedroski55
  PyYAML read list of int zisco 2 356 Apr-02-2024, 12:36 PM
Last Post: zisco
  Recommended way to read/create PDF file? Winfried 3 2,939 Nov-26-2023, 07:51 AM
Last Post: Pedroski55
  python Read each xlsx file and write it into csv with pipe delimiter mg24 4 1,536 Nov-09-2023, 10:56 AM
Last Post: mg24
  How to read module/class from list of strings? popular_dog 1 500 Oct-04-2023, 03:08 PM
Last Post: deanhystad
  read file txt on my pc to telegram bot api Tupa 0 1,158 Jul-06-2023, 01:52 AM
Last Post: Tupa
  parse/read from file seperated by dots giovanne 5 1,146 Jun-26-2023, 12:26 PM
Last Post: DeaD_EyE
  Formatting a date time string read from a csv file DosAtPython 5 1,369 Jun-19-2023, 02:12 PM
Last Post: DosAtPython
  How do I read and write a binary file in Python? blackears 6 6,919 Jun-06-2023, 06:37 PM
Last Post: rajeshgk
  Read csv file with inconsistent delimiter gracenz 2 1,221 Mar-27-2023, 08:59 PM
Last Post: deanhystad

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020