Python Forum
Trying to understand concurrent.futures.ThreadPoolExecutor
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Trying to understand concurrent.futures.ThreadPoolExecutor
#1
Hi There

I am trying to understand ThreadPoolExecutor api in python

Here is my code.
import concurrent.futures
import threading
import urllib.request

URLS = ['http://www.foxnews.com/',
        'http://www.cnn.com/',
        'http://europe.wsj.com/',
        'http://www.bbc.co.uk/',
        'http://some-made-up-domain.com/']


def load_url_one(url, timeout):
    print('Current thread in load_url_one()', threading.current_thread().getName())
    with urllib.request.urlopen(url, timeout=timeout) as conn:
        return conn.read()


def load_url_two(url, timeout):
    print('Current thread in load_url_two()', threading.current_thread().getName())
    with urllib.request.urlopen(url, timeout=timeout) as conn:
        return conn.read()


def get_data_from_load_url_one():
    with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
        future_to_url = {executor.submit(load_url_one, url, 60): url for url in URLS}
        for future in concurrent.futures.as_completed(future_to_url):
            url = future_to_url[future]
            try:
                data = future.result()
            except Exception as exc:
                print('%r generated an exception: %s' % (url, exc))
            else:
                print('%r page is %d bytes' % (url, len(data)))


def get_data_from_load_url_two():
    with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
        future_to_url = {executor.submit(load_url_two, url, 60): url for url in URLS}
        for future in concurrent.futures.as_completed(future_to_url):
            url = future_to_url[future]
            try:
                data = future.result()
            except Exception as exc:
                print('%r generated an exception: %s' % (url, exc))
            else:
                print('%r page is %d bytes' % (url, len(data)))


# Now get_data_from_load_url_one() and get_data_from_load_url_two() functions will triggered in a
# loop
for i in range(1, 5):
    get_data_from_load_url_one()
    get_data_from_load_url_two()
# once i ran this output i can the output as below (click to open):


From this output i am trying to understand is that,
Q1. What is mean by
ThreadPoolExecutor-0_0,ThreadPoolExecutor-0_1,ThreadPoolExecutor-0_2......
ThreadPoolExecutor-1_0,ThreadPoolExecutor-1_1,ThreadPoolExecutor-1_2......
ThreadPoolExecutor-2_0,ThreadPoolExecutor-2_1,ThreadPoolExecutor-2_2......
ThreadPoolExecutor-3_0,ThreadPoolExecutor-1_1,ThreadPoolExecutor-3_2......
Q2. is it really running parallelly?
Q3. are multiple ThreadPools getting created?

can any one please make me understand ThreadPoolExecutor concept clearly.

Thanks
Laxmi
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Concurrent futures threading running at same speed as non-threading billykid999 13 1,716 May-03-2023, 08:22 AM
Last Post: billykid999
  How to timeout a task using the ThreadpoolExecutor? lowercase00 2 2,399 Feb-07-2023, 05:44 PM
Last Post: deanhystad
  Class variables and Multiprocessing(or concurrent.futures.ProcessPoolExecutor) Tomli 5 3,784 Nov-12-2021, 09:55 PM
Last Post: snippsat
  Problem with concurrent.futures. thunderspeed 3 1,991 Sep-01-2021, 05:21 PM
Last Post: snippsat
  concurrent.futures help samuelbachorik 2 1,706 Aug-22-2021, 07:20 AM
Last Post: bowlofred
  ThreadPoolExecutor read file to list DaLiPy 3 6,166 Jun-11-2019, 05:55 AM
Last Post: DaLiPy
  asyncio: run_until_complete() returns when the first of gathered Futures is set saavedra29 0 3,918 Dec-23-2017, 10:24 PM
Last Post: saavedra29
  Need my program to run concurrent operations cygnus_X1 1 3,101 Jul-21-2017, 10:49 PM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020