Python Forum
Problem with concurrent.futures.
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Problem with concurrent.futures.
#1
Hello everyone,

I'm trying to use:

with concurrent.futures.ProcessPoolExecutor() as executor:

but the code just keeps proceeding further before the parallel processing finishes. I've searched online, but the solution was not clear to me.
Any hints would be highly appreciated.

Thanks
Reply
#2
Can give a working example that have used in my tutorial before.
So this downloads 200 images from xkcd in about 10-sec.
Without concurrent.futures the same will take about 1,15 minute.
import requests
from bs4 import BeautifulSoup
import concurrent.futures
import os

def image_down(url):
    url_get = requests.get(url)
    soup = BeautifulSoup(url_get.content, 'lxml')
    link = soup.find('div', id='comic').find('img').get('src')
    link = link.replace('//', 'http://')
    img_name = os.path.basename(link)
    img = requests.get(link)
    with open(img_name, 'wb') as f_out:
        f_out.write(img.content)

if __name__ == '__main__':
    start_img = 1
    stop_img = 200
    with concurrent.futures.ProcessPoolExecutor(max_workers=10) as executor:
        for numb in range(start_img, stop_img):
            url = f'http://xkcd.com/{numb}/'
            executor.submit(image_down, url)
Reply
#3
I'm using the following:

    if __name__ == '__main__':
        with concurrent.futures.ProcessPoolExecutor() as executor:
           f_map = executor.map(functionname,[input])
Yoriz write Sep-01-2021, 04:52 PM:
Please post all code, output and errors (in their entirety) between their respective tags. Refer to BBCode help topic on how to post. Use the "Preview Post" button to make sure the code is presented as you expect before hitting the "Post Reply/Thread" button.
Reply
#4
(Sep-01-2021, 04:49 PM)thunderspeed Wrote: I'm using the following:
This is not enough info.
Just [input] in the map argument what do think this will do?

If i should use map() in my code it could be done line this.
if __name__ == '__main__':
    with concurrent.futures.ProcessPoolExecutor(max_workers=10) as executor:   
        executor.map(image_down, [f'http://xkcd.com/{numb}/' for numb in range(1,200)])
So now use list comprehension to make a list with url's.
I preferer submit() in first code.
thunderspeed likes this post
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Concurrent futures threading running at same speed as non-threading billykid999 13 1,716 May-03-2023, 08:22 AM
Last Post: billykid999
  Class variables and Multiprocessing(or concurrent.futures.ProcessPoolExecutor) Tomli 5 3,779 Nov-12-2021, 09:55 PM
Last Post: snippsat
  concurrent.futures help samuelbachorik 2 1,706 Aug-22-2021, 07:20 AM
Last Post: bowlofred
  Trying to understand concurrent.futures.ThreadPoolExecutor Laxminarsaiah 0 1,594 Dec-18-2019, 12:43 PM
Last Post: Laxminarsaiah
  asyncio: run_until_complete() returns when the first of gathered Futures is set saavedra29 0 3,918 Dec-23-2017, 10:24 PM
Last Post: saavedra29
  Need my program to run concurrent operations cygnus_X1 1 3,100 Jul-21-2017, 10:49 PM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020