Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Multithreading
#1
This script is supposed to download images in multithreading processes.
#! python3
# multidx.py - Downloads XKCD comics using multiple threads.

import requests, os, bs4, threading

os.makedirs('xkcd', exist_ok=True)   # store comics in ./xkcd

def downloadXkcd(startComic, endComic):
	for urlNumber in range(startComic, endComic + 1):
		print(f'Downloading page http://xkcd.com/{urlNumber}...')
		res = requests.get(f'http://xkcd.com/{urlNumber}')
		res.raise_for_status()
		soup = bs4.BeautifulSoup(res.text)
		comicElem = soup.select('#comic img')
		if comicElem == []:
			print('Could not find comic image.')
		else:
			comicUrl = comicElem[0].get('src')
			print(f'Downloading image {comicUrl}')
			res = requests.get(comicUrl)
			res.raise_for_status()
			imageFile = open(os.path.join('xkcd', os.path.basename(comicUrl)), 'wb')
			for chunk in res.iter_content(100000):
				imageFile.write(chunk)
			imageFile.close()
			
downloadThreads = []             # a list of all Thread objects
for i in range(0, 1400, 100):    # loops 14 times, creates 14 threads
	downloadThread = threading.Thread(target=downloadXkcd, args(i, i + 99))
	downloadThreads.append(downloadThread)
	downloadThread.start()

# Wait for all threads to end
for downloadThread in downloadThreads:
	downloadThread.join()
print('Done.')
but I receive an error
Error:
C:\Python36\kodovi>multidx.py File "C:\Python36\kodovi\multidx.py", line 29 downloadThread = threading.Thread(target=downloadXkcd, args(i, i + 99)) ^ SyntaxError: positional argument follows keyword argument
I thought that the keyword argument goes first and then positional.

Also, I would appreciate if anyone could explain me the last chunk of code. Why calling a Thread object’s join() method will block until that thread has finished? The assumption is that there’s some code you don’t want to run in the main thread until all the threads have completed.
Reply


Messages In This Thread
Multithreading - by Truman - Feb-10-2019, 01:55 AM
RE: Multithreading - by stullis - Feb-10-2019, 02:57 AM
RE: Multithreading - by scidam - Feb-10-2019, 04:42 AM
RE: Multithreading - by Truman - Feb-11-2019, 01:44 AM
RE: Multithreading - by stullis - Feb-11-2019, 04:32 AM
RE: Multithreading - by Truman - Feb-12-2019, 01:22 AM

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020