Python Forum
Image Scraper (beautifulsoup), stopped working, need to help see why
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Image Scraper (beautifulsoup), stopped working, need to help see why
#9
Can show a example and i use my code Wink
Here use concurrent.futures

import requests
from bs4 import BeautifulSoup
from os import path
import concurrent.futures

def read_url(url):
    headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36'}
    response = requests.get(url, headers=headers)
    soup = BeautifulSoup(response.content, 'lxml')
    img_all = soup.select('div.thread_image_box > a')
    return img_all

def img_multi(img_link):
    print(f'Download --> {path.basename(img_link)}')
    with open(path.basename(img_link), 'wb') as f_out:
        f_out.write(requests.get(img_link).content)

if __name__ == '__main__':
    url = 'https://archive.4plebs.org/hr/thread/2866456/'
    img_all = read_url(url)
    # ThreadPoolExecutor | ProcessPoolExecutor
    with concurrent.futures.ThreadPoolExecutor(max_workers=8) as executor:
        for img in img_all:
            img_link = img.get('href')
            executor.submit(img_multi, img_link)
Reply


Messages In This Thread
RE: Image Scraper (beautifulsoup), stopped working, need to help see why - by snippsat - Jan-06-2021, 02:17 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Web scraper tomenzo123 8 4,451 Aug-18-2023, 12:45 PM
Last Post: Gaurav_Kumar
  Web scraper not populating .txt with scraped data BlackHeart 5 1,547 Apr-03-2023, 05:12 PM
Last Post: snippsat
  BeautifulSoup Showing none while extracting image url josephandrew 0 1,964 Sep-20-2021, 11:40 AM
Last Post: josephandrew
  Web scrapping - Stopped working peterjv26 2 3,120 Sep-23-2020, 08:30 AM
Last Post: peterjv26
  not getting image src in my BeautifulSoup csv file farhan275 11 3,795 Sep-14-2020, 04:52 PM
Last Post: buran
  Court Opinion Scraper in Python w/ BS4 (Currently exports to CSV) need help with SQL MidnightDreamer 4 3,052 Mar-12-2020, 09:57 AM
Last Post: BrandonKastning
  Python using BS scraper paulfearn100 1 2,585 Feb-07-2020, 10:22 PM
Last Post: snippsat
  web scraper using pathlib Larz60+ 1 3,230 Oct-16-2017, 05:27 PM
Last Post: Larz60+
  Need alittle hlpl with an image scraper. Blue Dog 8 7,792 Dec-24-2016, 08:09 PM
Last Post: Blue Dog
  Made a very simple email grabber(scraper) Blue Dog 4 6,930 Dec-13-2016, 06:25 AM
Last Post: wavic

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020