Python Forum
Image Scraper (beautifulsoup), stopped working, need to help see why
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Image Scraper (beautifulsoup), stopped working, need to help see why
#5
(Jan-05-2021, 12:00 AM)woodmister Wrote: So, I'll have to figure that out unless you know a quick way to manupulate it. I haven't fully tested the script you gave me yet, but I did notice it wasn't getting the full sized one.
My Code was just a quick test to see that user agent did work.
The full sized images are under tag a with href for large and src for small.
So if adjust a little.
import requests
from bs4 import BeautifulSoup

url = 'https://archive.4plebs.org/hr/thread/2866456/'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36'}
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.content, 'lxml')
img_all = soup.select('div.thread_image_box > a')
for img in img_all:
    print(img.get('href'))
Output:
https://i.4pcdn.org/hr/1487896415236.jpg https://i.4pcdn.org/hr/1487896485361.jpg https://i.4pcdn.org/hr/1487896543620.jpg https://i.4pcdn.org/hr/1487896605850.jpg https://i.4pcdn.org/hr/1487896666111.jpg https://i.4pcdn.org/hr/1487896726234.jpg .....
Reply


Messages In This Thread
RE: Image Scraper (beautifulsoup), stopped working, need to help see why - by snippsat - Jan-05-2021, 07:48 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Web scraper tomenzo123 8 4,458 Aug-18-2023, 12:45 PM
Last Post: Gaurav_Kumar
  Web scraper not populating .txt with scraped data BlackHeart 5 1,555 Apr-03-2023, 05:12 PM
Last Post: snippsat
  BeautifulSoup Showing none while extracting image url josephandrew 0 1,966 Sep-20-2021, 11:40 AM
Last Post: josephandrew
  Web scrapping - Stopped working peterjv26 2 3,126 Sep-23-2020, 08:30 AM
Last Post: peterjv26
  not getting image src in my BeautifulSoup csv file farhan275 11 3,802 Sep-14-2020, 04:52 PM
Last Post: buran
  Court Opinion Scraper in Python w/ BS4 (Currently exports to CSV) need help with SQL MidnightDreamer 4 3,054 Mar-12-2020, 09:57 AM
Last Post: BrandonKastning
  Python using BS scraper paulfearn100 1 2,587 Feb-07-2020, 10:22 PM
Last Post: snippsat
  web scraper using pathlib Larz60+ 1 3,236 Oct-16-2017, 05:27 PM
Last Post: Larz60+
  Need alittle hlpl with an image scraper. Blue Dog 8 7,795 Dec-24-2016, 08:09 PM
Last Post: Blue Dog
  Made a very simple email grabber(scraper) Blue Dog 4 6,934 Dec-13-2016, 06:25 AM
Last Post: wavic

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020