Python Forum
urlib - to use or not to use ( for web scraping )?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
urlib - to use or not to use ( for web scraping )?
#39
Try:
import requests
from bs4 import BeautifulSoup


url = "http://www.pythonscraping.com"
html = requests.get(url, stream=True)
if html.status_code == 200:
    bsObj = BeautifulSoup(html.content, 'html.parser')
    imageLocation = bsObj.find("a", {"id":"logo"}).find("img")["src"]

    image = requests.get(imageLocation)
    if image.status_code == 200:
        with open('img.jpg', 'wb') as out_file:
            out_file.write(image.content)
    else:
        print(f'Problem fetching image status code: {image.status_code}')
else:
    print(f'Problem fetching {url} status code: {html.status_code}')
-- Edit Modified 2nd request, should check status code --
Reply


Messages In This Thread
RE: urlib - to use or not to use ( for web scraping )? - by Larz60+ - Dec-12-2018, 12:44 AM

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020