Python Forum
urlib - to use or not to use ( for web scraping )?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
urlib - to use or not to use ( for web scraping )?
#44
This code takes images from a page and prints them:
import os
import requests
from bs4 import BeautifulSoup

downloadDirectory = "downloaded"
baseUrl = "http://pythonscraping.com"

def getAbsoluteURL(baseUrl, source):
    if source.startswith("http://www."):
        url = "http://"+source[11:]
    elif source.startswith("http://"):
        url = source
    elif source.startswith("www."):
        url = source[4:]
        url = "http://"+source
    else:
        url = baseUrl+"/"+source
    if baseUrl not in url:
        return None
    return url 
	
def getDownloadPath(baseUrl, absoluteUrl, downloadDirectory):
	path = absoluteUrl.replace("www.", "")
	path = path.replace(baseUrl, "")
	path = downloadDirectory+path
	directory = os.path.dirname(path)
	if not os.path.exists(directory):
		os.makedirs(directory)
	return path 

html = requests.get("http://www.pythonscraping.com")
bsObj = BeautifulSoup(html.content, 'html.parser')
downloadList = bsObj.find_all(src=True)
now I want to download these images to the directory downloaded so I added this piece of code:
with open(fileUrl, getDownloadPath(baseUrl, fileUrl, downloadDirectory)) as out_file:
    out_file.write(fileUrl.content)
but I get this:
Error:
http://pythonscraping.com/misc/jquery.js?v=1.4.4 http://pythonscraping.com/misc/jquery.once.js?v=1.2 http://pythonscraping.com/misc/drupal.js?pa2nir http://pythonscraping.com/sites/all/themes/skeletontheme/js/jquery.mobilemenu.js ?pa2nir http://pythonscraping.com/sites/all/modules/google_analytics/googleanalytics.js? pa2nir http://pythonscraping.com/sites/default/files/lrg_0.jpg http://pythonscraping.com/img/lrg%20(1).jpg Traceback (most recent call last): File "C:\Python36\kodovi\crawler3.py", line 76, in <module> with open(fileUrl, getDownloadPath(baseUrl, fileUrl, downloadDirectory)) as out_file: ValueError: invalid mode: 'downloaded/img/lrg%20(1).jpg'
p.s. to add that execution of this code opened folder downloaded and empty folder img within it.
Reply


Messages In This Thread
RE: urlib - to use or not to use ( for web scraping )? - by Truman - Dec-15-2018, 12:34 AM

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020