Python Forum
urlib - to use or not to use ( for web scraping )?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
urlib - to use or not to use ( for web scraping )?
#41
If I understand well os.path.basename is used to get the original name of the image, we can do without it, just by creating any file within open. When you two write a code it looks very simple.
Reply
#42
(Dec-12-2018, 11:09 PM)Truman Wrote: If I understand well os.path.basename is used to get the original name of the image, we can do without it,
Yes it was just to show that option.
A tips get used to test stuff out interactive bye taking parts of code out,a better REPL like IPython or ptpython(what i use) also help.
>>> import os
>>> 
>>> image_location = 'http://www.pythonscraping.com/sites/default/files/lrg_0.jpg'
>>> os.path.basename(image_location)
'lrg_0.jpg'
>>> 
>>> help(os.path.basename)
Help on function basename in module ntpath:

basename(p)
    Returns the final component of a pathname
Reply
#43
Jupyter with python looks useful:
https://hub.mybinder.org/user/ipython-ip...ndex.ipynb#
Now trying to execute some code - Jupyter doesn't recognize bs4

I'm working now on a more complex code and will put it here when I try smth.
Reply
#44
This code takes images from a page and prints them:
import os
import requests
from bs4 import BeautifulSoup

downloadDirectory = "downloaded"
baseUrl = "http://pythonscraping.com"

def getAbsoluteURL(baseUrl, source):
    if source.startswith("http://www."):
        url = "http://"+source[11:]
    elif source.startswith("http://"):
        url = source
    elif source.startswith("www."):
        url = source[4:]
        url = "http://"+source
    else:
        url = baseUrl+"/"+source
    if baseUrl not in url:
        return None
    return url 
	
def getDownloadPath(baseUrl, absoluteUrl, downloadDirectory):
	path = absoluteUrl.replace("www.", "")
	path = path.replace(baseUrl, "")
	path = downloadDirectory+path
	directory = os.path.dirname(path)
	if not os.path.exists(directory):
		os.makedirs(directory)
	return path 

html = requests.get("http://www.pythonscraping.com")
bsObj = BeautifulSoup(html.content, 'html.parser')
downloadList = bsObj.find_all(src=True)
now I want to download these images to the directory downloaded so I added this piece of code:
with open(fileUrl, getDownloadPath(baseUrl, fileUrl, downloadDirectory)) as out_file:
    out_file.write(fileUrl.content)
but I get this:
Error:
http://pythonscraping.com/misc/jquery.js?v=1.4.4 http://pythonscraping.com/misc/jquery.once.js?v=1.2 http://pythonscraping.com/misc/drupal.js?pa2nir http://pythonscraping.com/sites/all/themes/skeletontheme/js/jquery.mobilemenu.js ?pa2nir http://pythonscraping.com/sites/all/modules/google_analytics/googleanalytics.js? pa2nir http://pythonscraping.com/sites/default/files/lrg_0.jpg http://pythonscraping.com/img/lrg%20(1).jpg Traceback (most recent call last): File "C:\Python36\kodovi\crawler3.py", line 76, in <module> with open(fileUrl, getDownloadPath(baseUrl, fileUrl, downloadDirectory)) as out_file: ValueError: invalid mode: 'downloaded/img/lrg%20(1).jpg'
p.s. to add that execution of this code opened folder downloaded and empty folder img within it.
Reply
#45
one error corrected by putting open under for loop
for download in downloadList:
    fileUrl = getAbsoluteURL(baseUrl,download["src"])
    if fileUrl is not None:
        print(fileUrl)
    with open(fileUrl, 'wb', getDownloadPath(baseUrl, fileUrl, downloadDirectory)) as out_file:
        out_file.write(fileUrl.content)
but now an another one appears
Error:
http://pythonscraping.com/misc/jquery.js?v=1.4.4 Traceback (most recent call last): File "C:\Python36\kodovi\crawler3.py", line 76, in <module> with open(fileUrl, 'wb', getDownloadPath(baseUrl, fileUrl, downloadDirectory )) as out_file: TypeError: an integer is required (got type str)
Reply
#46
import os
import requests
from bs4 import BeautifulSoup

downloadDirectory = "downloaded"
baseUrl = "http://pythonscraping.com"

def getAbsoluteURL(baseUrl, source):
    if source.startswith("http://www."):
        url = "http://"+source[11:]
    elif source.startswith("http://"):
        url = source
    elif source.startswith("www."):
        url = source[4:]
        url = "http://"+source
    else:
        url = baseUrl+"/"+source
    if baseUrl not in url:
        return None
    return url 
	
def getDownloadPath(baseUrl, absoluteUrl, downloadDirectory):
    path = absoluteUrl.replace("www.", "")
    path = path.replace(baseUrl, "")
    path = downloadDirectory+path
    directory = os.path.dirname(path)
    if not os.path.exists(directory):
        os.makedirs(directory)
    return path


html = requests.get("http://www.pythonscraping.com")
bsObj = BeautifulSoup(html.content, 'html.parser')
downloadList = bsObj.find_all('img')

for download in downloadList:
    fileUrl = getAbsoluteURL(baseUrl,download["src"])
    if fileUrl is not None:
        print(fileUrl)
    r = requests.get(fileUrl, allow_redirects=True)
    filename = fileUrl.split('/')[-1]
    with open(filename, 'wb') as out_file:
        out_file.write(r.content)
I made some correction in last 10 lines but the problem is now that it completely ommits folder 'downloaded' and getDownloadPath function.
Reply


Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020