Python Forum

Full Version: Retrieve website content using Python?
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
If it helps, I'm using Python version 3.1.

This site has good users, and I'm sure you can help me. Anyway, I am trying to access the contents of this weblink. I researched Google and tried several ways, but they were unsuccessful. I expected this task to be simple, but I'm having trouble. When I write a project, I install packages from PyPI. The best and most common library for this is requested. It provides many handy but powerful features. But my project doesn't install its dependencies, i.e., it's limited to things built into the standard library. What's your advice? Huh



urllib and urllib2 output:
>>> import urllib2
Traceback (most recent call last):
  File "<pyshell#0>", line 1, in <module>
    import urllib2
ImportError: No module named urllib2
>>> import urllib
>>> urllib.urlopen("http://www.python.org")
Traceback (most recent call last):
  File "<pyshell#2>", line 1, in <module>
    urllib.urlopen("http://www.python.org")
AttributeError: 'module' object has no attribute 'urlopen'
>>> 
Python 3 response
I appreciate it, Jason.
import urllib.request
page = urllib.request.urlopen('http://services.runescape.com/m=hiscore/ranking?table=0&category_type=0&time_filter=0&date=1519066080774&user=zezima')
print(page.read())
from urllib.request import Request, urlopen

req = Request('http://services.runescape.com/m=hiscore/ranking?table=0&category_type=0&time_filter=0&date=1519066080774&user=zezima', 
                              headers={'User-Agent': 'Mozilla/5.0'})
page = urlopen(req).read().decode()
print(page)