Python Forum
Retrieve website content using Python?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Retrieve website content using Python?
#1
If it helps, I'm using Python version 3.1.

This site has good users, and I'm sure you can help me. Anyway, I am trying to access the contents of this weblink. I researched Google and tried several ways, but they were unsuccessful. I expected this task to be simple, but I'm having trouble. When I write a project, I install packages from PyPI. The best and most common library for this is requested. It provides many handy but powerful features. But my project doesn't install its dependencies, i.e., it's limited to things built into the standard library. What's your advice? Huh



urllib and urllib2 output:
>>> import urllib2
Traceback (most recent call last):
  File "<pyshell#0>", line 1, in <module>
    import urllib2
ImportError: No module named urllib2
>>> import urllib
>>> urllib.urlopen("http://www.python.org")
Traceback (most recent call last):
  File "<pyshell#2>", line 1, in <module>
    urllib.urlopen("http://www.python.org")
AttributeError: 'module' object has no attribute 'urlopen'
>>> 
Python 3 response
I appreciate it, Jason.
import urllib.request
page = urllib.request.urlopen('http://services.runescape.com/m=hiscore/ranking?table=0&category_type=0&time_filter=0&date=1519066080774&user=zezima')
print(page.read())
Reply
#2
from urllib.request import Request, urlopen

req = Request('http://services.runescape.com/m=hiscore/ranking?table=0&category_type=0&time_filter=0&date=1519066080774&user=zezima', 
                              headers={'User-Agent': 'Mozilla/5.0'})
page = urlopen(req).read().decode()
print(page)
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Python Obstacles | Krav Maga | Wiki Scraped Content [Column Copy] BrandonKastning 4 2,238 Jan-03-2022, 06:59 AM
Last Post: BrandonKastning
  Python Obstacles | Kapap | Wiki Scraped Content [Column Nulling] BrandonKastning 2 1,736 Jan-03-2022, 04:26 AM
Last Post: BrandonKastning
  I want to create an automated website in python mkdhrub1 2 2,431 Dec-27-2021, 11:27 PM
Last Post: Larz60+
  Python Web Scraping can not getting all HTML content yqqwe123 0 1,649 Aug-02-2021, 08:56 AM
Last Post: yqqwe123
  Python to build website Methew324 1 2,243 Dec-15-2020, 05:57 AM
Last Post: buran
  Scraping all website text using Python MKMKMKMK 1 2,093 Nov-26-2020, 10:35 PM
Last Post: Larz60+
  Python Webscraping with a Login Website warriordazza 0 2,610 Jun-07-2020, 07:04 AM
Last Post: warriordazza
  Python tool based on website? zarize 2 2,496 Mar-21-2020, 02:25 PM
Last Post: zarize
  requests problem in python script "content type" abdlwafitahiri 4 3,218 Dec-29-2019, 02:29 PM
Last Post: abdlwafitahiri
  Python + request from specific website - please help hoff1022 8 4,325 Feb-14-2019, 06:52 PM
Last Post: buran

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020