Python Forum
Retrieve website content using Python?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Retrieve website content using Python?
#1
If it helps, I'm using Python version 3.1.

This site has good users, and I'm sure you can help me. Anyway, I am trying to access the contents of this weblink. I researched Google and tried several ways, but they were unsuccessful. I expected this task to be simple, but I'm having trouble. When I write a project, I install packages from PyPI. The best and most common library for this is requested. It provides many handy but powerful features. But my project doesn't install its dependencies, i.e., it's limited to things built into the standard library. What's your advice? Huh



urllib and urllib2 output:
>>> import urllib2
Traceback (most recent call last):
  File "<pyshell#0>", line 1, in <module>
    import urllib2
ImportError: No module named urllib2
>>> import urllib
>>> urllib.urlopen("http://www.python.org")
Traceback (most recent call last):
  File "<pyshell#2>", line 1, in <module>
    urllib.urlopen("http://www.python.org")
AttributeError: 'module' object has no attribute 'urlopen'
>>> 
Python 3 response
I appreciate it, Jason.
import urllib.request
page = urllib.request.urlopen('http://services.runescape.com/m=hiscore/ranking?table=0&category_type=0&time_filter=0&date=1519066080774&user=zezima')
print(page.read())
Reply
#2
from urllib.request import Request, urlopen

req = Request('http://services.runescape.com/m=hiscore/ranking?table=0&category_type=0&time_filter=0&date=1519066080774&user=zezima', 
                              headers={'User-Agent': 'Mozilla/5.0'})
page = urlopen(req).read().decode()
print(page)
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Python Obstacles | Krav Maga | Wiki Scraped Content [Column Copy] BrandonKastning 4 2,244 Jan-03-2022, 06:59 AM
Last Post: BrandonKastning
  Python Obstacles | Kapap | Wiki Scraped Content [Column Nulling] BrandonKastning 2 1,743 Jan-03-2022, 04:26 AM
Last Post: BrandonKastning
  I want to create an automated website in python mkdhrub1 2 2,441 Dec-27-2021, 11:27 PM
Last Post: Larz60+
  Python Web Scraping can not getting all HTML content yqqwe123 0 1,651 Aug-02-2021, 08:56 AM
Last Post: yqqwe123
  Python to build website Methew324 1 2,244 Dec-15-2020, 05:57 AM
Last Post: buran
  Scraping all website text using Python MKMKMKMK 1 2,095 Nov-26-2020, 10:35 PM
Last Post: Larz60+
  Python Webscraping with a Login Website warriordazza 0 2,610 Jun-07-2020, 07:04 AM
Last Post: warriordazza
  Python tool based on website? zarize 2 2,500 Mar-21-2020, 02:25 PM
Last Post: zarize
  requests problem in python script "content type" abdlwafitahiri 4 3,235 Dec-29-2019, 02:29 PM
Last Post: abdlwafitahiri
  Python + request from specific website - please help hoff1022 8 4,326 Feb-14-2019, 06:52 PM
Last Post: buran

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020