Python Forum
Beautifulsoup doesn't scrape page (python 2.7)
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Beautifulsoup doesn't scrape page (python 2.7)
#1
Hi,

Python newb here so go easy on me, using 2.7 because I am following a video course that involves scraping web pages for data and the instructor is demonstrating with that version, but when he does it with Spyder, the console shows the output data. Mine doesn't.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
from os import chdir
import pandas as pd
import csv
#webscraping functions
from bs4 import BeautifulSoup
import urllib2
 
chdir("C:\\Users\\Admin\\Documents\\test")
 
header = {'User-Agent': 'Mozilla/5.0'}
req = urllib2.Request(istst, headers = header)
page = urllib2.urlopen(req)
soup = BeautifulSoup(page, "lxml")
soup
What I wanna do is save the data to a .csv but right now I'm trying to find out why beautifulsoup is failing.

Thanks.

EDIT: I am on Windows 7 x64.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  how to scrape page that works dynamicaly? samuelbachorik 0 1,382 Sep-23-2023, 10:38 AM
Last Post: samuelbachorik
  Strange ModuleNotFound Error on BeautifulSoup for Python 3.11 Gaberson19 1 2,976 Jul-13-2023, 10:38 AM
Last Post: Gaurav_Kumar
  How can I web scrape the "alt" attribute from a "img" tag with Python? cisky 1 6,422 Aug-19-2022, 04:59 AM
Last Post: snippsat
Lightbulb Python Obstacles | Kung-Fu | Full File HTML Document Scrape and Store it in MariaDB BrandonKastning 5 4,158 Dec-29-2021, 02:26 AM
Last Post: BrandonKastning
  Python Obstacles | American Kenpo | Wiki Scrape URL/Table and Store it in MariaDB BrandonKastning 6 4,139 Dec-29-2021, 12:38 AM
Last Post: BrandonKastning
  Python Obstacles | Karate | HTML/Scrape Specific Tag and Store it in MariaDB BrandonKastning 8 4,665 Nov-22-2021, 01:38 AM
Last Post: BrandonKastning
  Python BeautifulSoup gives unusable text? dggo666 0 1,830 Oct-29-2021, 05:12 AM
Last Post: dggo666
  Python BeautifulSoup IndexError: list index out of range rhat398 1 7,675 May-28-2021, 09:09 PM
Last Post: Daring_T
  Python 3.9 : BeautifulSoup: 'NoneType' object has no attribute 'text' fudgemasterultra 1 10,942 Mar-03-2021, 09:40 AM
Last Post: Larz60+
  to scrape wiki-page: getting back the results - can i use pandas also apollo 2 3,451 Feb-09-2021, 03:57 PM
Last Post: apollo

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020