Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How do I scrape a web page?
#1
Photo 
All:
I want to scrape this web-page using Python, but specifically I want to retrieve the data that I've indicated below:
Here's the webpage:
https://www.baseball-reference.com/teams...ders.shtml

The specific data is located about 2/3 of the way down the page and has the title "Most Common Batting Orders".
I need the data from the first column:
Rose
Griffey
Morgan
Bench
Perez
Geronimo
Concepcion
Foster
Pitcher


Thanks is advance.
Reply
#2
most of this page is rendered from JavaScript, so you must use a scraping tool that will expand the JavaScript before searching for the table. This is best done using selenium.

run through the web scraping tutorials on this forum here (only takes a few hours):
web scraping part 1
web scraping part 2

Selenium is covered in part 2, but running through part 1 would be beneficial.
Reply
#3
You can use the Python module selenium to scrape this web page. Selenium will starts your browser and interact with it through Python. Selenium needs a driver, if you use Firefox you need GeckoDriver and for Chrome you need ChromeDriver.

Then you can start it like so

from selenium import webdriver
 
driver = webdriver.Firefox()
driver.get("https://dev.to")
and find the element by id or name

You can find the element id in html, the table has the id "id="grid_table_642496". Then it would be

element=browser.find_element(By.ID,"grid_table_642496")
from there grab the data you need
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  how to scrape page that works dynamicaly? samuelbachorik 0 683 Sep-23-2023, 10:38 AM
Last Post: samuelbachorik
  to scrape wiki-page: getting back the results - can i use pandas also apollo 2 2,602 Feb-09-2021, 03:57 PM
Last Post: apollo
  Beautifulsoup doesn't scrape page (python 2.7) Hikki 0 1,951 Aug-01-2020, 05:54 PM
Last Post: Hikki
  use Xpath in Python :: libxml2 for a page-to-page skip-setting apollo 2 3,580 Mar-19-2020, 06:13 PM
Last Post: apollo
  scrape data 1 go to next page scrape data 2 and so on alkaline3 6 5,090 Mar-13-2020, 07:59 PM
Last Post: alkaline3
  How do i scrape website whose page changes using javsacript _dopostback function and Prince_Bhatia 1 7,161 Aug-06-2018, 09:45 AM
Last Post: wavic
  Scrape Facebook page user posts text stockholm 6 8,285 May-08-2017, 12:24 PM
Last Post: Joseph_f2

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020