Python Forum
Web Scraping Inquiry (Extracting content from a table in asubdomain)
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Web Scraping Inquiry (Extracting content from a table in asubdomain)
#4
(Aug-17-2020, 03:35 AM)DustinKlent Wrote: How do I go about going into the link to extract the stock ticker symbol?
You can open link same way as did with first url.
Could now also make function for open url to avoid the repeat code.
import requests
from bs4 import BeautifulSoup

# Get info from main site
url = 'https://mattrode.com/blog/robinhood-collections-list/'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36'}
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.content, 'lxml')
ol_tag = soup.find('ol')
li_tag = ol_tag.find('li')
link = li_tag.a.get('href')
#print(link)
response = requests.get(link, headers=headers)
new_soup = BeautifulSoup(response.content, 'lxml')
Test.
>>> new_soup.select_one('div.col-13 > section > div > table > tbody > tr:nth-child(1) > td:nth-child(2) > a')
<a class="rh-hyperlink qD5a4psv-CV7GnWdHxLvn AaXTyP3x99eRIDW0ExfYP" href="/stocks/CPRX" rel=""><div><span>CPRX</span></div></a>
>>> new_soup.select_one('div.col-13 > section > div > table > tbody > tr:nth-child(1) > td:nth-child(2) > a').text
'CPRX'
>>> new_soup.select_one('div.col-13 > section > div > table > tbody > tr:nth-child(1) > td:nth-child(3) > a').text
'$3.43'
>>> new_soup.select_one('div.col-13 > section > div > table > tbody > tr:nth-child(1) > td:nth-child(4) > a').text
'0.59%'
>>> new_soup.select_one('div.col-13 > section > div > table > tbody > tr:nth-child(1) > td:nth-child(5) > a').text
'359.02M'
Reply


Messages In This Thread
RE: Web Scraping Inquiry (Extracting content from a table in asubdomain) - by snippsat - Aug-17-2020, 10:10 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Help Scraping links and table from link cartonics 11 1,657 Oct-12-2023, 06:42 AM
Last Post: cartonics
  Scraping data from table into existing dataframe vincer58 1 2,034 Jan-09-2022, 05:15 PM
Last Post: vincer58
  Scraping the page without distorting content oleglpts 5 2,514 Dec-16-2021, 05:08 PM
Last Post: oleglpts
  Python Web Scraping can not getting all HTML content yqqwe123 0 1,654 Aug-02-2021, 08:56 AM
Last Post: yqqwe123
  Need help scraping wikipedia table bborusz2 6 3,278 Dec-01-2020, 11:31 PM
Last Post: snippsat
  Scraping a dynamic data-table in python through AJAX request filozofo 1 3,904 Aug-14-2020, 10:13 AM
Last Post: kashcode
  scraping multiple pages from table bandar 1 2,722 Jun-27-2020, 10:43 PM
Last Post: Larz60+
  BeautifulSoup: Error while extracting a value from an HTML table kawasso 3 3,257 Aug-25-2019, 01:13 AM
Last Post: kawasso
  Web scraping User Generated Content StephenG93 2 2,970 Oct-10-2018, 12:17 AM
Last Post: StephenG93
  Web scraping "fancy" table acehole60 2 4,932 Dec-16-2016, 09:17 AM
Last Post: acehole60

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020