Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Scrap Yahoo Finance using BS4
#1
Hi All,

I am trying to scrap some information from yahoo finance.
I could collect what I want but I have an issue to scrap the price !
I attached a picture to show the information about the website code :
https://finance.yahoo.com/quote/INTC/key...ics?p=INTC


   
LastPrice = soup.find_all('span',attrs={'class':'Trsdu(0.3s) Fw(b) Fz(36px) Mb(-4px) D(ib)'})
print(LastPrice)
the output is always empty
Output:
[]
any reason why it is empty ?
Reply
#2
don't parse the website with BeautifulSoup. Have a better look at the requests that are send via browser and you will see you can get directly all information as json
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply
#3
(Aug-23-2018, 10:48 AM)buran Wrote: don't parse the website with BeautifulSoup. Have a better look at the requests that are send via browser and you will see you can get directly all information as json

JSON doesn't have all information that exists on the website! Many missing information. you can look for the moving average 50 and 200. they don't exist !
thats why I want to scrape the website better.
Reply
#4
Quote:don't parse the website with BeautifulSoup. Have a better look at the requests that are send via browser and you will see you can get directly all information as json
How is this done?, I don't know about this
Reply
#5
(Aug-23-2018, 05:47 PM)Larz60+ Wrote: How is this done?, I don't know about this

   

as you can see there are several json files that are used to transfer information, e.g.

https://query1.finance.yahoo.com/v8/fina...m&range=1d

you can have a more detail look at them if you wish, to see available info. However OP was right - I was not able to find 50-day and 200-day MA. Maybe I didn't search thoroughly or they are calculated.

I also have script to download option chains using these json files:
https://github.com/boyank/yoc
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply
#6
Ok, now I understand, this applies to the specific URL.
I was thinking (hoping) there might have a hidden json menu or button on my browser.
Reply
#7
(Aug-23-2018, 06:16 PM)buran Wrote:
(Aug-23-2018, 05:47 PM)Larz60+ Wrote: How is this done?, I don't know about this



as you can see there are several json files that are used to transfer information, e.g.

https://query1.finance.yahoo.com/v8/fina...m&range=1d

you can have a more detail look at them if you wish, to see available info. However OP was right - I was not able to find 50-day and 200-day MA. Maybe I didn't search thoroughly or they are calculated.

I also have script to download option chains using these json files:
https://github.com/boyank/yoc

The 50-day and 200-day MA are collected from other websites.
Yahoo states this : 3 Data derived from multiple sources or calculated by Yahoo Finance.

I still need the help for my Beautiful soup scraping !
do anyone know to get the price from the link I shared above ?
Reply
#8
see: https://python-forum.io/Thread-Web-Scraping-part-1
and
https://python-forum.io/Thread-Web-Scraping-part-2

also take a look at: https://gist.github.com/scrapehero/516fc...d41a69b496
which does it all using lxml

but best to use Buran's suggestion
you can fetch the json data with requests, then to load simple as
import json

with open(filename) as fp:
    mydata = json.load(fp)
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Web scrap --Need help Lizardpython 4 954 Oct-01-2023, 11:37 AM
Last Post: Lizardpython
  I tried every way to scrap morningstar financials data without success so far sparkt 2 8,171 Oct-20-2020, 05:43 PM
Last Post: sparkt
  Web Scraping with Yahoo Finance miloellison 1 2,029 Jul-03-2020, 11:12 PM
Last Post: Larz60+
  Web scrap multiple pages anilacem_302 3 3,783 Jul-01-2020, 07:50 PM
Last Post: mlieqo
  Need logic on how to scrap 100K URLs goodmind 2 2,570 Jun-29-2020, 09:53 AM
Last Post: goodmind
  getting financial data from yahoo finance asiaphone12 7 6,876 Jun-15-2020, 05:49 AM
Last Post: mick_g
  Django finance tracker mkb3112 1 1,905 Apr-04-2020, 01:21 PM
Last Post: leeacto
  Scrap a dynamic span hefaz 0 2,659 Mar-07-2020, 02:56 PM
Last Post: hefaz
  scrap by defining 3 functions zarize 0 1,834 Feb-18-2020, 03:55 PM
Last Post: zarize
  Skipping anti-scrap zarize 0 1,853 Jan-17-2020, 11:51 AM
Last Post: zarize

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020