Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 Scrap Yahoo Finance using BS4
#1
Hi All,

I am trying to scrap some information from yahoo finance.
I could collect what I want but I have an issue to scrap the price !
I attached a picture to show the information about the website code :
https://finance.yahoo.com/quote/INTC/key...ics?p=INTC


   
LastPrice = soup.find_all('span',attrs={'class':'Trsdu(0.3s) Fw(b) Fz(36px) Mb(-4px) D(ib)'})
print(LastPrice)
the output is always empty
Output:
[]
any reason why it is empty ?
Quote
#2
don't parse the website with BeautifulSoup. Have a better look at the requests that are send via browser and you will see you can get directly all information as json
Quote
#3
(Aug-23-2018, 10:48 AM)buran Wrote: don't parse the website with BeautifulSoup. Have a better look at the requests that are send via browser and you will see you can get directly all information as json

JSON doesn't have all information that exists on the website! Many missing information. you can look for the moving average 50 and 200. they don't exist !
thats why I want to scrape the website better.
Quote
#4
Quote:don't parse the website with BeautifulSoup. Have a better look at the requests that are send via browser and you will see you can get directly all information as json
How is this done?, I don't know about this
Quote
#5
(Aug-23-2018, 05:47 PM)Larz60+ Wrote: How is this done?, I don't know about this

   

as you can see there are several json files that are used to transfer information, e.g.

https://query1.finance.yahoo.com/v8/fina...m&range=1d

you can have a more detail look at them if you wish, to see available info. However OP was right - I was not able to find 50-day and 200-day MA. Maybe I didn't search thoroughly or they are calculated.

I also have script to download option chains using these json files:
https://github.com/boyank/yoc
Quote
#6
Ok, now I understand, this applies to the specific URL.
I was thinking (hoping) there might have a hidden json menu or button on my browser.
Quote
#7
(Aug-23-2018, 06:16 PM)buran Wrote:
(Aug-23-2018, 05:47 PM)Larz60+ Wrote: How is this done?, I don't know about this



as you can see there are several json files that are used to transfer information, e.g.

https://query1.finance.yahoo.com/v8/fina...m&range=1d

you can have a more detail look at them if you wish, to see available info. However OP was right - I was not able to find 50-day and 200-day MA. Maybe I didn't search thoroughly or they are calculated.

I also have script to download option chains using these json files:
https://github.com/boyank/yoc

The 50-day and 200-day MA are collected from other websites.
Yahoo states this : 3 Data derived from multiple sources or calculated by Yahoo Finance.

I still need the help for my Beautiful soup scraping !
do anyone know to get the price from the link I shared above ?
Quote
#8
see: https://python-forum.io/Thread-Web-Scraping-part-1
and
https://python-forum.io/Thread-Web-Scraping-part-2

also take a look at: https://gist.github.com/scrapehero/516fc...d41a69b496
which does it all using lxml

but best to use Buran's suggestion
you can fetch the json data with requests, then to load simple as
import json

with open(filename) as fp:
    mydata = json.load(fp)
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  page impossible to scrap? :O zarize 2 116 Oct-03-2019, 02:44 PM
Last Post: zarize
  Scrap a value from website harsush 1 154 Aug-29-2019, 01:57 PM
Last Post: snippsat
  Scrap text out of td table from URLS Gochix2020 4 324 Aug-03-2019, 02:56 AM
Last Post: Larz60+
  scrap macrotrends mr_byte31 7 460 Aug-02-2019, 12:02 AM
Last Post: mr_byte31
  Scrap arbitrage odds -help Gochix2020 3 264 Jul-31-2019, 10:45 AM
Last Post: Gochix2020
  i am trying to web scrap this .asp nufan0000 1 253 May-30-2019, 02:27 AM
Last Post: Larz60+
  Searching yahoo with selenium Truman 19 1,747 Oct-13-2018, 11:56 PM
Last Post: snippsat
  [Hlep]Scrap webiste mr_byte31 2 688 Jul-21-2018, 08:11 PM
Last Post: Larz60+
  webscraping yahoo data - custom date implementation Jens89 4 1,286 Jun-19-2018, 08:02 AM
Last Post: Jens89
  beatifulsoup scrap td tag. piuk3man 1 773 Jun-11-2018, 06:16 AM
Last Post: buran

Forum Jump:


Users browsing this thread: 1 Guest(s)