Python Forum
scrap macrotrends - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html)
+--- Thread: scrap macrotrends (/thread-20197.html)



scrap macrotrends - mr_byte31 - Jul-31-2019

I was trying to collect some information about stocks I like.
example for stock : https://www.macrotrends.net/stocks/charts/MSFT/microsoft/income-statement?freq=A

I made a script to scrap the table contents in the website.

myURL = 'https://www.macrotrends.net/stocks/charts/MSFT/microsoft/income-statement?freq=A'
    page = requests.get(myURL)
    soup = bs4.BeautifulSoup(page.content, 'html.parser')
    stockData = soup.find_all(class_="jqx-grid-cell jqx-item jqx-grid-cell-selected jqx-fill-state-pressed")
    print(stockData)
The stockData is always empty.
I printed the soup variable and noticed that no values exist in the table.

I tried to check if the website load a json file that has the table content but couldn't find any.
I only found a json files that has the stock names and ticker which is not useful to me.

it seems the website is tricky.

I appreciate any help.


RE: scrap macrotrends - ichabod801 - Jul-31-2019

The json for the table data appears to be in the javascript on the page. I found one of the values in the table on line 1055 of the web site source code. That's inside a <script type="text/javascript"> tag.


RE: scrap macrotrends - snippsat - Jul-31-2019

It's a common problem with sites that use JavaScript,look at Web-scraping part-2.

A quick test,here send page source(now with executed JavaScript) to BS and do parsing there.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
from bs4 import BeautifulSoup
import time

#--| Setup
options = Options()
#options.add_argument("--headless")
browser = webdriver.Chrome(executable_path=r'C:\cmder\bin\chromedriver.exe', options=options)
#--| Parse or automation
browser.get('https://www.macrotrends.net/stocks/charts/MSFT/microsoft/income-statement?freq=A')
time.sleep(3)
soup = BeautifulSoup(browser.page_source, 'lxml')
table = soup.select('#contentjqxgrid > div.jqx-grid-content.jqx-widget-content')
#print(table)

# First row test
#revenue = soup.select('#row0jqxgrid')
#print(revenue)
first_val = soup.select('#row0jqxgrid > div:nth-child(3) > div')
#print('-' * 20)
print(first_val[0].text)
Output:
$125 843



RE: scrap macrotrends - mr_byte31 - Jul-31-2019

(Jul-31-2019, 01:03 PM)ichabod801 Wrote: The json for the table data appears to be in the javascript on the page. I found one of the values in the table on line 1055 of the web site source code. That's inside a <script type="text/javascript"> tag.

I could see the values of the table on line 1055 but this line doesn't exist when I request the link with my script!
it seems this line is created on the fly.

can you provide more information how to get this content from the json ?


@snippsat, selenium will make the whole script very slow since i have a lot of stocks to go through them.


RE: scrap macrotrends - ichabod801 - Jul-31-2019

If the javascript isn't coming through with requests, I don't know what to do. I would defer to snippsat on web scraping. It is not my area of expertise, and he wrote our tutorial on it.


RE: scrap macrotrends - snippsat - Jul-31-2019

Should look if there a is API for Macrotrends.
Can't find a doc for it,but there is way to download.
http://download.macrotrends.net/assets/php/stock_data_export.php?t=[SYMBOL NAME]

# Eg give csv file back
http://download.macrotrends.net/assets/php/stock_data_export.php?t=MSFT
Can be faster if not loading browser using --headless and don't need time.sleep() as put in a first test.
Still take some time as stock sites take some time to load.

The other way can be to inspect site eg network traffic and see if can catch eg json data.
This can vary from easy to very tricky.


RE: scrap macrotrends - mr_byte31 - Aug-01-2019

(Jul-31-2019, 08:34 PM)snippsat Wrote: Should look if there a is API for Macrotrends.
Can't find a doc for it,but there is way to download.
http://download.macrotrends.net/assets/php/stock_data_export.php?t=[SYMBOL NAME]

# Eg give csv file back
http://download.macrotrends.net/assets/php/stock_data_export.php?t=MSFT
Can be faster if not loading browser using --headless and don't need time.sleep() as put in a first test.
Still take some time as stock sites take some time to load.

The other way can be to inspect site eg network traffic and see if can catch eg json data.
This can vary from easy to very tricky.

The API you provided is about price and volume not about financial reports.
anyway thanks for the help.
will the code of the java script give a clue where the data come from ?


RE: scrap macrotrends - mr_byte31 - Aug-02-2019

I looked again in the soup contents.
I think I have the information I need in the following line :
I tried to use the class name = fas fa-chart-bar but i list was empty.

Output:
var originalData = [{"field_name":"<a href='\/stocks\/charts\/MSFT\/microsoft\/revenue'>Revenue<\/a>","popup_icon":"<div class='ajax-chart' data-tipped-options=\"ajax: {data: { t: 'MSFT', s: 'revenue', freq: 'A', statement: 'income-statement' }}\"><i style='font-size:18px; color:#337ab7;' class='fas fa-chart-bar'><\/i><\/span><\/div>","2019-06-30":"125843.00000","2018-06-30":"110360.00000","2017-06-30":"96571.00000","2016-06-30":"91154.00000","2015-06-30":"93580.00000","2014-06-30":"86833.00000","2013-06-30":"77849.00000","2012-06-30":"73723.00000","2011-06-30":"69943.00000","2010-06-30":"62484.00000","2009-06-30":"58437.00000","2008-06-30":"60420.00000","2007-06-30":"51122.00000","2006-06-30":"44282.00000","2005-06-30":"39788.00000"},{"field_name":"<a href='\/stocks\/charts\/MSFT\/microsoft\/cost-goods-sold'>Cost Of Goods Sold<\/a>","popup_icon":"<div class='ajax-chart' data-tipped-options=\"ajax: {data: { t: 'MSFT', s: 'cost-goods-sold', freq: 'A', statement: 'income-statement' }}\"><i style='font-size:18px; color:#337ab7;' class='fas fa-chart-bar'><\/i><\/span><\/div>","2019-06-30":"42910.00000","2018-06-30":"38353.00000","2017-06-30":"34261.00000","2016-06-30":"32780.00000","2015-06-30":"33038.00000","2014-06-30":"27078.00000","2013-06-30":"20385.00000","2012-06-30":"17530.00000","2011-06-30":"15577.00000","2010-06-30":"12395.00000","2009-06-30":"12155.00000","2008-06-30":"11598.00000","2007-06-30":"10693.00000","2006-06-30":"7650.00000","2005-06-30":"6031.00000"},{"field_name":"<a href='\/stocks\/charts\/MSFT\/microsoft\/gross-profit'>Gross Profit<\/a>","popup_icon":"<div class='ajax-chart' data-tipped-options=\"ajax: {data: { t: 'MSFT', s: 'gross-profit', freq: 'A', statement: 'income-statement' }}\"><i style='font-size:18px; color:#337ab7;' class='fas fa-chart-bar'><\/i><\/span><\/div>","2019-06-30":"82933.00000","2018-06-30":"72007.00000","2017-06-30":"62310.00000","2016-06-30":"58374.00000","2015-06-30":"60542.00000","2014-06-30":"59755.00000","2013-06-30":"57464.00000","2012-06-30":"56193.00000","2011-06-30":"54366.00000","2010-06-30":"50089.00000","2009-06-30":"46282.00000","2008-06-30":"48822.00000","2007-06-30":"40429.00000","2006-06-30":"36632.00000","2005-06-30":"33757.00000"},{"field_name":"<a href='\/stocks\/charts\/MSFT\/microsoft\/research-development-expenses'>Research And Development Expenses<\/a>","popup_icon":"<div class='ajax-chart' data-tipped-options=\"ajax: {data: { t: 'MSFT', s: 'research-development-expenses', freq: 'A', statement: 'income-statement' }}\"><i style='font-size:18px; color:#337ab7;' class='fas fa-chart-bar'><\/i><\/span><\/div>","2019-06-30":"16876.00000","2018-06-30":"14726.00000","2017-06-30":"13037.00000","2016-06-30":"11988.00000","2015-06-30":"12046.00000","2014-06-30":"11381.00000","2013-06-30":"10411.00000","2012-06-30":"9811.00000","2011-06-30":"9043.00000","2010-06-30":"8714.00000","2009-06-30":"9010.00000","2008-06-30":"8164.00000","2007-06-30":"7121.00000","2006-06-30":"6584.00000","2005-06-30":"6097.00000"},{"field_name":"<a href='\/stocks\/charts\/MSFT\/microsoft\/selling-general-administrative-expenses'>SG&A Expenses<\/a>","popup_icon":"<div class='ajax-chart' data-tipped-options=\"ajax: {data: { t: 'MSFT', s: 'selling-general-administrative-expenses', freq: 'A', statement: 'income-statement' }}\"><i style='font-size:18px; color:#337ab7;' class='fas fa-chart-bar'><\/i><\/span><\/div>","2019-06-30":"23098.00000","2018-06-30":"22223.00000","2017-06-30":"19942.00000","2016-06-30":"19198.00000","2015-06-30":"20324.00000","2014-06-30":"20488.00000","2013-06-30":"20289.00000","2012-06-30":"18426.00000","2011-06-30":"18162.00000","2010-06-30":"17277.00000","2009-06-30":"16909.00000","2008-06-30":"18387.00000","2007-06-30":"14870.00000","2006-06-30":"13576.00000","2005-06-30":"13099.00000"},{"field_name":"<span style='color:#337ab7;'>Other Operating Income Or Expenses<\/span>","popup_icon":"","2019-06-30":"","2018-06-30":"","2017-06-30":"","2016-06-30":"","2015-06-30":"","2014-06-30":"","2013-06-30":"","2012-06-30":"","2011-06-30":"","2010-06-30":"","2009-06-30":"","2008-06-30":"","2007-06-30":"","2006-06-30":"","2005-06-30":""},{"field_name":"<a href='\/stocks\/charts\/MSFT\/microsoft\/operating-expenses'>Operating Expenses<\/a>","popup_icon":"<div class='ajax-chart' data-tipped-options=\"ajax: {data: { t: 'MSFT', s: 'operating-expenses', freq: 'A', statement: 'income-statement' }}\"><i style='font-size:18px; color:#337ab7;' class='fas fa-chart-bar'><\/i><\/span><\/div>","2019-06-30":"82884.00000","2018-06-30":"75302.00000","2017-06-30":"67546.00000","2016-06-30":"65076.00000","2015-06-30":"75419.00000","2014-06-30":"59074.00000","2013-06-30":"51085.00000","2012-06-30":"51960.00000","2011-06-30":"42782.00000","2010-06-30":"38386.00000","2009-06-30":"38074.00000","2008-06-30":"38149.00000","2007-06-30":"32684.00000","2006-06-30":"27810.00000","2005-06-30":"25227.00000"},{"field_name":"<a href='\/stocks\/charts\/MSFT\/microsoft\/operating-income'>Operating Income<\/a>","popup_icon":"<div class='ajax-chart' data-tipped-options=\"ajax: {data: { t: 'MSFT', s: 'operating-income', freq: 'A', statement: 'income-statement' }}\"><i style='font-size:18px; color:#337ab7;' class='fas fa-chart-bar'><\/i><\/span><\/div>","2019-06-30":"42959.00000","2018-06-30":"35058.00000","2017-06-30":"29025.00000","2016-06-30":"26078.00000","2015-06-30":"18161.00000","2014-06-30":"27759.00000","2013-06-30":"26764.00000","2012-06-30":"21763.00000","2011-06-30":"27161.00000","2010-06-30":"24098.00000","2009-06-30":"20363.00000","2008-06-30":"22271.00000","2007-06-30":"18438.00000","2006-06-30":"16472.00000","2005-06-30":"14561.00000"},{"field_name":"<a href='\/stocks\/charts\/MSFT\/microsoft\/total-non-operating-income-expense'>Total Non-Operating Income\/Expense<\/a>","popup_icon":"<div class='ajax-chart' data-tipped-options=\"ajax: {data: { t: 'MSFT', s: 'total-non-operating-income-expense', freq: 'A', statement: 'income-statement' }}\"><i style='font-size:18px; color:#337ab7;' class='fas fa-chart-bar'><\/i><\/span><\/div>","2019-06-30":"729.00000","2018-06-30":"1416.00000","2017-06-30":"876.00000","2016-06-30":"-439.00000","2015-06-30":"346.00000","2014-06-30":"61.00000","2013-06-30":"288.00000","2012-06-30":"504.00000","2011-06-30":"910.00000","2010-06-30":"915.00000","2009-06-30":"-542.00000","2008-06-30":"1543.00000","2007-06-30":"1663.00000","2006-06-30":"1790.00000","2005-06-30":"2067.00000"},{"field_name":"<a href='\/stocks\/charts\/MSFT\/microsoft\/pre-tax-income'>Pre-Tax Income<\/a>","popup_icon":"<div class='ajax-chart' data-tipped-options=\"ajax: {data: { t: 'MSFT', s: 'pre-tax-income', freq: 'A', statement: 'income-statement' }}\"><i style='font-size:18px; color:#337ab7;' class='fas fa-chart-bar'><\/i><\/span><\/div>","2019-06-30":"43688.00000","2018-06-30":"36474.00000","2017-06-30":"29901.00000","2016-06-30":"25639.00000","2015-06-30":"18507.00000","2014-06-30":"27820.00000","2013-06-30":"27052.00000","2012-06-30":"22267.00000","2011-06-30":"28071.00000","2010-06-30":"25013.00000","2009-06-30":"19821.00000","2008-06-30":"23814.00000","2007-06-30":"20101.00000","2006-06-30":"18262.00000","2005-06-30":"16628.00000"},{"field_name":"<a href='\/stocks\/charts\/MSFT\/microsoft\/total-provision-income-taxes'>Income Taxes<\/a>","popup_icon":"<div class='ajax-chart' data-tipped-options=\"ajax: {data: { t: 'MSFT', s: 'total-provision-income-taxes', freq: 'A', statement: 'income-statement' }}\"><i style='font-size:18px; color:#337ab7;' class='fas fa-chart-bar'><\/i><\/span><\/div>","2019-06-30":"4448.00000","2018-06-30":"19903.00000","2017-06-30":"4412.00000","2016-06-30":"5100.00000","2015-06-30":"6314.00000","2014-06-30":"5746.00000","2013-06-30":"5189.00000","2012-06-30":"5289.00000","2011-06-30":"4921.00000","2010-06-30":"6253.00000","2009-06-30":"5252.00000","2008-06-30":"6133.00000","2007-06-30":"6036.00000","2006-06-30":"5663.00000","2005-06-30":"4374.00000"},{"field_name":"<a href='\/stocks\/charts\/MSFT\/microsoft\/income-after-taxes'>Income After Taxes<\/a>","popup_icon":"<div class='ajax-chart' data-tipped-options=\"ajax: {data: { t: 'MSFT', s: 'income-after-taxes', freq: 'A', statement: 'income-statement' }}\"><i style='font-size:18px; color:#337ab7;' class='fas fa-chart-bar'><\/i><\/span><\/div>","2019-06-30":"39240.00000","2018-06-30":"16571.00000","2017-06-30":"25489.00000","2016-06-30":"20539.00000","2015-06-30":"12193.00000","2014-06-30":"22074.00000","2013-06-30":"21863.00000","2012-06-30":"16978.00000","2011-06-30":"23150.00000","2010-06-30":"18760.00000","2009-06-30":"14569.00000","2008-06-30":"17681.00000","2007-06-30":"14065.00000","2006-06-30":"12599.00000","2005-06-30":"12254.00000"},{"field_name":"<span style='color:#337ab7;'>Other Income<\/span>","popup_icon":"","2019-06-30":"","2018-06-30":"","2017-06-30":"","2016-06-30":"","2015-06-30":"","2014-06-30":"","2013-06-30":"","2012-06-30":"","2011-06-30":"","2010-06-30":"","2009-06-30":"","2008-06-30":"","2007-06-30":"","2006-06-30":"","2005-06-30":""},{"field_name":"<a href='\/stocks\/charts\/MSFT\/microsoft\/income-from-continuous-operations'>Income From Continuous Operations<\/a>","popup_icon":"<div class='ajax-chart' data-tipped-options=\"ajax: {data: { t: 'MSFT', s: 'income-from-continuous-operations', freq: 'A', statement: 'income-statement' }}\"><i style='font-size:18px; color:#337ab7;' class='fas fa-chart-bar'><\/i><\/span><\/div>","2019-06-30":"39240.00000","2018-06-30":"16571.00000","2017-06-30":"25489.00000","2016-06-30":"20539.00000","2015-06-30":"12193.00000","2014-06-30":"22074.00000","2013-06-30":"21863.00000","2012-06-30":"16978.00000","2011-06-30":"23150.00000","2010-06-30":"18760.00000","2009-06-30":"14569.00000","2008-06-30":"17681.00000","2007-06-30":"14065.00000","2006-06-30":"12599.00000","2005-06-30":"12254.00000"},{"field_name":"<span style='color:#337ab7;'>Income From Discontinued Operations<\/span>","popup_icon":"","2019-06-30":"","2018-06-30":"","2017-06-30":"","2016-06-30":"","2015-06-30":"","2014-06-30":"","2013-06-30":"","2012-06-30":"","2011-06-30":"","2010-06-30":"","2009-06-30":"","2008-06-30":"","2007-06-30":"","2006-06-30":"","2005-06-30":""},{"field_name":"<a href='\/stocks\/charts\/MSFT\/microsoft\/net-income'>Net Income<\/a>","popup_icon":"<div class='ajax-chart' data-tipped-options=\"ajax: {data: { t: 'MSFT', s: 'net-income', freq: 'A', statement: 'income-statement' }}\"><i style='font-size:18px; color:#337ab7;' class='fas fa-chart-bar'><\/i><\/span><\/div>","2019-06-30":"39240.00000","2018-06-30":"16571.00000","2017-06-30":"25489.00000","2016-06-30":"20539.00000","2015-06-30":"12193.00000","2014-06-30":"22074.00000","2013-06-30":"21863.00000","2012-06-30":"16978.00000","2011-06-30":"23150.00000","2010-06-30":"18760.00000","2009-06-30":"14569.00000","2008-06-30":"17681.00000","2007-06-30":"14065.00000","2006-06-30":"12599.00000","2005-06-30":"12254.00000"},{"field_name":"<a href='\/stocks\/charts\/MSFT\/microsoft\/ebitda'>EBITDA<\/a>","popup_icon":"<div class='ajax-chart' data-tipped-options=\"ajax: {data: { t: 'MSFT', s: 'ebitda', freq: 'A', statement: 'income-statement' }}\"><i style='font-size:18px; color:#337ab7;' class='fas fa-chart-bar'><\/i><\/span><\/div>","2019-06-30":"54641.00000","2018-06-30":"45319.00000","2017-06-30":"37803.00000","2016-06-30":"33330.00000","2015-06-30":"31616.00000","2014-06-30":"32971.00000","2013-06-30":"30519.00000","2012-06-30":"30923.00000","2011-06-30":"29927.00000","2010-06-30":"26771.00000","2009-06-30":"22925.00000","2008-06-30":"24327.00000","2007-06-30":"19878.00000","2006-06-30":"17375.00000","2005-06-30":"15416.00000"},{"field_name":"<a href='\/stocks\/charts\/MSFT\/microsoft\/ebit'>EBIT<\/a>","popup_icon":"<div class='ajax-chart' data-tipped-options=\"ajax: {data: { t: 'MSFT', s: 'ebit', freq: 'A', statement: 'income-statement' }}\"><i style='font-size:18px; color:#337ab7;' class='fas fa-chart-bar'><\/i><\/span><\/div>","2019-06-30":"42959.00000","2018-06-30":"35058.00000","2017-06-30":"29025.00000","2016-06-30":"26078.00000","2015-06-30":"18161.00000","2014-06-30":"27759.00000","2013-06-30":"26764.00000","2012-06-30":"21763.00000","2011-06-30":"27161.00000","2010-06-30":"24098.00000","2009-06-30":"20363.00000","2008-06-30":"22271.00000","2007-06-30":"18438.00000","2006-06-30":"16472.00000","2005-06-30":"14561.00000"},{"field_name":"<a href='\/stocks\/charts\/MSFT\/microsoft\/basic-shares-outstanding'>Basic Shares Outstanding<\/a>","popup_icon":"<div class='ajax-chart' data-tipped-options=\"ajax: {data: { t: 'MSFT', s: 'basic-shares-outstanding', freq: 'A', statement: 'income-statement' }}\"><i style='font-size:18px; color:#337ab7;' class='fas fa-chart-bar'><\/i><\/span><\/div>","2019-06-30":"7673.00000","2018-06-30":"7700.00000","2017-06-30":"7746.00000","2016-06-30":"7925.00000","2015-06-30":"8177.00000","2014-06-30":"8299.00000","2013-06-30":"8375.00000","2012-06-30":"8396.00000","2011-06-30":"8490.00000","2010-06-30":"8813.00000","2009-06-30":"8945.00000","2008-06-30":"9328.00000","2007-06-30":"9742.00000","2006-06-30":"10438.00000","2005-06-30":"10839.00000"},{"field_name":"<a href='\/stocks\/charts\/MSFT\/microsoft\/shares-outstanding'>Shares Outstanding<\/a>","popup_icon":"<div class='ajax-chart' data-tipped-options=\"ajax: {data: { t: 'MSFT', s: 'shares-outstanding', freq: 'A', statement: 'income-statement' }}\"><i style='font-size:18px; color:#337ab7;' class='fas fa-chart-bar'><\/i><\/span><\/div>","2019-06-30":"7753.00000","2018-06-30":"7794.00000","2017-06-30":"7832.00000","2016-06-30":"8013.00000","2015-06-30":"8254.00000","2014-06-30":"8399.00000","2013-06-30":"8470.00000","2012-06-30":"8506.00000","2011-06-30":"8593.00000","2010-06-30":"8927.00000","2009-06-30":"8996.00000","2008-06-30":"9470.00000","2007-06-30":"9886.00000","2006-06-30":"10531.00000","2005-06-30":"10906.00000"},{"field_name":"<a href='\/stocks\/charts\/MSFT\/microsoft\/eps-basic-net-earnings-per-share'>Basic EPS<\/a>","popup_icon":"<div class='ajax-chart' data-tipped-options=\"ajax: {data: { t: 'MSFT', s: 'eps-basic-net-earnings-per-share', freq: 'A', statement: 'income-statement' }}\"><i style='font-size:18px; color:#337ab7;' class='fas fa-chart-bar'><\/i><\/span><\/div>","2019-06-30":"5.11000","2018-06-30":"2.15000","2017-06-30":"3.29000","2016-06-30":"2.59000","2015-06-30":"1.49000","2014-06-30":"2.66000","2013-06-30":"2.61000","2012-06-30":"2.02000","2011-06-30":"2.73000","2010-06-30":"2.13000","2009-06-30":"1.63000","2008-06-30":"1.90000","2007-06-30":"1.44000","2006-06-30":"1.21000","2005-06-30":"1.13000"},{"field_name":"<a href='\/stocks\/charts\/MSFT\/microsoft\/eps-earnings-per-share-diluted'>EPS - Earnings Per Share<\/a>","popup_icon":"<div class='ajax-chart' data-tipped-options=\"ajax: {data: { t: 'MSFT', s: 'eps-earnings-per-share-diluted', freq: 'A', statement: 'income-statement' }}\"><i style='font-size:18px; color:#337ab7;' class='fas fa-chart-bar'><\/i><\/span><\/div>","2019-06-30":"5.06000","2018-06-30":"2.13000","2017-06-30":"3.25000","2016-06-30":"2.56000","2015-06-30":"1.48000","2014-06-30":"2.63000","2013-06-30":"2.58000","2012-06-30":"2.00000","2011-06-30":"2.69000","2010-06-30":"2.10000","2009-06-30":"1.62000","2008-06-30":"1.87000","2007-06-30":"1.42000","2006-06-30":"1.20000","2005-06-30":"1.12000"}];