Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 Unable to access javaScript generated data with selenium and headless FireFox.
#1
There is a table that is being rendered by javaScript on the page. I would like to retrieve that data in my python code. Below I have include
1. Python Code
2. Relavent HTML
3. The things that I have tried.

1. Python Code
from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options
opts = Options()
opts.set_headless()
assert opts.headless  # Operating in headless mode
browser = Firefox(options=opts)

url="https://www.sec.gov/cgi-bin/viewer?action=view&cik=1596532&accession_number=0001596532-19-000027&xbrl_type=v#"
browser.get(url)

## first click 
link1=browser.find_elements_by_link_text("Financial Statements")
link1[0].click()
## See below for what I have tried here

## second click 
link2=browser.find_elements_by_link_text("Consolidated Balance Sheets")
link2[0].click()

## See below for what I have tried here

browser.save_screenshot("file.png")
## file.png had the correct data so I know that the clicks are working correctly
## 
browser.close()
quit()

2. Relavent HTML code
# This is the HTML from the initial URL

!-- cover_category = 0 -->    <li class="accordion">
      <a id="menu_cat1" href="#">Cover</a>
      <ul>
            <li class="accordion" id="r1" ><a class="xbrlviewer" onClick="javascript:highlight(\
this);" href="javascript:loadReport(1);">Document and Entity Information</a></li>
      </ul>
    </li>
    <li class="accordion">
      <a id="menu_cat2" href="#">Financial Statements</a>
      <ul>
            <li class="accordion" id="r2" ><a class="xbrlviewer" onClick="javascript:highlight(\
this);" href="javascript:loadReport(2);">Consolidated Balance Sheets</a></li>
            <li class="accordion" id="r3" ><a class="xbrlviewer" onClick="javascript:highlight(\
this);" href="javascript:loadReport(3);">Consolidated Balance Sheets (Parenthetical)</a></li>
            <li class="accordion" id="r4" ><a class="xbrlviewer" onClick="javascript:highlight(\
this);" href="javascript:loadReport(4);">Consolidated Statements of Income</a></li>
            <li class="accordion" id="r5" ><a class="xbrlviewer" onClick="javascript:highlight(\
this);" href="javascript:loadReport(5);">Consolidated Statements of Comprehensive Income</a></l\
i>
            <li class="accordion" id="r6" ><a class="xbrlviewer" onClick="javascript:highlight(\
this);" href="javascript:loadReport(6);">Consolidated Statements of Stockholders' Equity</a></l\
i>
            <li class="accordion" id="r7" ><a class="xbrlviewer" onClick="javascript:highlight(\
this);" href="javascript:loadReport(7);">Consolidated Statements of Cash Flows</a></li>
      </ul>
    </li>
3. what I have tried both after click1 and click2

data = browser.execute_script("javascript:loadReport(2)" )
result None

data = browser.execute_script("loadReport(2)" )
returns:
<selenium.webdriver.firefox.webelement.FirefoxWebElement (session="70cabf50-eee7-486f-829c-1d4e9b0e0dfe", element="b3a93cc8-6b8f-431c-a117-3fcb97456631")>

data= browser.find_element_by_link_text("Consolidated Balance Sheets")
returns None

I guess I should have played a little more before asking for help
after the second click
data=browser.page_source does the trick
Larz60+ wrote Aug-04-2019, 12:57 PM:
For future posts, even though this one resolved:

Please post all code, output and errors (in it's entirety) between their respective tags. I did it for you this time, Here are instructions on how to do it yourself next time.
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  Selenium get data from newly accessed page hoff1022 2 199 Oct-09-2019, 06:52 PM
Last Post: hoff1022
  How to Caputre Data After Selenium Scroll ahmedwaqas92 3 440 Aug-18-2019, 12:43 PM
Last Post: ahmedwaqas92
  How can get url from JavaScript in Selenium (Python 3)? m0ntecr1st0 3 721 Feb-19-2019, 12:35 AM
Last Post: m0ntecr1st0
  Selenium Parsing (unable to Parse page after loading) oneclick 6 865 Jan-13-2019, 03:10 AM
Last Post: oneclick
  Firefox Selenium (open new tab) oneclick 1 1,550 Dec-29-2018, 06:59 AM
Last Post: hbknjr
  Headless browser Truman 9 1,819 Oct-16-2018, 10:35 PM
Last Post: Truman
  selenium not running firefox Sanlus 5 1,130 Aug-31-2018, 10:37 PM
Last Post: snippsat
  Proxy Variable in Selenium wont work with FireFox Profile Proxy Setting. MIPython 0 4,674 Jul-13-2018, 05:43 PM
Last Post: MIPython
  Django - Passing data from view for use client side with JavaScript in a template bountyhuntr 0 1,155 Jun-11-2018, 06:04 AM
Last Post: bountyhuntr
  scraping javascript websites with selenium DoctorEvil 1 1,047 Jun-08-2018, 06:40 PM
Last Post: DoctorEvil

Forum Jump:


Users browsing this thread: 1 Guest(s)