Python Forum
Unable to access javaScript generated data with selenium and headless FireFox.
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Unable to access javaScript generated data with selenium and headless FireFox.
#1
There is a table that is being rendered by javaScript on the page. I would like to retrieve that data in my python code. Below I have include
1. Python Code
2. Relavent HTML
3. The things that I have tried.

1. Python Code
from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options
opts = Options()
opts.set_headless()
assert opts.headless  # Operating in headless mode
browser = Firefox(options=opts)

url="https://www.sec.gov/cgi-bin/viewer?action=view&cik=1596532&accession_number=0001596532-19-000027&xbrl_type=v#"
browser.get(url)

## first click 
link1=browser.find_elements_by_link_text("Financial Statements")
link1[0].click()
## See below for what I have tried here

## second click 
link2=browser.find_elements_by_link_text("Consolidated Balance Sheets")
link2[0].click()

## See below for what I have tried here

browser.save_screenshot("file.png")
## file.png had the correct data so I know that the clicks are working correctly
## 
browser.close()
quit()

2. Relavent HTML code
# This is the HTML from the initial URL

!-- cover_category = 0 -->    <li class="accordion">
      <a id="menu_cat1" href="#">Cover</a>
      <ul>
            <li class="accordion" id="r1" ><a class="xbrlviewer" onClick="javascript:highlight(\
this);" href="javascript:loadReport(1);">Document and Entity Information</a></li>
      </ul>
    </li>
    <li class="accordion">
      <a id="menu_cat2" href="#">Financial Statements</a>
      <ul>
            <li class="accordion" id="r2" ><a class="xbrlviewer" onClick="javascript:highlight(\
this);" href="javascript:loadReport(2);">Consolidated Balance Sheets</a></li>
            <li class="accordion" id="r3" ><a class="xbrlviewer" onClick="javascript:highlight(\
this);" href="javascript:loadReport(3);">Consolidated Balance Sheets (Parenthetical)</a></li>
            <li class="accordion" id="r4" ><a class="xbrlviewer" onClick="javascript:highlight(\
this);" href="javascript:loadReport(4);">Consolidated Statements of Income</a></li>
            <li class="accordion" id="r5" ><a class="xbrlviewer" onClick="javascript:highlight(\
this);" href="javascript:loadReport(5);">Consolidated Statements of Comprehensive Income</a></l\
i>
            <li class="accordion" id="r6" ><a class="xbrlviewer" onClick="javascript:highlight(\
this);" href="javascript:loadReport(6);">Consolidated Statements of Stockholders' Equity</a></l\
i>
            <li class="accordion" id="r7" ><a class="xbrlviewer" onClick="javascript:highlight(\
this);" href="javascript:loadReport(7);">Consolidated Statements of Cash Flows</a></li>
      </ul>
    </li>
3. what I have tried both after click1 and click2

data = browser.execute_script("javascript:loadReport(2)" )
result None

data = browser.execute_script("loadReport(2)" )
returns:
<selenium.webdriver.firefox.webelement.FirefoxWebElement (session="70cabf50-eee7-486f-829c-1d4e9b0e0dfe", element="b3a93cc8-6b8f-431c-a117-3fcb97456631")>

data= browser.find_element_by_link_text("Consolidated Balance Sheets")
returns None

I guess I should have played a little more before asking for help
after the second click
data=browser.page_source does the trick
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Connect to existing Firefox session with Selenium euras 0 1,316 Feb-11-2021, 02:54 PM
Last Post: euras
  Using Python request without selenium on html form with javascript onclick submit but eraosa 0 1,237 Jan-09-2021, 06:08 PM
Last Post: eraosa
  Selenium Parsing (unable to Parse page after loading) oneclick 7 3,268 Oct-30-2020, 08:13 PM
Last Post: tomalex
  Extract data with Selenium and BeautifulSoup nestor 3 1,728 Jun-06-2020, 01:34 AM
Last Post: Larz60+
  Unable to get the data from web API using authentication key lokamaba 0 932 May-15-2020, 05:07 AM
Last Post: lokamaba
  question about using javascript on python selenium Kai 1 925 Apr-12-2020, 04:28 AM
Last Post: Larz60+
  Clicking on element not triggering event in Selenium Python (Event Key is not in data dkaeloredo 2 2,150 Feb-16-2020, 05:50 AM
Last Post: dkaeloredo
  Headless Chrome: How to login into a webpage? amandacstr 1 1,770 Feb-06-2020, 02:19 AM
Last Post: metulburr
  error when running headless selenium julio2000 2 1,717 Feb-01-2020, 12:41 PM
Last Post: julio2000
  Scrapping javascript website with Selenium where pages randomly fail to load JuanJuan 14 3,412 Dec-27-2019, 12:32 PM
Last Post: JuanJuan

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020