Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 Unable to access javaScript generated data with selenium and headless FireFox.
#1
There is a table that is being rendered by javaScript on the page. I would like to retrieve that data in my python code. Below I have include
1. Python Code
2. Relavent HTML
3. The things that I have tried.

1. Python Code
from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options
opts = Options()
opts.set_headless()
assert opts.headless  # Operating in headless mode
browser = Firefox(options=opts)

url="https://www.sec.gov/cgi-bin/viewer?action=view&cik=1596532&accession_number=0001596532-19-000027&xbrl_type=v#"
browser.get(url)

## first click 
link1=browser.find_elements_by_link_text("Financial Statements")
link1[0].click()
## See below for what I have tried here

## second click 
link2=browser.find_elements_by_link_text("Consolidated Balance Sheets")
link2[0].click()

## See below for what I have tried here

browser.save_screenshot("file.png")
## file.png had the correct data so I know that the clicks are working correctly
## 
browser.close()
quit()

2. Relavent HTML code
# This is the HTML from the initial URL

!-- cover_category = 0 -->    <li class="accordion">
      <a id="menu_cat1" href="#">Cover</a>
      <ul>
            <li class="accordion" id="r1" ><a class="xbrlviewer" onClick="javascript:highlight(\
this);" href="javascript:loadReport(1);">Document and Entity Information</a></li>
      </ul>
    </li>
    <li class="accordion">
      <a id="menu_cat2" href="#">Financial Statements</a>
      <ul>
            <li class="accordion" id="r2" ><a class="xbrlviewer" onClick="javascript:highlight(\
this);" href="javascript:loadReport(2);">Consolidated Balance Sheets</a></li>
            <li class="accordion" id="r3" ><a class="xbrlviewer" onClick="javascript:highlight(\
this);" href="javascript:loadReport(3);">Consolidated Balance Sheets (Parenthetical)</a></li>
            <li class="accordion" id="r4" ><a class="xbrlviewer" onClick="javascript:highlight(\
this);" href="javascript:loadReport(4);">Consolidated Statements of Income</a></li>
            <li class="accordion" id="r5" ><a class="xbrlviewer" onClick="javascript:highlight(\
this);" href="javascript:loadReport(5);">Consolidated Statements of Comprehensive Income</a></l\
i>
            <li class="accordion" id="r6" ><a class="xbrlviewer" onClick="javascript:highlight(\
this);" href="javascript:loadReport(6);">Consolidated Statements of Stockholders' Equity</a></l\
i>
            <li class="accordion" id="r7" ><a class="xbrlviewer" onClick="javascript:highlight(\
this);" href="javascript:loadReport(7);">Consolidated Statements of Cash Flows</a></li>
      </ul>
    </li>
3. what I have tried both after click1 and click2

data = browser.execute_script("javascript:loadReport(2)" )
result None

data = browser.execute_script("loadReport(2)" )
returns:
<selenium.webdriver.firefox.webelement.FirefoxWebElement (session="70cabf50-eee7-486f-829c-1d4e9b0e0dfe", element="b3a93cc8-6b8f-431c-a117-3fcb97456631")>

data= browser.find_element_by_link_text("Consolidated Balance Sheets")
returns None

I guess I should have played a little more before asking for help
after the second click
data=browser.page_source does the trick
Larz60+ wrote Aug-04-2019, 12:57 PM:
For future posts, even though this one resolved:

Please post all code, output and errors (in it's entirety) between their respective tags. I did it for you this time, Here are instructions on how to do it yourself next time.
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  Extract data with Selenium and BeautifulSoup nestor 3 368 Jun-06-2020, 01:34 AM
Last Post: Larz60+
  Unable to get the data from web API using authentication key lokamaba 0 298 May-15-2020, 05:07 AM
Last Post: lokamaba
  question about using javascript on python selenium Kai 1 264 Apr-12-2020, 04:28 AM
Last Post: Larz60+
  Clicking on element not triggering event in Selenium Python (Event Key is not in data dkaeloredo 2 542 Feb-16-2020, 05:50 AM
Last Post: dkaeloredo
  Headless Chrome: How to login into a webpage? amandacstr 1 390 Feb-06-2020, 02:19 AM
Last Post: metulburr
  error when running headless selenium julio2000 2 370 Feb-01-2020, 12:41 PM
Last Post: julio2000
  Scrapping javascript website with Selenium where pages randomly fail to load JuanJuan 14 1,247 Dec-27-2019, 12:32 PM
Last Post: JuanJuan
  Selenium get data from newly accessed page hoff1022 2 538 Oct-09-2019, 06:52 PM
Last Post: hoff1022
  How to Caputre Data After Selenium Scroll ahmedwaqas92 3 1,907 Aug-18-2019, 12:43 PM
Last Post: ahmedwaqas92
  How can get url from JavaScript in Selenium (Python 3)? m0ntecr1st0 3 1,151 Feb-19-2019, 12:35 AM
Last Post: m0ntecr1st0

Forum Jump:


Users browsing this thread: 1 Guest(s)