Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 Unable to access javaScript generated data with selenium and headless FireFox.
#1
There is a table that is being rendered by javaScript on the page. I would like to retrieve that data in my python code. Below I have include
1. Python Code
2. Relavent HTML
3. The things that I have tried.

1. Python Code
from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options
opts = Options()
opts.set_headless()
assert opts.headless  # Operating in headless mode
browser = Firefox(options=opts)

url="https://www.sec.gov/cgi-bin/viewer?action=view&cik=1596532&accession_number=0001596532-19-000027&xbrl_type=v#"
browser.get(url)

## first click 
link1=browser.find_elements_by_link_text("Financial Statements")
link1[0].click()
## See below for what I have tried here

## second click 
link2=browser.find_elements_by_link_text("Consolidated Balance Sheets")
link2[0].click()

## See below for what I have tried here

browser.save_screenshot("file.png")
## file.png had the correct data so I know that the clicks are working correctly
## 
browser.close()
quit()

2. Relavent HTML code
# This is the HTML from the initial URL

!-- cover_category = 0 -->    <li class="accordion">
      <a id="menu_cat1" href="#">Cover</a>
      <ul>
            <li class="accordion" id="r1" ><a class="xbrlviewer" onClick="javascript:highlight(\
this);" href="javascript:loadReport(1);">Document and Entity Information</a></li>
      </ul>
    </li>
    <li class="accordion">
      <a id="menu_cat2" href="#">Financial Statements</a>
      <ul>
            <li class="accordion" id="r2" ><a class="xbrlviewer" onClick="javascript:highlight(\
this);" href="javascript:loadReport(2);">Consolidated Balance Sheets</a></li>
            <li class="accordion" id="r3" ><a class="xbrlviewer" onClick="javascript:highlight(\
this);" href="javascript:loadReport(3);">Consolidated Balance Sheets (Parenthetical)</a></li>
            <li class="accordion" id="r4" ><a class="xbrlviewer" onClick="javascript:highlight(\
this);" href="javascript:loadReport(4);">Consolidated Statements of Income</a></li>
            <li class="accordion" id="r5" ><a class="xbrlviewer" onClick="javascript:highlight(\
this);" href="javascript:loadReport(5);">Consolidated Statements of Comprehensive Income</a></l\
i>
            <li class="accordion" id="r6" ><a class="xbrlviewer" onClick="javascript:highlight(\
this);" href="javascript:loadReport(6);">Consolidated Statements of Stockholders' Equity</a></l\
i>
            <li class="accordion" id="r7" ><a class="xbrlviewer" onClick="javascript:highlight(\
this);" href="javascript:loadReport(7);">Consolidated Statements of Cash Flows</a></li>
      </ul>
    </li>
3. what I have tried both after click1 and click2

data = browser.execute_script("javascript:loadReport(2)" )
result None

data = browser.execute_script("loadReport(2)" )
returns:
<selenium.webdriver.firefox.webelement.FirefoxWebElement (session="70cabf50-eee7-486f-829c-1d4e9b0e0dfe", element="b3a93cc8-6b8f-431c-a117-3fcb97456631")>

data= browser.find_element_by_link_text("Consolidated Balance Sheets")
returns None

I guess I should have played a little more before asking for help
after the second click
data=browser.page_source does the trick
Larz60+ wrote Aug-04-2019, 12:57 PM:
For future posts, even though this one resolved:

Please post all code, output and errors (in it's entirety) between their respective tags. I did it for you this time, Here are instructions on how to do it yourself next time.
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  How to Caputre Data After Selenium Scroll ahmedwaqas92 3 128 Aug-18-2019, 12:43 PM
Last Post: ahmedwaqas92
  Scape webpage that has text - javascript? lonelygirl 2 132 Aug-08-2019, 12:42 AM
Last Post: lonelygirl
  Access my organization Salesforce and read report DannyPt 2 222 Jun-30-2019, 10:58 PM
Last Post: micseydel
  Access my webpage and download files from Python Pedroski55 7 385 May-26-2019, 12:08 PM
Last Post: snippsat
  Download images generated by user input one_of_us 0 318 Mar-26-2019, 07:58 AM
Last Post: one_of_us
  How can get url from JavaScript in Selenium (Python 3)? m0ntecr1st0 3 533 Feb-19-2019, 12:35 AM
Last Post: m0ntecr1st0
  How to access routes using Bottle Framework nikos 5 438 Feb-13-2019, 11:43 PM
Last Post: nikos
  Selenium Parsing (unable to Parse page after loading) oneclick 6 641 Jan-13-2019, 03:10 AM
Last Post: oneclick
  unable to import pymysql IMuriel 3 665 Jan-08-2019, 08:56 PM
Last Post: IMuriel
  unable to replcae a pattern using method re.sub() Jeevananda 4 1,716 Jan-05-2019, 04:31 PM
Last Post: oneclick

Forum Jump:


Users browsing this thread: 1 Guest(s)