Python Forum
Selenium get data from newly accessed page
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Selenium get data from newly accessed page
#1
Don't know how to get data from a newly loaded/clicked page using Selenium.
I have 2 questions:

1) I load a page with Selenium (asp) enter some value and a new page loads. On this new page I want to access this data: <td class="queryfield" valign="top">05/09/2019&nbsp;</td>. How do I do that? How do I make selenium grab data from the 'newly' loaded page and not the original page and how do I access this item specifically - looking to return the 05/09/2019 data.

2) How do I make Selenium wait until the page loads? I currently use sleep(.5) - see my code below. Thanks.


[inline]
FMCSA_SNAPSHOT_WEBSITE = "https://safer.fmcsa.dot.gov/CompanySnapshot.aspx"
driver = webdriver.Firefox()
driver.get(FMCSA_SNAPSHOT_WEBSITE)
	sleep(.5)
	input_dot = driver.find_element_by_id('4')
	sleep(.5)
	input_dot.send_keys(dot) #use 348313 for dot value as test case
	input_dot.send_keys(Keys.ENTER)
    # how do I load data from this new page?
[/inline]
Reply
#2
check the proper waiting methods instead of using time sleep. A 1/2 of a second might not be long enough to obtain the page. It all depends on the internet connection and speed, and computer. That is why using time.sleep is not a good option.

This loads the pages from thereafter execution of whatever
driver.get(FMCSA_SNAPSHOT_WEBSITE)
For example, if hitting search and loads a new page, driver should contain that information when you get the HTML driver.page_source. You just have to make sure to properly wait for it. On occasions this may not work as sometimes the new windows is in a different frame or new tab in which you would have to switch to it to get the content.

Another alternative is to use the embedded API by inputting the content in directly in the URL, such as searching for Jim and selecting name would provide the URL of
https://safer.fmcsa.dot.gov/keywordx.asp?searchstring=%2AJIM%2A&SEARCHTYPE=
and it immediately goes to what i am assuming is the page you are referring to.
Recommended Tutorials:
Reply
#3
@metulburr


1)
https://safer.fmcsa.dot.gov/keywordx.asp...SEARCHTYPE=

this query only searches by the "name" = "JIM"
what if I want to use the search by "USDOT Number" not name?

2) also when I do driver.page_source I get data on the original page "https://safer.fmcsa.dot.gov/CompanySnapshot.aspx" not the newly loaded page with the data. How do I get data from the newly loaded page with the info:
Entity Type: CARRIER
Operating Status: ACTIVE Out of Service Date: None
Legal Name: PRINTLINK SHORT RUN BUSINESS FORMS
DBA Name: PRINTLINK PALMER
Physical Address: 309 FRITZ KEIPER BLVD
BATTLE CREEK, MI 49037
Phone: (269) 965-1336
Mailing Address: P O BOX 428
BATTLE CREEK, MI 49016-0428
USDOT Number: 348313 State Carrier ID Number:
MC/MX/FF Number(s): DUNS Number: 72-574-015
Power Units: 1 Drivers: 2
MCS-150 Form Date: 05/09/2019 MCS-150 Mileage (Year): 52,025 (2013)
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Click on a button on web page using Selenium Pavel_47 7 4,567 Jan-05-2023, 04:20 AM
Last Post: ellapurnellrt
  Python Selenium (Dropdown-) data Robin_at_Cantelli 2 6,158 Dec-29-2021, 03:16 AM
Last Post: ondreweil
  trying to save data automatically from this page thunderspeed 1 1,973 Sep-19-2021, 04:57 AM
Last Post: ndc85430
  Scraping a page with log in data (security, proxies) iamaghost 0 2,103 Mar-27-2021, 02:56 PM
Last Post: iamaghost
  Selenium/Helium loads up a blank web page firaki12345 0 2,008 Mar-23-2021, 11:51 AM
Last Post: firaki12345
  Saving html page and reloading into selenium while developing all xpaths Larz60+ 4 4,104 Feb-04-2021, 07:01 AM
Last Post: jonathanwhite1
  Selenium Parsing (unable to Parse page after loading) oneclick 7 5,892 Oct-30-2020, 08:13 PM
Last Post: tomalex
  Selenium Page Object Model with Python Cryptus 5 3,894 Aug-19-2020, 06:30 AM
Last Post: mlieqo
  Selenium on Angular page Martinelli 3 5,598 Jul-28-2020, 12:40 PM
Last Post: Martinelli
  Extract data with Selenium and BeautifulSoup nestor 3 3,818 Jun-06-2020, 01:34 AM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020