Python Forum

Full Version: scraping with multiple iframe
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
HI All

Im a php developer and switching to python,

Basically I was able to get some scraping from one website with basic login and session using beautifulsoup

now, here comes the burden the other website is using multiple iframe and when I view the iframe . there is no data showing, I assume that the data is coming from parent window , the site has a lot of javascript as well

to make is simpler

--------------------------------------------------------------------------------------------------------
iframe 1 ( profile data), when open this iframe to other tab. it redirect to login
--------------------------------------------------------------------------------------------------------

iframe2 ( form) data is triggered when frame3 data is click
----------------------------------------------------------
iframe3 (data list). when data is click, it will trigger form (iframe2) with some data




==========================================
issue 1.

when iframe link is open to other tab. it redirect to login

isue 2.

when iframe3 link is open to other tab, no data list


at least can you give me some idea or some code snippet or anyone encountered this kind of problem and how it solved?



Im not good in english so if some is not clear, please let me know so i can explain as much as possible
Selenium can be used for this.
Example:
browser = webdriver.Chrome()
# Give some time to load
time.sleep(3)
# Switch to iframe
browser.switch_to.frame(driver.find_element_by_tag_name("iframe"))
# Do some stuff with iframe
elem = driver.find_element_by_xpath("body/p/strong")
elem.send_keys("Lorem Ipsum")
# Back to default content,also out of iframe
driver.switch_to.default_content()
There are also Waits.
wait = WebDriverWait(driver, 300)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.NAME,'iframe_name')))
More info here.