Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
scraping with multiple iframe
#1
HI All

Im a php developer and switching to python,

Basically I was able to get some scraping from one website with basic login and session using beautifulsoup

now, here comes the burden the other website is using multiple iframe and when I view the iframe . there is no data showing, I assume that the data is coming from parent window , the site has a lot of javascript as well

to make is simpler

--------------------------------------------------------------------------------------------------------
iframe 1 ( profile data), when open this iframe to other tab. it redirect to login
--------------------------------------------------------------------------------------------------------

iframe2 ( form) data is triggered when frame3 data is click
----------------------------------------------------------
iframe3 (data list). when data is click, it will trigger form (iframe2) with some data




==========================================
issue 1.

when iframe link is open to other tab. it redirect to login

isue 2.

when iframe3 link is open to other tab, no data list


at least can you give me some idea or some code snippet or anyone encountered this kind of problem and how it solved?



Im not good in english so if some is not clear, please let me know so i can explain as much as possible
Reply
#2
Selenium can be used for this.
Example:
browser = webdriver.Chrome()
# Give some time to load
time.sleep(3)
# Switch to iframe
browser.switch_to.frame(driver.find_element_by_tag_name("iframe"))
# Do some stuff with iframe
elem = driver.find_element_by_xpath("body/p/strong")
elem.send_keys("Lorem Ipsum")
# Back to default content,also out of iframe
driver.switch_to.default_content()
There are also Waits.
wait = WebDriverWait(driver, 300)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.NAME,'iframe_name')))
More info here.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
Information Web-scraping, multiple webpages Pabloty92 1 1,246 Dec-28-2022, 02:09 PM
Last Post: Yoriz
  scraping multiple pages from table bandar 1 2,651 Jun-27-2020, 10:43 PM
Last Post: Larz60+
  Scraping Multiple Pages mbadatanut 1 4,178 May-08-2020, 02:30 AM
Last Post: Larz60+
  selenium click in iframe fails 3Pinter 6 5,025 Apr-29-2020, 12:59 PM
Last Post: Larz60+
  Scraping from multiple URLS to print in a single line. jb89 4 3,306 Jan-29-2020, 06:12 AM
Last Post: perfringo
  How to find which frame/iframe my element belongs to? smaria 1 2,034 Nov-18-2019, 09:21 PM
Last Post: Larz60+
  MaxRetryError while scraping a website multiple times kawasso 6 17,267 Aug-29-2019, 05:25 PM
Last Post: kawasso
  Looping actions in an iframe using Selenium on Python amyd 3 2,967 Mar-06-2019, 08:31 PM
Last Post: Larz60+
  scraping multiple pages of a website. Blue Dog 14 22,273 Jun-21-2018, 09:03 PM
Last Post: Blue Dog
  Unable to select iframe using selenium endejoli 1 2,941 Jan-11-2018, 05:13 PM
Last Post: j.crater

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020