Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
selneium JS bypass
#11
ummm.... the main benefit of scraping the code is to actually click the button to obtain the coupon for the account. Not to scrape it to get a database of it. So you dont own the coupon until it is actually clicked. Because its a javascript button it needs selenium. It also requires a scroll down to load the rest of the page's coupons.
Recommended Tutorials:
Reply
#12
so as a side note: Ive tried hacking this with just killing the load after its loaded the page. It seems to be consistent as i need to kill it once for login and kill it 3 times for clipping coupons.

ive tried this
webdriver.ActionChains(browser).send_keys(Keys.ESCAPE).perform()
but it doesnt seem to stop the loading. However manually hitting escape does work to stop it loading. But the code i believe should replicate that.

ive also tried:
    try:
        browser.set_page_load_timeout(10)
        browser.get("https://dg.coupons.com/signin/") 
    except Exception:
        print('time out')
        webdriver.ActionChains(browser).send_keys(Keys.ESCAPE).perform()
        webdriver.ActionChains(browser).send_keys(Keys.CONTROL +'Escape')
but that doesnt seem to work either.

Ive also tried:
driver.execute_script("window.stop();")
but nothing seems to happen.

How can you replicate hitting escape to stop it from loading?
Recommended Tutorials:
Reply
#13
ok i think i found a solution. Although i dont understand it, it does work. And its a stable solution compared to mashing the escape key every time.

I get ir from the first answer
https://stackoverflow.com/questions/4450...ets-loaded

and the part that fixed it is
capa = DesiredCapabilities.CHROME
capa["pageLoadStrategy"] = "none"

driver = webdriver.Chrome(desired_capabilities=capa)
for some reason i dont even have to call window.stop(). The above code added fixes it.

Quote: From the Webdriver specs:

For commands that cause a new document to load, the point at which the command returns is determined by the session’s page loading strategy.

When Page Loading takes too much time and you need to stop downloading additional subresources (images, css, js etc) you can change the pageLoadStrategy through the webdriver.

As of this writing, pageLoadStrategy supports the following values :

normal

This stategy causes Selenium to wait for the full page loading (html content and subresources downloaded and parsed).

eager

This stategy causes Selenium to wait for the DOMContentLoaded event (html content downloaded and parsed only).

none

This strategy causes Selenium to return immediately after the initial page content is fully received (html content downloaded).

By default, when Selenium loads a page, it follows the normal pageLoadStrategy.

Im not sure what this even does or more importantly what future problems this fix will induce, but the script is now working as intended again.
Recommended Tutorials:
Reply
#14
Apparently, that sets the browser's definition of a complete page load. "None" indicates that the browser will stop loading additional resources after it has the initial results. As I understand it, that means no pictures, video, or sound because those require additional HTTP requests to retrieve.
Reply
#15
it definitely still has pictures though. Not sure about video/audio. I was thinking it stopped additional javascript/AJAX. I thought they added something to break bots lol.
Recommended Tutorials:
Reply
#16
Quote:So you dont own the coupon until it is actually clicked. Because its a javascript button it needs selenium. It also requires a scroll down to load the rest of the page's coupons
For some reason, even though I knew this, it didn't register at first!

This is a good one to know about. Glad you found the answer.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Selenium - bypass Cloudflare bot detection klaarnou 4 16,108 Jul-29-2023, 01:42 AM
Last Post: scrapingbypass
  How to check HTTP error 500 and bypass SriMekala 3 10,379 May-04-2019, 02:07 PM
Last Post: snippsat
  selenium bypass javascript popup box metulburr 6 8,309 Jun-02-2017, 07:15 PM
Last Post: snippsat

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020