I have had a program running on a loop executing a program every so often. So i dont check it often. It was using selenium and phantomjs to do headless browser activities. Today i noticed the activity stopped. I initially assumed they changed their website HTML and broke the program. However, as i was troubleshooting the program noticed that Chrome (without any change) works as expected, while phantomjs doesnt do the require activity anymore.
Is there a reason that phantomjs would just stop all of a sudden? You cant really diagnose the issue as you cant view what is going on like you can with Chrome/Firefox mimic'ing activities. But shouldnt they be the same in PhantomJS?
Not sure how PhantomJS introduses itself to the servers but try to change the User-agent.
Well PhantomJS successfully logs into the accounts, and does move around the site correctly. It just doesnt click a javascript button like it use to do. While the Chrome one still does. ITs the same code, it just switches between whether it does it with PhantomJS or Chrome webdrivers.
(Oct-10-2017, 04:47 PM)metulburr Wrote: [ -> ]It just doesnt click a javascript button like it use to do. While the Chrome one still does.
For this has to looker closer at source,and find what execute the click() as PhantomJs dos not have a
.click() method.
Example of what i have used before.
driver.execute_script("$('#calendarbody > div:nth-child(12) > div.col-sm-2.calendar-home-weekly > button').click()")
yeah i use execute script
def clip_all_btns(browser, number_of_coupons, username):
'''
clip ALL available coupons
'''
browser.execute_script("document.getElementsByClassName('badge')[0].click()")
time.sleep(2 * MULT) #allow time to load javascript of checkbox (first click only)
browser.execute_script("document.getElementsByClassName('close-btn')[0].click()")
for i in range(number_of_coupons): #now we are verified, clip all coupons
browser.execute_script("document.getElementsByClassName('badge')[{}].click()".format(i))
print_color('clipped all coupons on {}'.format(username), GREEN)The first time you click on a button, a sub-window shows up that doesnt permit you to click on the others until you close that sub-window. So they are handling a general first click and close of that sub-window to bypass the anti-bot measures, and then clicking the rest...in which it doesnt do....in phantomjs.
Ive tried getting ghostdriver logs, tried setting the window min-size on it in case the buttons were hidden for some reason, nothing seems to work in headless. Again no, error, it just doesnt click the button. Im not sure how you are suppose to debug phantomjs?
I found a form of headless option for chrome/firefox though if using python + ubuntu (or debian based such as mint, etc.). Seems to work quite well and i like that i can troubleshoot in Chrome and then just set it headless and know that it is going to get the same content as when i was troubleshooting it.
Quote:PyVirtualDisplay needs Xvfb as a dependency. On Ubuntu, first install Xvfb:
sudo apt-get install xvfb
then install PyVirtualDisplay from Pypi:
pip install pyvirtualdisplay
Sample Selenium script in Python in a headless mode with PyVirtualDisplay:
#!/usr/bin/env python
from pyvirtualdisplay import Display
from selenium import webdriver
display = Display(visible=0, size=(800, 600))
display.start()
# now Firefox will run in a virtual display.
# you will not see the browser.
browser = webdriver.Firefox()
browser.get('http://www.google.com')
print browser.title
browser.quit()
display.stop()
whereas if i try the headless argument alternative for chrome
options = webdriver.ChromeOptions()
options.add_argument('headless')
browser = webdriver.Chrome(chromedriver, chrome_options=options)i get an error
Error:
Message: unknown error: cannot get automation extension
from unknown error: page could not be found: chrome-extension://aapnijgdinlhnhlmodcfapnahmbfebeb/_generated_background_page.html
(Session info: headless chrome=58.0.3029.110)
(Driver info: chromedriver=2.29.461571 (8a88bbe0775e2a23afda0ceaf2ef7ee74e822cc5),platform=Linux 4.4.0-96-generic x86_64)
ive noticed a significant CPU usage, as well as having to restart my desktop more often due to being laggy since using the virtual display method. So i am all ears.
Ive wanted to update this thread. It seems since chrome/firefox has added the headless option that PhantomJS seems to be weeded out? As well as the headless option for chrome now makes this issue moot.