Python Forum
seleniem PhantomJS stops working, while Chome still working
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
seleniem PhantomJS stops working, while Chome still working
#1
I have had a program running on a loop executing a program every so often. So i dont check it often. It was using selenium and phantomjs to do headless browser activities. Today i noticed the activity stopped. I initially assumed they changed their website HTML and broke the program. However, as i was troubleshooting the program noticed that Chrome (without any change) works as expected, while phantomjs doesnt do the require activity anymore.

Is there a reason that phantomjs would just stop all of a sudden? You cant really diagnose the issue as you cant view what is going on like you can with Chrome/Firefox mimic'ing activities. But shouldnt they be the same in PhantomJS?
Recommended Tutorials:
Reply
#2
Not sure how PhantomJS introduses itself to the servers but try to change the User-agent.
"As they say in Mexico 'dosvidaniya'. That makes two vidaniyas."
https://freedns.afraid.org
Reply
#3
Well PhantomJS successfully logs into the accounts, and does move around the site correctly. It just doesnt click a javascript button like it use to do. While the Chrome one still does. ITs the same code, it just switches between whether it does it with PhantomJS or Chrome webdrivers.
Recommended Tutorials:
Reply
#4
(Oct-10-2017, 04:47 PM)metulburr Wrote: It just doesnt click a javascript button like it use to do. While the Chrome one still does.
For this has to looker closer at source,and find what execute the click() as PhantomJs dos not have a .click() method. 
Example of what i have used before.
driver.execute_script("$('#calendarbody > div:nth-child(12) > div.col-sm-2.calendar-home-weekly > button').click()")
Reply
#5
yeah i use execute script

def clip_all_btns(browser, number_of_coupons, username):
    '''
    clip ALL available coupons
    '''
    browser.execute_script("document.getElementsByClassName('badge')[0].click()")
    time.sleep(2 * MULT) #allow time to load javascript of checkbox (first click only)
    browser.execute_script("document.getElementsByClassName('close-btn')[0].click()")
    for i in range(number_of_coupons): #now we are verified, clip all coupons
        browser.execute_script("document.getElementsByClassName('badge')[{}].click()".format(i))
    print_color('clipped all coupons on {}'.format(username), GREEN)
The first time you click on a button, a sub-window shows up that doesnt permit you to click on the others until you close that sub-window. So they are handling a general first click and close of that sub-window to bypass the anti-bot measures, and then clicking the rest...in which it doesnt do....in phantomjs.
Recommended Tutorials:
Reply
#6
Ive tried getting ghostdriver logs, tried setting the window min-size on it in case the buttons were hidden for some reason, nothing seems to work in headless. Again no, error, it just doesnt click the button. Im not sure how you are suppose to debug phantomjs?

I found a form of headless option for chrome/firefox though if using python + ubuntu (or debian based such as mint, etc.). Seems to work quite well and i like that i can troubleshoot in Chrome and then just set it headless and know that it is going to get the same content as when i was troubleshooting it.
Quote:PyVirtualDisplay needs Xvfb as a dependency. On Ubuntu, first install Xvfb:

sudo apt-get install xvfb
then install PyVirtualDisplay from Pypi:

pip install pyvirtualdisplay
Sample Selenium script in Python in a headless mode with PyVirtualDisplay:
#!/usr/bin/env python

from pyvirtualdisplay import Display
from selenium import webdriver

display = Display(visible=0, size=(800, 600))
display.start()

# now Firefox will run in a virtual display. 
# you will not see the browser.
browser = webdriver.Firefox()
browser.get('http://www.google.com')
print browser.title
browser.quit()

display.stop()

whereas if i try the headless argument alternative for chrome
    options = webdriver.ChromeOptions()
    options.add_argument('headless')
    browser = webdriver.Chrome(chromedriver, chrome_options=options)
i get an error
Error:
Message: unknown error: cannot get automation extension from unknown error: page could not be found: chrome-extension://aapnijgdinlhnhlmodcfapnahmbfebeb/_generated_background_page.html (Session info: headless chrome=58.0.3029.110) (Driver info: chromedriver=2.29.461571 (8a88bbe0775e2a23afda0ceaf2ef7ee74e822cc5),platform=Linux 4.4.0-96-generic x86_64)
Recommended Tutorials:
Reply
#7
ive noticed a significant CPU usage, as well as having to restart my desktop more often due to being laggy since using the virtual display method. So i am all ears.
Recommended Tutorials:
Reply
#8
Ive wanted to update this thread. It seems since chrome/firefox has added the headless option that PhantomJS seems to be weeded out? As well as the headless option for chrome now makes this issue moot.
Recommended Tutorials:
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
Exclamation Debian 10 Buster Environment - Python 3.x (MariaDB 10.4.21) | Working Connector? BrandonKastning 9 4,122 Jan-04-2022, 08:27 PM
Last Post: BrandonKastning
  <title> django page title dynamic and other field (not working) lemonred 1 2,073 Nov-04-2021, 08:50 PM
Last Post: lemonred
  AttributeError: 'NoneType' object in a parser - stops it apollo 4 3,964 May-28-2021, 02:13 PM
Last Post: Daring_T
  BeautifulSoup: 6k records - but stops after parsing 20 lines apollo 0 1,788 May-10-2021, 05:08 PM
Last Post: apollo
  I am trying to space the registration forms using css. But it is not working. tree 0 1,380 Apr-11-2021, 03:31 AM
Last Post: tree
  Python Stored Procedure call is not working vj78 5 2,882 Apr-04-2021, 03:17 PM
Last Post: vj78
  Parsing html page and working with checkbox (on a captcha) straannick 17 11,045 Feb-04-2021, 02:54 PM
Last Post: snippsat
Question Flask, send_from_directory not working : solved SpongeB0B 2 7,346 Jan-26-2021, 07:02 AM
Last Post: SpongeB0B
  Image Scraper (beautifulsoup), stopped working, need to help see why woodmister 9 3,961 Jan-12-2021, 04:10 PM
Last Post: woodmister
  webbrowser not working in screen management in kivy thousif 1 1,856 Nov-25-2020, 04:02 PM
Last Post: Axel_Erfurt

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020