Python Forum

Full Version: HOWTO? Login DSL Modem with Python Requests: need Click "Apply" Button
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Esteemed Forum Participants & Lurkers:
NOTE: Inexperienced with Web Development ...

I am trying to make a simple tool to track my Inet usage so that I don't hit my ISP usage cap. My goal is a web scraper to fetch the "Total Usage" value out of the Browser-based control panel for my DSL modem/router (ZyXEL PK5001Z).

I found "Python Requests" and it seemed to be a reasonable tool for my task, and I got started with it. The big problem is that the modem login can NOT be completed with a <Return> character ... there is an "Apply" button that must be "clicked" in order to complete the login. I don't know if the login page even accepts my request into the appropriate text entry boxes, or how to activate the "Apply" button function.
>>> import requests
>>> r = requests.get('<http modem URL>', auth=('<webtest>', '<modem-pswd>'))
>>> r.status_code
>>> r.headers['content-type']
'text/html; charset=iso-8859-1'
>>> r.encoding
>>> r.text
<Entire page code of>: <title>Century Link  Modem Configurator</title>
I actually have already figured out how to fetch the parameter I need ... I was playing around with the Developer pane in Firefox and found that AFTER I get logged in, I only need to access a specific page URL that returns the data that loads the DSL Status page! I can parse that easily. Here is the beginning of the modem login page AFTER I issued the above "request.get":
 . . . . .
<title>Century Link  Modem Configurator</title>
<link href="../_css/basic.css" rel="stylesheet" type="text/css" />\r\n<script language="Javascript"
 . . . . .
 . . . . .
<title>Century Link Modem Configurator</title>
<link href="../_css/index.css" . . . 
For the next step, the modem does NOT accept the password with a <Return> keystroke ... you MUST click the "Apply" button in order to actually log in. I think this is the relevant code on the login page:
<div class="stepWrapper">
  <p><strong> 2. Click &quot;Apply&quot; to log in.</strong></p>
  <a href="#" onclick="ButtonClick(1)" class="btn apply_btn">Apply</a>
Roger Shruber over on the #python chat tried to help and suggested that I might try Selenium, but I couldn't figure out how to get it installed into Linux Mint 17.3.

Thank you for any and all comments, suggestions, and gracious assistance.
Blessings in abundance, all the best, & ENJOY!
Art in Carlisle, PA USA
I think you need a Session. No problem, requests have this already.
First create a session object, then get the login page:

session = requests.Session()
response = session.get('') # in my case
# then you look in response.text for a form, there is an action and it's value is the url based
# on the current path you're on, in my case i used BeatifulSoup to find it.
# but you can use also a webbrowser
# <form action="sendResult.cgi?section=login" id="mainForm" method="post">
# <input name="loginDelay" type="hidden" value="0"/>
# <input name="page" type="hidden" value="login"/>
# <div>Benutzername</div>
# <input name="username" tabindex="1" type="text"/>
# <div>Passwort</div>
# <input name="password" tabindex="2" type="password"/>
# </form>

# so we need something to prepare the data for the form
# all fields should be submitted, also the hidden fields
# with requests you can just use a dict for this task
login = {'loginDelay': '0', 'page': 'login', 'username': 'admin', 'password': 'admin'}

# then you have to use the post method together with the target (action url) and
# login data
response ='', data=login)
if response.status_code == 200:
For further webscraping you should use Beautiful Soup or Scrapy.
You can use also regex, but this is more complicated.

For full help, you've to post the whole html code. Without this we can only guess. I think it's still a form on your side wrapped around with javascript for the fancy stuff.
(Jul-29-2017, 10:29 PM)Webtest Wrote: [ -> ]you MUST click the "Apply" button in order to actually log in.
Is not so straightforward to figure it out.
ButtonClick(1) is a function/method in JavaScript look at what it dos,
at some point i guess a POST request will be made to a HTTP endpoint.
Tool for inspecting Chrome DevTools,Firefox Developer Tools.
This request is what you most send with Requests to stimulate that button click.
Quote:but I couldn't figure out how to get it installed into Linux Mint 17.3.
With Selenium and often in combo with PhantomJS(Selenium can load it).
It easier for there are method click() to click buttons.

Install Selenium it should work with pip
The default python3 in Mint17.3 is python3.4.
So sudo pip3 install selenium for using it for Python 3.
python3 to run Python 3.
Linux Python 3 environment
I have a demo with Selenium here.
Esteemed Forum Participants & Lurkers:
I have been working on an application to login and download "DSL Status" information from my ZyXEL 5100Z DSL modem/router to fetch monthly DSL usage information so that I don't overrun the data cap limit set my my phone co./ISP. I have finally developed a totally software solution to this question. I have comprehensive posts on the Linux Mint forum, and I posted a copy of the successful test code on The answer to the question in the original post is answered there. Here are the links for your reference.

SOLVED: How To? Basic Inet Monthly Usage Monitor
SOLVED: HOWTO? Login DSL Modem with Python Requests: need Click "Apply" Button
SOLVED: HOW TO? Load crontab job from script?
Test Scraper - IT WORKS! <-

Blessings in abundance, all the best, & ENJOY!
Art in Carlisle, PA USA
I was also searching for a solution for my DSL Modem and found it really helpful.
Thanks to @DeaD_EyE for your comment.