Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 HOWTO? Login DSL Modem with Python Requests: need Click "Apply" Button
#1
Esteemed Forum Participants & Lurkers:
======================================
NOTE: Inexperienced with Web Development ...

I am trying to make a simple tool to track my Inet usage so that I don't hit my ISP usage cap. My goal is a web scraper to fetch the "Total Usage" value out of the Browser-based control panel for my DSL modem/router (ZyXEL PK5001Z).

I found "Python Requests" and it seemed to be a reasonable tool for my task, and I got started with it. The big problem is that the modem login can NOT be completed with a <Return> character ... there is an "Apply" button that must be "clicked" in order to complete the login. I don't know if the login page even accepts my request into the appropriate text entry boxes, or how to activate the "Apply" button function.
>>> import requests
>>> r = requests.get('<http modem URL>', auth=('<webtest>', '<modem-pswd>'))
>>> r.status_code
200
>>> r.headers['content-type']
'text/html; charset=iso-8859-1'
>>> r.encoding
'iso-8859-1'
>>> r.text
<Entire page code of>: <title>Century Link  Modem Configurator</title>
I actually have already figured out how to fetch the parameter I need ... I was playing around with the Developer pane in Firefox and found that AFTER I get logged in, I only need to access a specific page URL that returns the data that loads the DSL Status page! I can parse that easily. Here is the beginning of the modem login page AFTER I issued the above "request.get":
LOGIN PAGE UPLOADED:
 . . . . .
<title>Century Link  Modem Configurator</title>
<link href="../_css/basic.css" rel="stylesheet" type="text/css" />\r\n<script language="Javascript"
 . . . . .
? SHOULD BE WELCOME PAGE ?:
 . . . . .
<title>Century Link Modem Configurator</title>
<link href="../_css/index.css" . . . 
For the next step, the modem does NOT accept the password with a <Return> keystroke ... you MUST click the "Apply" button in order to actually log in. I think this is the relevant code on the login page:
<div class="stepWrapper">
  <p><strong> 2. Click &quot;Apply&quot; to log in.</strong></p>
  <a href="#" onclick="ButtonClick(1)" class="btn apply_btn">Apply</a>
</div>
Roger Shruber over on the #python chat tried to help and suggested that I might try Selenium, but I couldn't figure out how to get it installed into Linux Mint 17.3.

Thank you for any and all comments, suggestions, and gracious assistance.
Blessings in abundance, all the best, & ENJOY!
Art in Carlisle, PA USA
Quote
#2
I think you need a Session. No problem, requests have this already.
First create a session object, then get the login page:

session = requests.Session()
response = session.get('http://192.168.192.1/cgi-bin/status.cgi') # in my case
# then you look in response.text for a form, there is an action and it's value is the url based
# on the current path you're on, in my case i used BeatifulSoup to find it.
# but you can use also a webbrowser
# <form action="sendResult.cgi?section=login" id="mainForm" method="post">
# <input name="loginDelay" type="hidden" value="0"/>
# <input name="page" type="hidden" value="login"/>
# <div>Benutzername</div>
# <input name="username" tabindex="1" type="text"/>
# <div>Passwort</div>
# <input name="password" tabindex="2" type="password"/>
# </form>

# so we need something to prepare the data for the form
# all fields should be submitted, also the hidden fields
# with requests you can just use a dict for this task
login = {'loginDelay': '0', 'page': 'login', 'username': 'admin', 'password': 'admin'}

# then you have to use the post method together with the target (action url) and
# login data
response = session.post('http://192.168.192.1/cgi-bin/sendResult.cgi?section=login', data=login)
if response.status_code == 200:
    print('Success')
    print(response.text)
For further webscraping you should use Beautiful Soup or Scrapy.
You can use also regex, but this is more complicated.

For full help, you've to post the whole html code. Without this we can only guess. I think it's still a form on your side wrapped around with javascript for the fancy stuff.
My code examples are always for Python >=3.6.0
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Quote
#3
(Jul-29-2017, 10:29 PM)Webtest Wrote: you MUST click the "Apply" button in order to actually log in.
Is not so straightforward to figure it out.
onclick="ButtonClick(1)"
ButtonClick(1) is a function/method in JavaScript look at what it dos,
at some point i guess a POST request will be made to a HTTP endpoint.
Tool for inspecting Chrome DevTools,Firefox Developer Tools.
This request is what you most send with Requests to stimulate that button click.
Quote:but I couldn't figure out how to get it installed into Linux Mint 17.3.
With Selenium and often in combo with PhantomJS(Selenium can load it).
It easier for there are method click() to click buttons.

Install Selenium it should work with pip
The default python3 in Mint17.3 is python3.4.
So sudo pip3 install selenium for using it for Python 3.
python3 my_prog.py to run Python 3.
Linux Python 3 environment
I have a demo with Selenium here.
Quote
#4
Esteemed Forum Participants & Lurkers:
I have been working on an application to login and download "DSL Status" information from my ZyXEL 5100Z DSL modem/router to fetch monthly DSL usage information so that I don't overrun the data cap limit set my my phone co./ISP. I have finally developed a totally software solution to this question. I have comprehensive posts on the Linux Mint forum, and I posted a copy of the successful test code on Pastebin.com. The answer to the question in the original post is answered there. Here are the links for your reference.

SOLVED: How To? Basic Inet Monthly Usage Monitor
SOLVED: HOWTO? Login DSL Modem with Python Requests: need Click "Apply" Button
SOLVED: HOW TO? Load crontab job from script?
Test Scraper - IT WORKS! <- Pastebin.com

Blessings in abundance, all the best, & ENJOY!
Art in Carlisle, PA USA
Quote
#5
I was also searching for a solution for my DSL Modem and found it really helpful.
Thanks to @DeaD_EyE for your comment.
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  Contact form button click action Man_from_India 1 239 Feb-01-2020, 06:21 PM
Last Post: snippsat
  Scraping problems with Python requests. gtlhbkkj 1 143 Jan-22-2020, 11:00 AM
Last Post: gtlhbkkj
  requests problem in python script "content type" abdlwafitahiri 4 284 Dec-29-2019, 02:29 PM
Last Post: abdlwafitahiri
  Python-selenium script for automated web-login does not work hectorKJ 2 729 Sep-10-2019, 01:29 PM
Last Post: buran
  Selenium click on popup button??? GuJu 7 1,697 Jul-20-2019, 09:21 AM
Last Post: Nizam
  How to trigger click event on Button without ID/Name gahhon 7 1,708 Feb-10-2019, 03:06 PM
Last Post: gahhon
  Click Element if displayed using Selenium and Python giaco__mar 1 1,163 Dec-27-2018, 06:19 PM
Last Post: metulburr
  Python - Scrapy Login in Baggelhsk95 3 1,478 Oct-23-2018, 04:24 PM
Last Post: stranac
  Python - Scrapy Login form Baggelhsk95 4 4,648 Oct-16-2018, 08:01 AM
Last Post: Baggelhsk95
  Requests login failure test 10 2,334 Sep-13-2018, 08:48 AM
Last Post: test

Forum Jump:


Users browsing this thread: 1 Guest(s)