Python Forum
HOWTO? Login DSL Modem with Python Requests: need Click "Apply" Button
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
HOWTO? Login DSL Modem with Python Requests: need Click "Apply" Button
#1
Esteemed Forum Participants & Lurkers:
======================================
NOTE: Inexperienced with Web Development ...

I am trying to make a simple tool to track my Inet usage so that I don't hit my ISP usage cap. My goal is a web scraper to fetch the "Total Usage" value out of the Browser-based control panel for my DSL modem/router (ZyXEL PK5001Z).

I found "Python Requests" and it seemed to be a reasonable tool for my task, and I got started with it. The big problem is that the modem login can NOT be completed with a <Return> character ... there is an "Apply" button that must be "clicked" in order to complete the login. I don't know if the login page even accepts my request into the appropriate text entry boxes, or how to activate the "Apply" button function.
>>> import requests
>>> r = requests.get('<http modem URL>', auth=('<webtest>', '<modem-pswd>'))
>>> r.status_code
200
>>> r.headers['content-type']
'text/html; charset=iso-8859-1'
>>> r.encoding
'iso-8859-1'
>>> r.text
<Entire page code of>: <title>Century Link  Modem Configurator</title>
I actually have already figured out how to fetch the parameter I need ... I was playing around with the Developer pane in Firefox and found that AFTER I get logged in, I only need to access a specific page URL that returns the data that loads the DSL Status page! I can parse that easily. Here is the beginning of the modem login page AFTER I issued the above "request.get":
LOGIN PAGE UPLOADED:
 . . . . .
<title>Century Link  Modem Configurator</title>
<link href="../_css/basic.css" rel="stylesheet" type="text/css" />\r\n<script language="Javascript"
 . . . . .
? SHOULD BE WELCOME PAGE ?:
 . . . . .
<title>Century Link Modem Configurator</title>
<link href="../_css/index.css" . . . 
For the next step, the modem does NOT accept the password with a <Return> keystroke ... you MUST click the "Apply" button in order to actually log in. I think this is the relevant code on the login page:
<div class="stepWrapper">
  <p><strong> 2. Click &quot;Apply&quot; to log in.</strong></p>
  <a href="#" onclick="ButtonClick(1)" class="btn apply_btn">Apply</a>
</div>
Roger Shruber over on the #python chat tried to help and suggested that I might try Selenium, but I couldn't figure out how to get it installed into Linux Mint 17.3.

Thank you for any and all comments, suggestions, and gracious assistance.
Blessings in abundance, all the best, & ENJOY!
Art in Carlisle, PA USA
Reply
#2
I think you need a Session. No problem, requests have this already.
First create a session object, then get the login page:

session = requests.Session()
response = session.get('http://192.168.192.1/cgi-bin/status.cgi') # in my case
# then you look in response.text for a form, there is an action and it's value is the url based
# on the current path you're on, in my case i used BeatifulSoup to find it.
# but you can use also a webbrowser
# <form action="sendResult.cgi?section=login" id="mainForm" method="post">
# <input name="loginDelay" type="hidden" value="0"/>
# <input name="page" type="hidden" value="login"/>
# <div>Benutzername</div>
# <input name="username" tabindex="1" type="text"/>
# <div>Passwort</div>
# <input name="password" tabindex="2" type="password"/>
# </form>

# so we need something to prepare the data for the form
# all fields should be submitted, also the hidden fields
# with requests you can just use a dict for this task
login = {'loginDelay': '0', 'page': 'login', 'username': 'admin', 'password': 'admin'}

# then you have to use the post method together with the target (action url) and
# login data
response = session.post('http://192.168.192.1/cgi-bin/sendResult.cgi?section=login', data=login)
if response.status_code == 200:
    print('Success')
    print(response.text)
For further webscraping you should use Beautiful Soup or Scrapy.
You can use also regex, but this is more complicated.

For full help, you've to post the whole html code. Without this we can only guess. I think it's still a form on your side wrapped around with javascript for the fancy stuff.
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply
#3
(Jul-29-2017, 10:29 PM)Webtest Wrote: you MUST click the "Apply" button in order to actually log in.
Is not so straightforward to figure it out.
onclick="ButtonClick(1)"
ButtonClick(1) is a function/method in JavaScript look at what it dos,
at some point i guess a POST request will be made to a HTTP endpoint.
Tool for inspecting Chrome DevTools,Firefox Developer Tools.
This request is what you most send with Requests to stimulate that button click.
Quote:but I couldn't figure out how to get it installed into Linux Mint 17.3.
With Selenium and often in combo with PhantomJS(Selenium can load it).
It easier for there are method click() to click buttons.

Install Selenium it should work with pip
The default python3 in Mint17.3 is python3.4.
So sudo pip3 install selenium for using it for Python 3.
python3 my_prog.py to run Python 3.
Linux Python 3 environment
I have a demo with Selenium here.
Reply
#4
Esteemed Forum Participants & Lurkers:
I have been working on an application to login and download "DSL Status" information from my ZyXEL 5100Z DSL modem/router to fetch monthly DSL usage information so that I don't overrun the data cap limit set my my phone co./ISP. I have finally developed a totally software solution to this question. I have comprehensive posts on the Linux Mint forum, and I posted a copy of the successful test code on Pastebin.com. The answer to the question in the original post is answered there. Here are the links for your reference.

SOLVED: How To? Basic Inet Monthly Usage Monitor
SOLVED: HOWTO? Login DSL Modem with Python Requests: need Click "Apply" Button
SOLVED: HOW TO? Load crontab job from script?
Test Scraper - IT WORKS! <- Pastebin.com

Blessings in abundance, all the best, & ENJOY!
Art in Carlisle, PA USA
Reply
#5
I was also searching for a solution for my DSL Modem and found it really helpful.
Thanks to @DeaD_EyE for your comment.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  cant click button by script at page michael1834 1 995 Dec-08-2023, 04:44 PM
Last Post: SpongeB0B
  Click on a button on web page using Selenium Pavel_47 7 4,564 Jan-05-2023, 04:20 AM
Last Post: ellapurnellrt
  POST requests - different requests return the same response Default_001 3 1,901 Mar-10-2022, 11:26 PM
Last Post: Default_001
  Show HTML in Python application and handle click SamHobbs 2 2,662 Sep-28-2021, 06:27 PM
Last Post: SamHobbs
  Login and download an exported csv file within a ribbon/button in a website Alekhya 0 2,616 Feb-26-2021, 04:15 PM
Last Post: Alekhya
  button click error rafarangel 2 3,082 Feb-11-2021, 08:19 PM
Last Post: buran
  Problem with logging in on website - python w/ requests GoldeNx 6 5,210 Sep-25-2020, 10:52 AM
Last Post: snippsat
  Log In Button Won't Click - Python Selenium Webdriver samlee916 2 3,771 Jun-07-2020, 04:42 PM
Last Post: samlee916
  Python Webscraping with a Login Website warriordazza 0 2,571 Jun-07-2020, 07:04 AM
Last Post: warriordazza
  How to click facebook message button JanelleGuthrie 2 2,370 May-14-2020, 06:02 PM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020