Jul-29-2017, 10:29 PM
Esteemed Forum Participants & Lurkers:
======================================
NOTE: Inexperienced with Web Development ...
I am trying to make a simple tool to track my Inet usage so that I don't hit my ISP usage cap. My goal is a web scraper to fetch the "Total Usage" value out of the Browser-based control panel for my DSL modem/router (ZyXEL PK5001Z).
I found "Python Requests" and it seemed to be a reasonable tool for my task, and I got started with it. The big problem is that the modem login can NOT be completed with a <Return> character ... there is an "Apply" button that must be "clicked" in order to complete the login. I don't know if the login page even accepts my request into the appropriate text entry boxes, or how to activate the "Apply" button function.
Thank you for any and all comments, suggestions, and gracious assistance.
Blessings in abundance, all the best, & ENJOY!
Art in Carlisle, PA USA
======================================
NOTE: Inexperienced with Web Development ...
I am trying to make a simple tool to track my Inet usage so that I don't hit my ISP usage cap. My goal is a web scraper to fetch the "Total Usage" value out of the Browser-based control panel for my DSL modem/router (ZyXEL PK5001Z).
I found "Python Requests" and it seemed to be a reasonable tool for my task, and I got started with it. The big problem is that the modem login can NOT be completed with a <Return> character ... there is an "Apply" button that must be "clicked" in order to complete the login. I don't know if the login page even accepts my request into the appropriate text entry boxes, or how to activate the "Apply" button function.
>>> import requests >>> r = requests.get('<http modem URL>', auth=('<webtest>', '<modem-pswd>')) >>> r.status_code 200 >>> r.headers['content-type'] 'text/html; charset=iso-8859-1' >>> r.encoding 'iso-8859-1' >>> r.text <Entire page code of>: <title>Century Link Modem Configurator</title>I actually have already figured out how to fetch the parameter I need ... I was playing around with the Developer pane in Firefox and found that AFTER I get logged in, I only need to access a specific page URL that returns the data that loads the DSL Status page! I can parse that easily. Here is the beginning of the modem login page AFTER I issued the above "request.get":
LOGIN PAGE UPLOADED: . . . . . <title>Century Link Modem Configurator</title> <link href="../_css/basic.css" rel="stylesheet" type="text/css" />\r\n<script language="Javascript" . . . . . ? SHOULD BE WELCOME PAGE ?: . . . . . <title>Century Link Modem Configurator</title> <link href="../_css/index.css" . . .For the next step, the modem does NOT accept the password with a <Return> keystroke ... you MUST click the "Apply" button in order to actually log in. I think this is the relevant code on the login page:
<div class="stepWrapper"> <p><strong> 2. Click "Apply" to log in.</strong></p> <a href="#" onclick="ButtonClick(1)" class="btn apply_btn">Apply</a> </div>Roger Shruber over on the #python chat tried to help and suggested that I might try Selenium, but I couldn't figure out how to get it installed into Linux Mint 17.3.
Thank you for any and all comments, suggestions, and gracious assistance.
Blessings in abundance, all the best, & ENJOY!
Art in Carlisle, PA USA