Python Forum
Scraping a javascript RSSI indicator
Thread Rating:
  • 1 Vote(s) - 3 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Scraping a javascript RSSI indicator
#1
I have a Raspberry Pi connected to a Verizon MiFi LTE modem/router. I am trying to scrape the RSSI signal from the 'welcome page' of the device. No login is required and I'm able to get the 200 status code and also view the page content. I'm not really sure where to go from there. I'm not sure any lxml class would recognize it. Any ideas?

from lxml import html
import requests

page = requests.get('<removed>',  headers={'User-Agent':'test'})

tree = html.fromstring(page.content)

rssi = tree.xpath('//*[@id="statusBar_rssi"]')

print (rssi)
returns []

Here is what Chrome inspector shows -
<li id="item_statusBar_rssi"><span id="statusBar_rssi" style=""><div class="rssi_4"></div>Data</span></li>
The div changes with the number of bars displayed.

Thank you.
Reply
#2
I've seen this RSSI Bar on a Huawei LTE Router.
Press F12 before you open the status page and go to the network tab.
Observe the Requests. The Huawei LTE Router is doing many requests the whole time.
A JavaScript is replacing the elements in the DOM.

I made a session with requests and use the session to get as first the status page.
After this I was requesting the status, which I got from the network tab.

The message was XML.
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply
#3
Thanks, I wasn't looking in the right place.

I see <removed>://192.168.1.1/srv/status? updating constantly.

That page outputs this

{"statusData":{"fotaData":0,"fotaState":0,"statusBarAirplaneMode":"AirplaneModeOff","statusBarBatteryChargingSource":"ChargingSourceAC","statusBarBatteryChargingState":"false","statusBarBatteryDetection":"Present","statusBarBatteryPercent":"100","statusBarBytesReceived":"574042","statusBarBytesTotal":"1117102","statusBarBytesTransmitted":"543060","statusBarClientListSize":"2","statusBarConnectionDuration":"680","statusBarConnectionState":"Connected","statusBarFemtoCellStatus":"0","statusBarGuestClientListSize":"0","statusBarMaxPrimaryClientListSize":"10","statusBarMnsAvailable":1,"statusBarMnsInUse":0,"statusBarMnsScan":2085,"statusBarNetwork":"Verizon","statusBarNetworkID":"311480","statusBarNotificationFlag":"1","statusBarPrimaryClientListSize":"0","statusBarRoaming":"None","statusBarSignalBars":"4","statusBarSimCarrierBlockedStatus":"0","statusBarSimStatus":"Ready","statusBarSmsUnreadCount":"0","statusBarTechnology":"LTE","statusBarTechnologyText":"","statusBarTrafficStatus":"None","statusBarVoiceFemtoCellStatus":"0","statusBarVoiceSignalBars":"5","statusBarVoiceSignalOperName":"Verizon Wireless","statusBarVoiceSignalRoam":"None","statusBarVoiceSignalTech":"CDMA 1XRTT","statusBarVoiceSignalTechText":"","statusBarWiFiClientListSize":"0","statusBarWiFiEnabled":0}}


I'm still not sure how to parse out "statusBarSignalBars":"4". For now all I need is to debug it to the screen.
Reply
#4
(Aug-10-2017, 02:44 PM)jon333 Wrote: I'm still not sure how to parse out "statusBarSignalBars":"4". For now all I need is to debug it to the screen.
The output is JSON,which is very common in web API's.
If just copy our output and assassin to variable data.
I can parse it.
>>> data['statusData']["statusBarSignalBars"]
'4'
You can try to get JSON with Requests,it parse JSON to a Python dictionary (same as json.load() from standard library).
import requests

page = requests.get('<removed>://192.168.1.1/srv/status?')
data = page.json
print(data['statusData']["statusBarSignalBars"])
Reply
#5
Seems like it would work, but I get

I get TypeError: 'method' object is not scriptable.
Reply
#6
(Aug-10-2017, 03:22 PM)jon333 Wrote: I get TypeError: 'method' object is not scriptable.
So shall we just guess how your code look Undecided

Check if you get 200 back.
import requests

page = requests.get('<removed>://192.168.1.1/srv/status?')
print(page.status_code) # 200 The request has succeeded
# Look at what you get
print(page.content)
Reply
#7
Sorry for not posting it, but my code looks exactly like what you suggested.

Using your last suggestion I get the 200 back, and then the same JSON I listed above. So it looks like everything is reading properly.
Reply
#8
I forgot () in json,try this.
import requests
 
page = requests.get('<removed>://192.168.1.1/srv/status?')
data = page.json()
print(data['statusData']["statusBarSignalBars"])
Reply
#9
Thank you so much!
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  CandleStick & MACD Indicator using plotly.graph_object bianca 0 528 Aug-18-2023, 06:46 PM
Last Post: bianca
Brick Javascript based web page scraping amjadraza26 1 1,427 Oct-21-2021, 09:36 AM
Last Post: Larz60+
  Web scraping Possible JavaScript issue johnboy1974 2 2,022 Apr-11-2021, 08:53 AM
Last Post: johnboy1974
  Progress Indicator for Xmodem 0.4.6 KenHorse 1 1,931 Jan-30-2021, 07:12 PM
Last Post: bowlofred
  Creating a list of RSSI value and send it to the server. Azuan 0 2,614 Jun-08-2020, 11:22 PM
Last Post: Azuan

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020