Scraping a javascript RSSI indicator

[jon333]

I have a Raspberry Pi connected to a Verizon MiFi LTE modem/router. I am trying to scrape the RSSI signal from the 'welcome page' of the device. No login is required and I'm able to get the 200 status code and also view the page content. I'm not really sure where to go from there. I'm not sure any lxml class would recognize it. Any ideas?

1 2 3 4 5 6 7 8 9 10

from lxml import html import requests   page = requests.get('<removed>',  headers={'User-Agent':'test'})   tree = html.fromstring(page.content)   rssi = tree.xpath('//*[@id="statusBar_rssi"]')   print (rssi)

returns []

Here is what Chrome inspector shows -

1	<li id="item_statusBar_rssi"><span id="statusBar_rssi" style=""><div class="rssi_4"></div>Data</span></li>

The div changes with the number of bars displayed.

Thank you.

[DeaD_EyE]

I've seen this RSSI Bar on a Huawei LTE Router.
Press F12 before you open the status page and go to the network tab.
Observe the Requests. The Huawei LTE Router is doing many requests the whole time.
A JavaScript is replacing the elements in the DOM.

I made a session with requests and use the session to get as first the status page.
After this I was requesting the status, which I got from the network tab.

The message was XML.

[jon333]

Thanks, I wasn't looking in the right place.

I see <removed>://192.168.1.1/srv/status? updating constantly.

That page outputs this

{"statusData":{"fotaData":0,"fotaState":0,"statusBarAirplaneMode":"AirplaneModeOff","statusBarBatteryChargingSource":"ChargingSourceAC","statusBarBatteryChargingState":"false","statusBarBatteryDetection":"Present","statusBarBatteryPercent":"100","statusBarBytesReceived":"574042","statusBarBytesTotal":"1117102","statusBarBytesTransmitted":"543060","statusBarClientListSize":"2","statusBarConnectionDuration":"680","statusBarConnectionState":"Connected","statusBarFemtoCellStatus":"0","statusBarGuestClientListSize":"0","statusBarMaxPrimaryClientListSize":"10","statusBarMnsAvailable":1,"statusBarMnsInUse":0,"statusBarMnsScan":2085,"statusBarNetwork":"Verizon","statusBarNetworkID":"311480","statusBarNotificationFlag":"1","statusBarPrimaryClientListSize":"0","statusBarRoaming":"None","statusBarSignalBars":"4","statusBarSimCarrierBlockedStatus":"0","statusBarSimStatus":"Ready","statusBarSmsUnreadCount":"0","statusBarTechnology":"LTE","statusBarTechnologyText":"","statusBarTrafficStatus":"None","statusBarVoiceFemtoCellStatus":"0","statusBarVoiceSignalBars":"5","statusBarVoiceSignalOperName":"Verizon Wireless","statusBarVoiceSignalRoam":"None","statusBarVoiceSignalTech":"CDMA 1XRTT","statusBarVoiceSignalTechText":"","statusBarWiFiClientListSize":"0","statusBarWiFiEnabled":0}}


I'm still not sure how to parse out "statusBarSignalBars":"4". For now all I need is to debug it to the screen.

[snippsat]

I'm still not sure how to parse out "statusBarSignalBars":"4". For now all I need is to debug it to the screen.

The output is JSON,which is very common in web API's.
If just copy our output and assassin to variable data.
I can parse it.

1 2	>>> data['statusData']["statusBarSignalBars"] '4'

You can try to get JSON with Requests,it parse JSON to a Python dictionary (same as json.load() from standard library).

1 2 3 4 5

import requests   page = requests.get('<removed>://192.168.1.1/srv/status?') data = page.json print(data['statusData']["statusBarSignalBars"])

[jon333]

Seems like it would work, but I get

I get TypeError: 'method' object is not scriptable.

[snippsat]

I get TypeError: 'method' object is not scriptable.

So shall we just guess how your code look

Check if you get 200 back.

1 2 3 4 5 6

import requests   page = requests.get('<removed>://192.168.1.1/srv/status?') print(page.status_code) # 200 The request has succeeded # Look at what you get print(page.content)

[jon333]

Sorry for not posting it, but my code looks exactly like what you suggested.

Using your last suggestion I get the 200 back, and then the same JSON I listed above. So it looks like everything is reading properly.

[snippsat]

I forgot () in json,try this.

1 2 3 4 5

import requests    page = requests.get('<removed>://192.168.1.1/srv/status?') data = page.json() print(data['statusData']["statusBarSignalBars"])

[jon333]

Thank you so much!