Python Forum

Full Version: Extract data from a webpage
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Hello all;

I'm would like to ask some advices to the python community concerning a script I'm trying to develop.

In fact, I'm living near a lake in Italy, and from time to time the water level is very close from my house, so I'm looking a way to pick-up a value from a webpage dealing with the lake level, and sending me notification when this level reach a high value.

The webpage giving the value is this one :

the value I want to retrieve is the one after "Stazione di Leggiuno", by example today : 194.12, as indicated on the website.

Using examples found on the web, I used Request and beautifulsoup to retrieve this info :

import requests
from bs4 import BeautifulSoup

# using the requests module, we use the "get" funtion
result = requests.get("")


# let us store the page content of the website
# from requests to a variable

src = result.content
so, I receive the 'result' on my screen, with the data I want to import :
Quote:document.getElementById("Livello').InnerHTML="<strong>Stazione di Leggiuno: "+data.legb.livello[ data.legb.livello.lenght-1]+"<font color='#417 FDA'>

I'was thinking that the value I wanted to extract should be just after the mark "Stazione di Leggiuno", but instead, I got this "+data.legb.livello", and cannor recover the result displayed on the webpage (in this case 194.12).

Anyone of the python community has been face to this problem ? how is it possible to retrieve the numerical value , if possible ?

Many thanks in advance for your help !
I get an invalid URL when I attempt to request page.
Please advise of correct URL

I can confirm this URL :

strangely, when I try to open it with Safari, this is not working, but with Firefox, without any problem.
I think you're going to need selenium, here's some starter code:
from selenium import webdriver
from import By
from bs4 import BeautifulSoup
import time

class WaterLevel:
    def __init__(self):
    def start_browser(self):
        caps = webdriver.DesiredCapabilities().FIREFOX
        caps["marionette"] = True
        self.browser = webdriver.Firefox(capabilities=caps)
    def stop_browser(self):

    def analyze_page(self):
        url =  ''
        element = self.browser.find_element(By.XPATH, '/html/body/div[1]/div[4]/div[1]/div[2]/div/div/div/table[1]/tbody/tr[2]/td[1]/div/i')


if __name__ == '__main__':
which produces the following output:
(12-11-2019, ore 11.30)
If looking closer at page so dos it give back json data with values.
Then can use this json data and drop Selenium in this case.
import requests
from datetime import datetime

url = ''
response = requests.get(url)
livello = response.json()
livello_val = livello['legb']['livello_last']
livello_last = livello['legb']['livello_last_time']
livello_last = datetime.fromtimestamp(livello_last)
print(f'<{livello_val}> at date {livello_last}')
<194.13> at date 2019-11-12 18:10:00
I have a page with no problems.

Tell you managed to solve the problem?