Webscrape from my webpage and store in database and send to grafana

Webscrape from my webpage and store in database and send to grafana - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: Webscrape from my webpage and store in database and send to grafana (/thread-40320.html)

Webscrape from my webpage and store in database and send to grafana - Armond - Jul-10-2023

Hi!

I want to webscrape a value from my webpage and send it to a database every 5 minutes and upload it to grafana.
I have done the webscrape python:

import requests
from bs4 import BeautifulSoup
URL = "http://ludvikasegel.com/wx/cloudbase.asp"
page = requests.get(URL)
soup = BeautifulSoup(page.content, "html.parser")
molnbas = soup.find_all("div", class_="cloudbase")
print (molnbas[0].text.strip())

I am thinking of SQlite3 database.
I am not sure how to do it or how the python will look like.
I am also searching the internet about it.

RE: Webscrape from my webpage and store in database and send to grafana - LoriSattler - Jul-12-2023

Have you found your answer?

RE: Webscrape from my webpage and store in database and send to grafana - Armond - Jul-12-2023

(Jul-12-2023, 07:09 AM)LoriSattler Wrote: Have you found your answer?

Well, I have got some help and have this code now:

import requests
import sqlite3
import datetime 

from bs4 import BeautifulSoup
URL = "http://ludvikasegel.com/wx/cloudbase.asp"
page = requests.get(URL)
soup = BeautifulSoup(page.content, "html.parser")
molnbas = soup.find_all("div", class_="cloudbase")
print (molnbas[0].text.strip())


with sqlite3.connect('database.db') as con, open("d:\Cloudbase python\schema.sql") as f:
    con.executescript(f.read())


    desired_value = molnbas[0].text.strip()
    desired_value = molnbas[0].text.strip()
    timestamp = datetime.datetime.now().isoformat()  # timestamp, change to whatever you want

with sqlite3.connect("d:\Cloudbase python\database.db") as sqlite_connection:
    sqlite_connection.execute(
        "INSERT INTO table_name VALUES (?, ?)",
        (desired_value, timestamp)
    )

And I got values in SQlite database.
The question is if I am on the right track.
Next issue is to get the data to Grafana and plot a graph.