Python Forum
No Internet connection when running a Python script
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
No Internet connection when running a Python script
#4
import requests
import os
from bs4 import BeautifulSoup
import time
import logging
import smtplib as smtp

try:
    import lxml
except ImportError:
    raise RuntimeError("Please install lxml")


URL_TO_MONITOR = "https://www.yahoo.com/"  # change this to the URL you want to monitor
DELAY_TIME = 15  # seconds


def process_html(string):
    soup = BeautifulSoup(string, features="lxml")

    # make the html look good
    soup.prettify()

    # remove script tags
    for s in soup.select("script"):
        s.extract()

    # remove meta tags
    for s in soup.select("meta"):
        s.extract()

    # convert to a string, remove '\r', and return
    return str(soup).replace("\r", "")


def webpage_was_changed():
    """Returns true if the webpage was changed, otherwise false."""
    headers = {
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36",
        "Pragma": "no-cache",
        "Cache-Control": "no-cache",
    }
    response = requests.get(URL_TO_MONITOR, headers=headers)

    # create the previous_content.txt if it doesn't exist
    if not os.path.exists("previous_content.txt"):
        open("previous_content.txt", "w+").close()

    filehandle = open("previous_content.txt", "r")
    previous_response_html = filehandle.read()
    filehandle.close()

    processed_response_html = process_html(response.text)

    if processed_response_html == previous_response_html:
        return False
    else:
        filehandle = open("previous_content.txt", "w")
        filehandle.write(processed_response_html)
        filehandle.close()
        return True


def main():
    log = logging.getLogger(__name__)
    logging.basicConfig(
        level=os.environ.get("LOGLEVEL", "INFO"), format="%(asctime)s %(message)s"
    )
    log.info("Running Website Monitor")

    while True:
        try:
            if webpage_was_changed():
                log.info("WEBPAGE WAS CHANGED.")
            else:
                log.info("Webpage was not changed.")

        except Exception as e:
            log.exception(e)

        time.sleep(DELAY_TIME)


if __name__ == "__main__":
    main()
Do not use a bare except. It suppresses programming errors. In this case lxml wasn't installed, which is required by bs4 (explicit features="lxml").

I didn't check the other stuff, just used an format tool (ruff format).
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply


Messages In This Thread
RE: No Internet connection when running a Python script - by DeaD_EyE - Mar-10-2024, 05:53 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  I don't know what is wrong (Python and SQL connection) shereen 3 536 Apr-01-2024, 08:56 AM
Last Post: Pedroski55
Question Running Python script through Task Scheduler? Winfried 8 977 Mar-10-2024, 07:24 PM
Last Post: Winfried
  Connection LTspice-Python with PyLTSpice bartel90 0 483 Feb-05-2024, 11:46 AM
Last Post: bartel90
  Virtual Env changing mysql connection string in python Fredesetes 0 470 Dec-20-2023, 04:06 PM
Last Post: Fredesetes
  connection python and SQL dawid294 4 873 Dec-12-2023, 08:22 AM
Last Post: Pedroski55
  Help Running Python Script in Mac OS emojistickers 0 424 Nov-20-2023, 01:58 PM
Last Post: emojistickers
  Trying to make a board with turtle, nothing happens when running script Quascia 3 828 Nov-01-2023, 03:11 PM
Last Post: deanhystad
  Is there a *.bat DOS batch script to *.py Python Script converter? pstein 3 3,794 Jun-29-2023, 11:57 AM
Last Post: gologica
  Python script running under windows over nssm.exe JaroslavZ 0 820 May-12-2023, 09:22 AM
Last Post: JaroslavZ
  Networking Issues - Python GUI client and server connection always freezes Veritas_Vos_Liberabit24 0 820 Mar-21-2023, 03:18 AM
Last Post: Veritas_Vos_Liberabit24

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020