Good Afternoon,
I have a question. I am new to Python and I have a scraping script that is setup in Debian (moving to Raspberry Pi 5) where I scrap the market data from precious metals daily at 3:00pm US CST. The first day it would great.. .then I get scripting errors (Stacktrace) with multiple hexadecimal strings. I was told that it could be the issue with the chromedrive being out of sync. but when I check the versions it matches the version of chrome installed. I am using Selenium for this process.
When I reboot it acts like selenium is not even loaded but when I try to re-install it it states it is there. So honestly I am lost as to how to stablize this script to run daily via my CRON Job. I am sure I am missing something but I am not sure what it could be. I have attached the script since there is nothing in this that is critical or private.
I have a question. I am new to Python and I have a scraping script that is setup in Debian (moving to Raspberry Pi 5) where I scrap the market data from precious metals daily at 3:00pm US CST. The first day it would great.. .then I get scripting errors (Stacktrace) with multiple hexadecimal strings. I was told that it could be the issue with the chromedrive being out of sync. but when I check the versions it matches the version of chrome installed. I am using Selenium for this process.
When I reboot it acts like selenium is not even loaded but when I try to re-install it it states it is there. So honestly I am lost as to how to stablize this script to run daily via my CRON Job. I am sure I am missing something but I am not sure what it could be. I have attached the script since there is nothing in this that is critical or private.
import csv
import os
from datetime import datetime
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from webdriver_manager.chrome import ChromeDriverManager
def fetch_apmex_prices():
url = "https://www.apmex.com/gold-and-silver-price-charts"
output_file = "daily_prices.csv"
# Set up Chrome options (do NOT use headless, xvfb simulates display)
options = Options()
options.add_argument("--headless=new")
options.binary_location = "/usr/bin/google-chrome"
options.add_argument("--disable-gpu")
options.add_argument("--no-sandbox")
options.add_argument("--disable-dev-shm-usage")
# options.add_argument("--disable-software-rasterizer")
try:
# Start Chrome browser
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
driver.get("https://www.google.com")
# Wait for the elements to load
wait = WebDriverWait(driver, 15)
gold_elem = wait.until(EC.presence_of_element_located((
By.XPATH, "//div[contains(text(), 'Gold')]/following-sibling::div"
)))
silver_elem = wait.until(EC.presence_of_element_located((
By.XPATH, "//div[contains(text(), 'Silver')]/following-sibling::div"
)))
# Extract and log data
gold_price = gold_elem.text.strip()
silver_price = silver_elem.text.strip()
timestamp = datetime.now().isoformat()
print(f"[{timestamp}] ✅ Gold: {gold_price} | Silver: {silver_price}")
# Write to CSV file (create if it doesn't exist)
file_exists = os.path.isfile(output_file)
with open(output_file, "a", newline="") as csvfile:
writer = csv.writer(csvfile)
if not file_exists:
writer.writerow(["Timestamp", "Gold Price", "Silver Price"])
writer.writerow([timestamp, gold_price, silver_price])
except Exception as e:
print(f"❌ An error occurred: {e}")
finally:
driver.quit()
if __name__ == "__main__":
fetch_apmex_prices()