Python Forum
Web scraper not populating .txt with scraped data
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Web scraper not populating .txt with scraped data
#1
Hey everyone,

I was wondering if I could get you guys to help me out a little. I'm attempting to make a web scraper to scrape a site for some strings of numbers. I'm first trying to scrape a list of links and then join those URLs and scrape the strings of numbers that I'm looking for. At the end of that I'm just trying to save the scraped strings of numbers to a .txt file for later use.

I'm surprisingly not getting any errors, but my .txt file is not being populated with the scraped data. I'm thinking that maybe I'm not directing it correctly through the html via html tags/classes?


here is my code:

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

# Scrape the links to each monthly results page
url = "http://www.calotteryx.com/Fantasy-5/drawing-results-calendar.htm"
response = requests.get(url)
content = response.content
soup = BeautifulSoup(content, 'html.parser')
links = []
for link in soup.find_all('a', class_='noline'):
    anchor = link.find('a')
    if anchor and (href := anchor.get('href')):
        links.append(urljoin(url, href))

# Scrape the winning numbers for each monthly results page
winning_numbers = []
for link in links:
    response = requests.get(link)
    content = response.content
    soup = BeautifulSoup(content, 'html.parser')
    for tag in soup.find_all('div', class_='ball blue5 fcblack1'):
        numbers = tag.text.strip().split()
        winning_numbers.append(numbers)

# Write the winning numbers to a file
with open('winning_numbers.txt', 'w') as f:
    for numbers in winning_numbers:
        f.write(''.join(numbers) + '\n')
I joined a while ago, but I'm still not good at coding lol, so please forgive my ignorance, or my bad looking code. Any help in pointing out my mistakes and how to fix them would be greatly appreciated please.
Reply


Messages In This Thread
Web scraper not populating .txt with scraped data - by BlackHeart - Apr-01-2023, 07:08 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Weird characters scraped samuelbachorik 3 1,027 Oct-29-2023, 02:36 PM
Last Post: DeaD_EyE
  Web scraper tomenzo123 8 4,519 Aug-18-2023, 12:45 PM
Last Post: Gaurav_Kumar
  Python Obstacles | Krav Maga | Wiki Scraped Content [Column Copy] BrandonKastning 4 2,312 Jan-03-2022, 06:59 AM
Last Post: BrandonKastning
  Python Obstacles | Kapap | Wiki Scraped Content [Column Nulling] BrandonKastning 2 1,794 Jan-03-2022, 04:26 AM
Last Post: BrandonKastning
  Image Scraper (beautifulsoup), stopped working, need to help see why woodmister 9 4,200 Jan-12-2021, 04:10 PM
Last Post: woodmister
  Any way to remove HTML tags from scraped data? (I want text only) SeBz2020uk 1 3,548 Nov-02-2020, 08:12 PM
Last Post: Larz60+
  cant loop through scraped site matt42 3 2,497 Aug-12-2020, 06:48 AM
Last Post: ndc85430
  Court Opinion Scraper in Python w/ BS4 (Currently exports to CSV) need help with SQL MidnightDreamer 4 3,089 Mar-12-2020, 09:57 AM
Last Post: BrandonKastning
  Pre-populating WTForms form values for edit danfoster 0 2,491 Feb-25-2020, 01:37 PM
Last Post: danfoster
  Python using BS scraper paulfearn100 1 2,603 Feb-07-2020, 10:22 PM
Last Post: snippsat

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020