Python Forum
My first Web Scraping project
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
My first Web Scraping project
#1
So, recently, I have been learning web scraping with tutorials and videos and other things like that. Here's my first go at a simple project that helps in displaying software developer jobs available in New York from the Monster.com site
import requests
from bs4 import BeautifulSoup

URL = "https://www.monster.com/jobs/search/?q=Software-Developer&where=NewYork"
page = requests.get(URL)

soup = BeautifulSoup(page.content, 'html.parser')

results = soup.find(id='ResultsContainer')


job_elements = results.find_all('section', class_='card-content')

for job_element in job_elements:
    title_element = job_element.find('h2', class_='title')
    company_element = job_element.find('div', class_='company')
    location_element = job_element.find('div', class_='location')
    if None in (title_element, company_element, location_element):
        continue
    print(title_element.text.strip())
    print(company_element.text.strip())
    print(location_element.text.strip())
    print()
Any feedback would be appreciated!
(And yes, I had to take some help for this but please - I'm a beginner in this topic)
pyzyx3qwerty
"The greatest glory in living lies not in never falling, but in rising every time we fall." - Nelson Mandela
Need help on the forum? Visit help @ python forum
For learning more and more about python, visit Python docs
Reply
#2
Pretty clean compared to others i have seen. Web scraping can get messy quickly. So it is important to keep it clean and organized.
Recommended Tutorials:
Reply
#3
Well, I had learnt that not keeping it messy was one of the most important steps in web scrapping, so I did the best I could
pyzyx3qwerty
"The greatest glory in living lies not in never falling, but in rising every time we fall." - Nelson Mandela
Need help on the forum? Visit help @ python forum
For learning more and more about python, visit Python docs
Reply
#4
elements = (title_element, company_element, location_element)
if all(elements):
    for element in elements:
        print(element.text.strip())
    print()
the part where you search for each element can also be shortened with a loop if you have a container type with what you search for
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply
#5
Thanks for your suggestion @buran!
pyzyx3qwerty
"The greatest glory in living lies not in never falling, but in rising every time we fall." - Nelson Mandela
Need help on the forum? Visit help @ python forum
For learning more and more about python, visit Python docs
Reply


Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020