Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Scrape A tags from a website
#1
hi,

i am trying to scrape https://www.efinancialcareers.com/search...TANCE_FIVE

in this link i am trying to scrape the hidden jobs id:

jobs id are just below class_="jobPreview well which has class anchor
but when i run my code it exports csv without jobs id
can someone please tell what is wrong?
below is my code:
import requests
from bs4 import BeautifulSoup
import csv
import json
import time
import os
import datetime

url="https://www.efinancialcareers.com/search?page=1&sortBy=POSTED_DESC&searchMode=DEFAULT_SEARCH&jobSearchId=RUJFMEZDNjA2RTJEREJEMDcyMzlBQ0YyMEFDQjc1MjUuMTQ4NTE5MDY3NTI0Ni4tMTQ1Mjc4ODU3NQ%3D%3D&updateEmitter=SORT_BY&filterGroupForm.includeRefreshed=true&filterGroupForm.datePosted=OTHER"
final_data = []

r = requests.get(url)
data = r.text

if r.status_code == 200:
    print("Webiste is Perfect to Scrape, Please proceed and block yourself in JAIL")
else:
    quit

datetime.datetime.now()

soup = BeautifulSoup(data, "html.parser")
get_details = soup.find_all(class_="jobPreview well")
#while len(get_details)>0:
for jobs in get_details:
    title = jobs.find_all("h3")
    for job in title:
        geta = job.find_all("a")
        onlya = ""
        links = ""
        sublist=[]
        for a in geta:
            onlya = a.text
            getid = a.get("id")
            sublist.append(onlya)
            #sublist.append(getid)
        for link in geta:
            links = link.get("href")
            sublist.append(links)
            print(sublist)
    a_lst = jobs.find_all("a")
    for i in a_lst:
        an = i.find_all(class_="anchor")
        getid = ""
        for anch in an:
            getid = anch.get("id")
            sublist.append(getid)
            print(sublist)
        sublist.append(final_data)
filename = "jobs.csv"
with open("./"+filename, "w")as csvfile:
    csvfile = csv.writer(csvfile, delimiter=",")
    csvfile.writerow("")
    for i in range(0, len(final_data)):
        csvfile.writerow(final_data[i])
Reply


Messages In This Thread
Scrape A tags from a website - by Prince_Bhatia - Oct-14-2017, 10:53 PM
RE: Scrape A tags from a website - by metulburr - Oct-15-2017, 12:56 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Unable to Scrape Website muhamdasim 2 2,567 Dec-27-2021, 07:49 PM
Last Post: JohnRaven
  how to scrape a website from a keyword list greenpine 2 2,329 Dec-04-2020, 03:50 PM
Last Post: greenpine
  scrape data 1 go to next page scrape data 2 and so on alkaline3 6 5,091 Mar-13-2020, 07:59 PM
Last Post: alkaline3
  Read url from CSV and Scrape website Prince_Bhatia 3 10,192 Jan-08-2020, 09:08 AM
Last Post: binaryanimal
  why I can't scrape a website? kmkim319 7 7,396 Sep-27-2019, 03:14 PM
Last Post: kmkim319
  How do i scrape website whose page changes using javsacript _dopostback function and Prince_Bhatia 1 7,161 Aug-06-2018, 09:45 AM
Last Post: wavic

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020