Python3 + BeautifulSoup4 + lxml (HTML -> CSV) - How to loop to next HTML/new CSV Row - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html) +--- Thread: Python3 + BeautifulSoup4 + lxml (HTML -> CSV) - How to loop to next HTML/new CSV Row (/thread-25164.html) |
Python3 + BeautifulSoup4 + lxml (HTML -> CSV) - How to loop to next HTML/new CSV Row - BrandonKastning - Mar-22-2020 Hello Python Web Scrapers, This is what I am currently up against and was hoping somebody could point me in the right direction. Python3 + BeautifulSoup4 + lxml (HTML -> CSV): How to loop to the next HTML URL and save as new CSV Row in the existing .csv that the current code scrapes from. For instance: How would I make this URL do the above Next HTML URL: https://law.justia.com/cases/federal/appellate-courts/F2/999/666/308423/ Python3 Code: from urllib.request import urlopen from bs4 import BeautifulSoup html = urlopen("http://law.justia.com/cases/federal/appellate-courts/F2/999/663/308588/") bsObj = BeautifulSoup(html.read()) allOpinion = bsObj.findAll(id="opinion") import requests from bs4 import BeautifulSoup url = "http://law.justia.com/cases/federal/appellate-courts/F2/999/663/308588/" allTitle = bsObj.findAll({"title"}) allURL = url print(allOpinion) print(allTitle) print(allURL) import csv csvRow = [allOpinion,allTitle,allURL] csvfile = "current_F2_opinion_with_tags_current.csv" with open(csvfile, "a") as fp: wr = csv.writer(fp, dialect='excel') wr.writerow(csvRow) print(allOpinion[0].get_text(),url) import csv csvRow = [allOpinion[0].get_text(),allTitle[0].get_text(),allURL] csvfile = "current_F2_opinion_without_tags_current.csv" with open(csvfile, "a") as fp: wr = csv.writer(fp, dialect='excel') wr.writerow(csvRow)Thank you! Best Regards, Brandon Kastning P.S. - Everyone be safe! |