Python Forum
Python3 + BeautifulSoup4 + lxml (HTML -> CSV) - How to loop to next HTML/new CSV Row
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Python3 + BeautifulSoup4 + lxml (HTML -> CSV) - How to loop to next HTML/new CSV Row
#1
Hello Python Web Scrapers,

This is what I am currently up against and was hoping somebody could point me in the right direction.

Python3 + BeautifulSoup4 + lxml (HTML -> CSV):

How to loop to the next HTML URL and save as new CSV Row in the existing .csv that the current code scrapes from.

For instance: How would I make this URL do the above

Next HTML URL: https://law.justia.com/cases/federal/app...66/308423/

Python3 Code:

from urllib.request import urlopen
from bs4 import BeautifulSoup
html = urlopen("http://law.justia.com/cases/federal/appellate-courts/F2/999/663/308588/")
bsObj = BeautifulSoup(html.read())
allOpinion = bsObj.findAll(id="opinion")
import requests
from bs4 import BeautifulSoup

url = "http://law.justia.com/cases/federal/appellate-courts/F2/999/663/308588/"
allTitle = bsObj.findAll({"title"})
allURL = url

print(allOpinion)
print(allTitle)
print(allURL)

import csv
csvRow = [allOpinion,allTitle,allURL]
csvfile = "current_F2_opinion_with_tags_current.csv"
with open(csvfile, "a") as fp:
    wr = csv.writer(fp, dialect='excel')
    wr.writerow(csvRow)

print(allOpinion[0].get_text(),url)
 
import csv
csvRow = [allOpinion[0].get_text(),allTitle[0].get_text(),allURL]
csvfile = "current_F2_opinion_without_tags_current.csv"
with open(csvfile, "a") as fp:
    wr = csv.writer(fp, dialect='excel')
    wr.writerow(csvRow)
Thank you!

Best Regards,

Brandon Kastning

P.S. - Everyone be safe!
“And one of the elders saith unto me, Weep not: behold, the Lion of the tribe of Juda, the Root of David, hath prevailed to open the book,...” - Revelation 5:5 (KJV)

“And oppress not the widow, nor the fatherless, the stranger, nor the poor; and ...” - Zechariah 7:10 (KJV)

#LetHISPeopleGo

Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Div Class HTML selector in Python Artur 1 450 5 hours ago
Last Post: StevenSnyder
  Trying to scrape data from HTML with no identifiers pythonpaul32 2 795 Dec-02-2023, 03:42 AM
Last Post: pythonpaul32
  How to find element in a deeply nested html structure Orientation_group 5 1,177 Oct-09-2023, 10:13 AM
Last Post: Larz60+
  Flask - use bootstrap styles in HTML Krayna 1 1,003 Aug-29-2023, 02:33 PM
Last Post: menator01
  Getting a URL from Amazon using requests-html, or beautifulsoup aaander 1 1,618 Nov-06-2022, 10:59 PM
Last Post: snippsat
  Populating list items to html code and create individualized html code files ChainyDaisy 0 1,560 Sep-21-2022, 07:18 PM
Last Post: ChainyDaisy
  selenium returns junk instead of html klaarnou 5 2,182 Mar-27-2022, 07:20 AM
Last Post: klaarnou
  requests-html + Beautifulsoup klaarnou 0 2,399 Mar-21-2022, 05:31 PM
Last Post: klaarnou
  Post Python to HTML/CSS Extra 3 1,839 Feb-17-2022, 06:07 PM
Last Post: snippsat
Question Python Obstacles | Jeet-Kune-Do | BS4 (Tags > MariaDB) [URL/Local HTML] BrandonKastning 0 1,400 Feb-08-2022, 08:55 PM
Last Post: BrandonKastning

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020