Python Forum
Read url from CSV and Scrape website
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Read url from CSV and Scrape website
#2
def getdata(url, values=None):
    r = requests.post(url, data=values, timeout=10)
    text = r.text
    r.close()
    return text
Your code is overcomplicated.

import csv
from bs4 import BeautifulSoup
import requests

def get(urls):
    for url in urls:
        yield requests.get(url).content.decode('utf-8')

with open('BOOK.csv') as csv_:
    reader = csv.reader(csv_)
    urls = [line[1] for line in urls if line]
    
    webpages = list(get(urls))
    
    for html in webpages:
        soup = BeautifulSoup(html, 'lxml')
        print(soup.pretify)
You will be able to put together the rest.
"As they say in Mexico 'dosvidaniya'. That makes two vidaniyas."
https://freedns.afraid.org
Reply


Messages In This Thread
RE: Read url from CSV and Scrape website - by wavic - Dec-21-2017, 12:22 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Unable to Scrape Website muhamdasim 2 2,676 Dec-27-2021, 07:49 PM
Last Post: JohnRaven
  how to scrape a website from a keyword list greenpine 2 2,437 Dec-04-2020, 03:50 PM
Last Post: greenpine
  scrape data 1 go to next page scrape data 2 and so on alkaline3 6 5,349 Mar-13-2020, 07:59 PM
Last Post: alkaline3
  why I can't scrape a website? kmkim319 7 7,686 Sep-27-2019, 03:14 PM
Last Post: kmkim319
  How do i scrape website whose page changes using javsacript _dopostback function and Prince_Bhatia 1 7,307 Aug-06-2018, 09:45 AM
Last Post: wavic
  Scrape A tags from a website Prince_Bhatia 1 4,275 Oct-15-2017, 12:56 AM
Last Post: metulburr

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020