Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
web crawler problems
#4
That is not an error, that just means than nothing is being printed out at all. You didnt find the href's

try this code:
import requests
from bs4 import BeautifulSoup
 
def trade_spider(max_pages):
    page = 1
    while page <= max_pages:
        url = 'http://books.toscrape.com/catalogue/page-{}.html'.format(page)
        source_code = requests.get(url)
        plain_text = source_code.text
        soup = BeautifulSoup(plain_text, "html.parser")
        books = soup.find_all('li', {'class':'col-xs-6 col-sm-4 col-md-3 col-lg-3'})
        for book in books:
            a = book.find('a')
            link = a['href']
            title = a.find('img')['alt']
            print(link)
            print(title)
        page += 1
 
trade_spider(1)
Recommended Tutorials:
Reply


Messages In This Thread
web crawler problems - by kid_with_polio - Jun-14-2019, 10:42 PM
RE: web crawler problems - by micseydel - Jun-14-2019, 11:38 PM
RE: web crawler problems - by kid_with_polio - Jun-14-2019, 11:45 PM
RE: web crawler problems - by metulburr - Jun-15-2019, 12:33 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Web Crawler help Mr_Mafia 2 1,895 Apr-04-2020, 07:20 PM
Last Post: Mr_Mafia
  Web Crawler help takaa 39 27,270 Apr-26-2019, 12:14 PM
Last Post: stateitreal

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020