Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 web crawler problems
#1
Hi, I followed along with a web crawler tutorial on Youtube but I can't seem to get the code to work even though I used an updated website. I have all the used packages installed and at first I got a traceback error with BeautifulSoup() that said it needed an additional parameter of features = "html.parser"

İmage


I inserted this and now the traceback I receive is nothing at all. Does anyone have a solution to this issue or know why it is caused?

İmage
Quote
#2
Please post code in code tags, we pretty much never want to see images of text.
Quote
#3
import requests
from bs4 import BeautifulSoup

def trade_spider(max_pages):
    page = 1
    while page <= max_pages:
        url = 'http://books.toscrape.com/catalogue/page-' + str(page) + '.html'
        source_code = requests.get(url)
        plain_text = source_code.text
        soup = BeautifulSoup(plain_text, features="html.parser")
        for link in soup.findAll('a', {'class': 'title'}):
            href = link.get('href')
            print(href)
        page += 1

trade_spider(1)
here's the traceback:

"C:\Users\Jake\PycharmProjects\practice baby\venv\Scripts\python.exe" "C:/Users/Jake/PycharmProjects/practice baby/web crawler 2.py"

Process finished with exit code 0
Quote
#4
That is not an error, that just means than nothing is being printed out at all. You didnt find the href's

try this code:
import requests
from bs4 import BeautifulSoup
 
def trade_spider(max_pages):
    page = 1
    while page <= max_pages:
        url = 'http://books.toscrape.com/catalogue/page-{}.html'.format(page)
        source_code = requests.get(url)
        plain_text = source_code.text
        soup = BeautifulSoup(plain_text, "html.parser")
        books = soup.find_all('li', {'class':'col-xs-6 col-sm-4 col-md-3 col-lg-3'})
        for book in books:
            a = book.find('a')
            link = a['href']
            title = a.find('img')['alt']
            print(link)
            print(title)
        page += 1
 
trade_spider(1)

Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  Web Crawler help takaa 39 10,476 Apr-26-2019, 12:14 PM
Last Post: stateitreal
  crawler: Get external page Truman 3 396 Nov-30-2018, 11:40 PM
Last Post: Truman
  email crawler in python aaa 0 533 May-18-2018, 07:03 PM
Last Post: aaa
  facebook friends crawler edithegodfather 12 8,020 Jan-15-2018, 07:07 AM
Last Post: qnkhuat
  Web crawler not returning links snakeyes22 3 946 Dec-29-2017, 11:07 PM
Last Post: Larz60+
  Python - Why multi threads are not working in this web crawler? ratanbhushan 1 745 Nov-17-2017, 05:21 PM
Last Post: Larz60+
  Web Crawler Not Working chrisdas 13 3,723 Feb-06-2017, 10:45 PM
Last Post: scriptso
  web crawler that retrieves data not stored in source code edithegodfather 14 4,045 Jan-14-2017, 01:01 AM
Last Post: edithegodfather
  Problems with moving from 3.5 to 3.6 Hellerick 7 4,938 Dec-27-2016, 04:38 PM
Last Post: Hellerick
  Web Crawler: How to find all divs that starts with... amandacstr 3 1,902 Oct-01-2016, 02:15 PM
Last Post: amandacstr

Forum Jump:


Users browsing this thread: 1 Guest(s)