Python Forum
Thread Rating:
  • 1 Vote(s) - 3 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Web Crawler Not Working
#2
Have not ran your code just yet... doing some much-needed cleaning but before I reboot and do run it at a quick look, you did no close the url ... in url = ....

(Jan-24-2017, 12:54 PM)chrisdas Wrote: Hi All, Not sure why my crawler isn't working. It's pretty simply pulling out the href, the brand, and the fit of t-shirts from a website. It manages to get the fit correct but the href and the brand just loop and repeat themselves for every output. Can't find the error. Thanks, Chris 

I've had to remove the http and www from in front of 'theiconic' as it wouldn't let me post with web links.


import requests
from bs4 import BeautifulSoup



def iconic_spider(max_pages):
    page = 1
    while page <= max_pages:
        url = theiconic.com.au/mens-clothing-tshirts-singlets/?page=' + str(page)
        source_code = requests.get(url)
        plain_text = source_code.text
        soup = BeautifulSoup(plain_text, "html.parser")
        for link in soup.findAll('a', {'class': 'product-details'}):
            href = theiconic.com.au/' + link.get('href')
        for link in soup.findAll('span', {'class': 'brand'}):
            brand = link.string
        for link in soup.findAll('span', {'class': 'name'}):
            fit = link.string
            print(href)
            print(brand)
            print(fit)
        page += 1

iconic_spider(2)
OKAY! I got your code to work with a couple edit... simple mistakes really...  But before I point them out I would ask you to run your script and read the error, 97% it says the underlying immediate error in code in my experience at the very begging or end of the stack trace...
Reply


Messages In This Thread
Web Crawler Not Working - by chrisdas - Jan-24-2017, 12:54 PM
RE: Web Crawler Not Working - by scriptso - Feb-02-2017, 11:06 PM
RE: Web Crawler Not Working - by wavic - Feb-02-2017, 11:23 PM
RE: Web Crawler Not Working - by scriptso - Feb-02-2017, 11:25 PM
RE: Web Crawler Not Working - by wavic - Feb-03-2017, 12:39 AM
RE: Web Crawler Not Working - by scriptso - Feb-03-2017, 01:18 AM
RE: Web Crawler Not Working - by wavic - Feb-03-2017, 01:34 AM
RE: Web Crawler Not Working - by scriptso - Feb-03-2017, 01:47 AM
RE: Web Crawler Not Working - by wavic - Feb-03-2017, 03:33 AM
RE: Web Crawler Not Working - by snippsat - Feb-03-2017, 05:59 AM
RE: Web Crawler Not Working - by wavic - Feb-03-2017, 08:43 AM
RE: Web Crawler Not Working - by snippsat - Feb-03-2017, 09:21 AM
RE: Web Crawler Not Working - by wavic - Feb-03-2017, 09:48 AM
RE: Web Crawler Not Working - by scriptso - Feb-06-2017, 10:45 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Web Crawler help Mr_Mafia 2 2,054 Apr-04-2020, 07:20 PM
Last Post: Mr_Mafia
  Web Crawler help takaa 39 28,263 Apr-26-2019, 12:14 PM
Last Post: stateitreal
  Python - Why multi threads are not working in this web crawler? ratanbhushan 1 2,894 Nov-17-2017, 05:21 PM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020