Python Forum
Thread Rating:
  • 1 Vote(s) - 3 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Web Crawler Not Working
#10
Nice that you fixed the code @scriptso.
(Feb-03-2017, 03:33 AM)wavic Wrote: This is strange. I am unable to get even product-details class
It do work.
Some formatting to better see the data.
_author_ = 'Erick'
import requests
from bs4 import BeautifulSoup

def iconic_spider(max_pages):
  page = 1
  print('******* page 1 ********')
  while page <= max_pages:
      url = 'http://theiconic.com.au/mens-clothing-tshirts-singlets/?page={}'.format(page)
      source_code = requests.get(url)
      plain_text = source_code.text
      soup = BeautifulSoup(plain_text, "html.parser")
      for link in soup.findAll('a', {'class': 'product-details'}):
          href = 'theiconic.com.au/' + link.get('href')
      for link in soup.findAll('span', {'class': 'brand'}):
          brand = link.string
      for link in soup.findAll('span', {'class': 'name'}):
          fit = link.string
          print('-----------')
          print(href)
          print(brand)
          print(fit)
      print('******* page {} ********'.format(page+1))
      page += 1

if __name__ == '__main__':
   pages = 2
   iconic_spider(pages)
Output:
******* page 1 ******** ----------- theiconic.com.au//basic-crew-neck-pima-tee-363464.html Lacoste 90s Short Sleeve T-Shirt ----------- theiconic.com.au//basic-crew-neck-pima-tee-363464.html Lacoste LA Skull Tee ----------- theiconic.com.au//basic-crew-neck-pima-tee-363464.html Lacoste The Original Print Tee ----------- theiconic.com.au//basic-crew-neck-pima-tee-363464.html Lacoste Men's Zonal Cooling Relay SS Tee ----------- .......... etc ******* page 2 ******** ----------- theiconic.com.au//venice-address-tee-199234.html Deus Ex Machina Basic Crew-Neck Pima Tee ----------- theiconic.com.au//venice-address-tee-199234.html Deus Ex Machina Crawley Tee .......... etc ******* page 3 ********   
Reply


Messages In This Thread
Web Crawler Not Working - by chrisdas - Jan-24-2017, 12:54 PM
RE: Web Crawler Not Working - by scriptso - Feb-02-2017, 11:06 PM
RE: Web Crawler Not Working - by wavic - Feb-02-2017, 11:23 PM
RE: Web Crawler Not Working - by scriptso - Feb-02-2017, 11:25 PM
RE: Web Crawler Not Working - by wavic - Feb-03-2017, 12:39 AM
RE: Web Crawler Not Working - by scriptso - Feb-03-2017, 01:18 AM
RE: Web Crawler Not Working - by wavic - Feb-03-2017, 01:34 AM
RE: Web Crawler Not Working - by scriptso - Feb-03-2017, 01:47 AM
RE: Web Crawler Not Working - by wavic - Feb-03-2017, 03:33 AM
RE: Web Crawler Not Working - by snippsat - Feb-03-2017, 05:59 AM
RE: Web Crawler Not Working - by wavic - Feb-03-2017, 08:43 AM
RE: Web Crawler Not Working - by snippsat - Feb-03-2017, 09:21 AM
RE: Web Crawler Not Working - by wavic - Feb-03-2017, 09:48 AM
RE: Web Crawler Not Working - by scriptso - Feb-06-2017, 10:45 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Web Crawler help Mr_Mafia 2 2,054 Apr-04-2020, 07:20 PM
Last Post: Mr_Mafia
  Web Crawler help takaa 39 28,263 Apr-26-2019, 12:14 PM
Last Post: stateitreal
  Python - Why multi threads are not working in this web crawler? ratanbhushan 1 2,894 Nov-17-2017, 05:21 PM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020