Nice that you fixed the code @scriptso.
Some formatting to better see the data.
(Feb-03-2017, 03:33 AM)wavic Wrote: This is strange. I am unable to get even product-details classIt do work.
Some formatting to better see the data.
_author_ = 'Erick' import requests from bs4 import BeautifulSoup def iconic_spider(max_pages): page = 1 print('******* page 1 ********') while page <= max_pages: url = 'http://theiconic.com.au/mens-clothing-tshirts-singlets/?page={}'.format(page) source_code = requests.get(url) plain_text = source_code.text soup = BeautifulSoup(plain_text, "html.parser") for link in soup.findAll('a', {'class': 'product-details'}): href = 'theiconic.com.au/' + link.get('href') for link in soup.findAll('span', {'class': 'brand'}): brand = link.string for link in soup.findAll('span', {'class': 'name'}): fit = link.string print('-----------') print(href) print(brand) print(fit) print('******* page {} ********'.format(page+1)) page += 1 if __name__ == '__main__': pages = 2 iconic_spider(pages)
Output:******* page 1 ********
-----------
theiconic.com.au//basic-crew-neck-pima-tee-363464.html
Lacoste
90s Short Sleeve T-Shirt
-----------
theiconic.com.au//basic-crew-neck-pima-tee-363464.html
Lacoste
LA Skull Tee
-----------
theiconic.com.au//basic-crew-neck-pima-tee-363464.html
Lacoste
The Original Print Tee
-----------
theiconic.com.au//basic-crew-neck-pima-tee-363464.html
Lacoste
Men's Zonal Cooling Relay SS Tee
-----------
.......... etc
******* page 2 ********
-----------
theiconic.com.au//venice-address-tee-199234.html
Deus Ex Machina
Basic Crew-Neck Pima Tee
-----------
theiconic.com.au//venice-address-tee-199234.html
Deus Ex Machina
Crawley Tee
.......... etc
******* page 3 ********