Python Forum

Hello guys...i was wondering....when you have platforms for vehicle cars like "Choose Manufacturer", next you have to choose brand, type and then do you want to do it over again to the next manufacturer/brand/type etc...how im supposed to do that?, i want to focus only on left side and ignore the rest of them just focus the make,model,engine



Here is an example of site CarParts



ps

im using python to crawl data (scrapy)

the logic here must be like :

Pick for example bmw then for brand pick like 3er series e36 and then loop all the engines, then go back to models choose the 2nd one loop the engines and over and over again and when the models have done then go back to manufacturer and pick the 2nd car and do the same thing

This is a site that doesn't want to be scraped, robots.txt below:

Output:
User-agent: *
Disallow: /*?list
Disallow: /pdf/*
Disallow: /thumb.php
Disallow: /basket*
Disallow: /register$
Disallow: /profile*
Disallow: /battery/*
Disallow: /oil/*
Disallow: /terms-conditions*
Disallow: /privacy-policy*
Disallow: /company*
Disallow: /rims/*
Disallow: /tires/*
Disallow: /return-of-goods*
Disallow: /*?keyword
Disallow: /spares-search.html

You should contact the company as they may have a way you could get access to their catalog for local copy.

(Dec-08-2018, 12:53 AM)Larz60+ Wrote: [ -> ]This is a site that doesn't want to be scraped, robots.txt below:

Output:
 User-agent: * Disallow: /*?list Disallow: /pdf/* Disallow: /thumb.php Disallow: /basket* Disallow: /register$ Disallow: /profile* Disallow: /battery/* Disallow: /oil/* Disallow: /terms-conditions* Disallow: /privacy-policy* Disallow: /company* Disallow: /rims/* Disallow: /tires/* Disallow: /return-of-goods* Disallow: /*?keyword Disallow: /spares-search.html 

You should contact the company as they may have a way you could get access to their catalog for local copy.

i know....i just wanted to see what logic i have to use to get this work...i dont care about this web site...i just wanted to see how can i do that :P

I would suggest snippsat's scraping tutorials on this forum
They are available here:
part1
part2