Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Python Looping
#1
Hello guys...i was wondering....when you have platforms for vehicle cars like "Choose Manufacturer", next you have to choose brand, type and then do you want to do it over again to the next manufacturer/brand/type etc...how im supposed to do that?, i want to focus only on left side and ignore the rest of them just focus the make,model,engine



Here is an example of site CarParts



ps

im using python to crawl data (scrapy)

the logic here must be like :

Pick for example bmw then for brand pick like 3er series e36 and then loop all the engines, then go back to models choose the 2nd one loop the engines and over and over again and when the models have done then go back to manufacturer and pick the 2nd car and do the same thing
Reply
#2
This is a site that doesn't want to be scraped, robots.txt below:
Output:
User-agent: * Disallow: /*?list Disallow: /pdf/* Disallow: /thumb.php Disallow: /basket* Disallow: /register$ Disallow: /profile* Disallow: /battery/* Disallow: /oil/* Disallow: /terms-conditions* Disallow: /privacy-policy* Disallow: /company* Disallow: /rims/* Disallow: /tires/* Disallow: /return-of-goods* Disallow: /*?keyword Disallow: /spares-search.html
You should contact the company as they may have a way you could get access to their catalog for local copy.
Reply
#3
(Dec-08-2018, 12:53 AM)Larz60+ Wrote: This is a site that doesn't want to be scraped, robots.txt below:
Output:
User-agent: * Disallow: /*?list Disallow: /pdf/* Disallow: /thumb.php Disallow: /basket* Disallow: /register$ Disallow: /profile* Disallow: /battery/* Disallow: /oil/* Disallow: /terms-conditions* Disallow: /privacy-policy* Disallow: /company* Disallow: /rims/* Disallow: /tires/* Disallow: /return-of-goods* Disallow: /*?keyword Disallow: /spares-search.html
You should contact the company as they may have a way you could get access to their catalog for local copy.


i know....i just wanted to see what logic i have to use to get this work...i dont care about this web site...i just wanted to see how can i do that :P
Reply
#4
I would suggest snippsat's scraping tutorials on this forum
They are available here:
part1
part2
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Looping actions in an iframe using Selenium on Python amyd 3 2,966 Mar-06-2019, 08:31 PM
Last Post: Larz60+
  Unable to print data while looping through list in csv for webscraping - Python Prince_Bhatia 1 3,476 Oct-04-2017, 11:18 AM
Last Post: wavic

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020