Python Looping

[Larz60+]

This is a site that doesn't want to be scraped, robots.txt below:

Output:
User-agent: *
Disallow: /*?list
Disallow: /pdf/*
Disallow: /thumb.php
Disallow: /basket*
Disallow: /register$
Disallow: /profile*
Disallow: /battery/*
Disallow: /oil/*
Disallow: /terms-conditions*
Disallow: /privacy-policy*
Disallow: /company*
Disallow: /rims/*
Disallow: /tires/*
Disallow: /return-of-goods*
Disallow: /*?keyword
Disallow: /spares-search.html

You should contact the company as they may have a way you could get access to their catalog for local copy.