Python Forum

I'm doing a fair amount of web scraping using generally a combination of Python, Selenium, BeautifulSoup & other libraries as needed.

The problem I'm hitting of course is IP address banning from sites after usually several days of slow scraping. I've found and used a few 3rd party IP address rotation utilities (free trial) and I've instantly been able to resume accessing/scraping sites that I'd previously been banned from due to too many hits from a single IP address.

I know there are many utilities out there to do this sort of thing, but I would love some input as to some of the more highly recommended ones that any of you have used or currently use. Many of the options out there don't seem to be terribly pricy in the grand scheme of things.

I'm wanting to find something that allows for scheduling of IP address rotation every "x" minutes or hours without human interaction to do so as I'll have several Python scripts that will be running for days, if not weeks, to scrape very large volumes of data.

Thanks for any recommendations!

What does "slow scraping" mean?  If you're going slow enough, you shouldn't be banned, as you wouldn't put any strain on the server.

I'm doing reasonably-paced scraping, not hitting page after page at a rapid speed. I've got a period of random wait times of up to 20 seconds in-between hitting each page, and every now and then, I incorporate a wait time of up to 10 min. before resuming scraping. I eventually do get banned by sites even though with this slower, respectful-speed of scraping.

Quote:random wait times up to 20 seconds

Does that include 0 seconds or .000000001 seconds?