Python Forum

Hey all!
I got to web scraping a site via the cfscrape mod. in a py file.
I am getting 10 responses but even after sleeping per each request a bit, and even setting
timeouts for the url etc, I can't get past the 10 requests for a script run. I mean if I continually to
manually run the script it will give me my new 10 requests write away but it will take forever that
way to scrape all the data I would like.

Now when I run like a shell script to run the py file to scrape where I include
a loop to retry the scraping py file for another 10 requests, though it hoses after like 1 or 2 requests.
I mean a pull the next two pages I want but that's it. I even include extra sleep time between script runs (see below)
but still I get muffed on the second and subsequent runs I would assume.

It is a https site btw.

Thanks so much for any help!!!
!the pyguy :)

#!/bin/bash

counter=0
while [ $counter -le 1 ]
do
echo $counter
str=$(printf "%04d" $counter)
python3 wsMain.py $str

((counter+=1))
python3 runit.py $str

done
echo All done

maybe you are not providing enough human delay time between running the scripts to give away the fact of it being a bot, or maybe wsMain.py is not completed before runit.py starts? If this is the case i would use the subprocess module and rewrite it in python.