Python Forum
sfscrape fails when repetitively executed
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
sfscrape fails when repetitively executed
#1
Hey all!
I got to web scraping a site via the cfscrape mod. in a py file.
I am getting 10 responses but even after sleeping per each request a bit, and even setting
timeouts for the url etc, I can't get past the 10 requests for a script run. I mean if I continually to
manually run the script it will give me my new 10 requests write away but it will take forever that
way to scrape all the data I would like.

Now when I run like a shell script to run the py file to scrape where I include
a loop to retry the scraping py file for another 10 requests, though it hoses after like 1 or 2 requests.
I mean a pull the next two pages I want but that's it. I even include extra sleep time between script runs (see below)
but still I get muffed on the second and subsequent runs I would assume.

It is a https site btw.

Thanks so much for any help!!!
!the pyguy :)


#!/bin/bash

counter=0
while [ $counter -le 1 ]
do
echo $counter
str=$(printf "%04d" $counter)
python3 wsMain.py $str

((counter+=1))
python3 runit.py $str

done
echo All done
Reply
#2
maybe you are not providing enough human delay time between running the scripts to give away the fact of it being a bot, or maybe wsMain.py is not completed before runit.py starts? If this is the case i would use the subprocess module and rewrite it in python.
Recommended Tutorials:
Reply


Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020