Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Error with crawling site.
#3
Yes. HEre is the code.
http_proxy = "xxxxx"


proxyDict = {
"http" : http_proxy
}
headers={'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
for i in range(20):
r = None
try:
r = requests.get(base_url, proxies=proxyDict, headers=headers)
data = r.text
except Exception as e:
print('Error getting requests')
print(e.__doc__)
print(e.message)

sleep(20)

I used fake_useragent as well..
same error message.
Reply


Messages In This Thread
Error with crawling site. - by jj2110 - Oct-10-2017, 04:11 PM
RE: Error with crawling site. - by buran - Oct-10-2017, 04:16 PM
RE: Error with crawling site. - by jj2110 - Oct-10-2017, 04:49 PM
RE: Error with crawling site. - by wavic - Oct-10-2017, 06:36 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  need help with crawling FB likes najla3 2 2,720 Jan-09-2018, 11:15 AM
Last Post: najla3

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020