[split] Web Crawler Error - Printable Version

[split] Web Crawler Error - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: [split] Web Crawler Error (/thread-2079.html)

[split] Web Crawler Error - takaa - Feb-17-2017

Today I am all off the sudden getting this error message after the crawler made its way through a few pages.

Error:    raise ConnectionError(err, request=request)

requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

Is this possibly that the website is blocking something? It seems to randomly happen. Most of the time i am able to extract everything without problem.

RE: Web Crawler help - wavic - Feb-17-2017

A website can block an IP if there is more than some number of requests per second. Print the response code.