Python Forum

Full Version: Web Crawler's Crawling Ability
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
How can I figure out how much a web crawler actually crawls on a website?
Is it your own web crawler or what?
This is crazy question. What are you asking?

As above, is it your own crawler? If so its defined and limited by your ability to build it/variables in the program.

If its not your own, does the source have documentation?

What do you want to accomplish? Give us more info! :-)
You can check and crawled website using website crawler tool Sitechecker Website Сrawler: Monitor Your Website Health Online. Click here to know more about about web development