Dec-15-2017, 06:09 PM
What is considered a part of the "entire web"? Not everything can be indexed, or crawled, and not everything is accessible over http.
You really only need the
You really only need the
requests module to get a page. Finding all links in that page would be easier with beautifulsoup (the package name is bs4). And unless you have infinite time, you probably want to store an indexed version of the page in some way, using some sort of database, which would be another package.