|
Search Engine
|
|
Dec-16-2020, 09:21 AM
Can anybody tell me what are the steps to create a search engine such as google? I've read that it involves using python libraries such as beautiful soup/scrapy to crawl through different websites. And I understand that Google has an index that stores all of this information across millions of sites on a sever somewhere. Is this something that I would be able to replicate on my own?
First, build a facility and load it with servers. In a July 2016 it was reported that Google at the time had 2.5 million servers.
To start scraping data, you can go through local tutorials to get started. web scraping part 1 web scraping part 2 I do a lot of web scraping, and have 30 Tb of disk space. You can do quite a lot with that. |
|
|
| Possibly Related Threads… | |||||
| Thread | Author | Replies | Views | Last Post | |
| problem python search engine | forumpy | 0 | 1,746 |
Sep-15-2020, 10:26 PM Last Post: forumpy |
|
| Voynich search engine in python using dashes & dot totals to find Italian words | Pleiades | 3 | 3,533 |
Oct-10-2019, 10:04 PM Last Post: Pleiades |
|
| Making an application with a 3d engine in it | moo5e | 1 | 2,129 |
May-27-2019, 03:17 PM Last Post: heiner55 |
|
| Python Trading Engine | icabero0225 | 3 | 5,201 |
May-27-2017, 09:53 PM Last Post: bigmit37 |
|
