Python Forum

Full Version: Is it possible to scrape this data from Google Searches
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Hi there,

I'm currently working on a repetitive task, which I'm wondering if I can automate with Python. I have never worked with Python before, so please forgive any part of my question being unclear/not asked correctly. So to my question: I have a list in Excel with about 700 University Names and need to find their websites. Through my own research, I've found that it is possible to crawl Google Search results with Python. However, I've found that the university's website is often not the first search result. Yet, for most Universities, Google displays a box on the right, with a 'Website' Button linking to the university's website (see picture). Is it possible to use Python to automize the process of extracting all these URL's and how would I best go about doing so?

Best,

rosjo

Screenshot-2020-11-05-at-17-43-40
There are many python packages that will allow you to search, get back a bunch of results, which you can then process, drilling down to the precise item. With school names, it should be a relatively easy task.

Here are some prospect packages: https://pypi.org/search/?q=google