Sep-15-2018, 04:27 PM
I am using the Princeton Review Top 384 Colleges as a source in a research project. Ultimately, what I want to do is create a code that will go to the websites of all of these colleges and create a database of professors listed by subject. For now, I'm concerned with writing the portion of the code that will find the URLs of each of the university websites. I am new to Python but have been reading about requests, urllib, BeautifulSoup, etc. But all the questions and tutorials I have found so far focus on requesting data from sites (one site or a short list of sites) where the URLs are already known. Obviously, it will take me hours to find the websites of each of the 384 colleges separately and put them into a list. I am trying to avoid that. Advice?