Python Forum
How to use BeautifulSoup to parse google search results
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How to use BeautifulSoup to parse google search results
#17
An other one one,i drop to do the search with Chrome/Phantom and use  search?q=
Have added next page search Google_Search('python forum', page=1).
from selenium import webdriver
from bs4 import BeautifulSoup

class Google_Search:
    def __init__(self, search, page=0):
        self.search = search
        self.page = page
        self.url = f'https://www.google.com/search?q={self.search}\
                    &ei=m3w9WuyXNJHMwALelovwAQ&start={str(self.page)+"0"}&sa=N&biw=848&bih=972'
        self.result()

    def result(self):
        browser = webdriver.PhantomJS()
        browser.get(self.url)
        soup = BeautifulSoup(browser.page_source, 'lxml')
        name_link = soup.find_all('h3', class_='r')
        link = soup.find_all('cite')
        for n_link, l in zip(name_link,link):
            print(f'{n_link.text}\n{l.text}')
            print('---------')

if __name__ == '__main__':
    Google_Search('python forum')
    #Google_Search('python forum', page=1)
Output:
Forums | Python.org https://www.python.org/community/forums/ --------- Python Forum https://python-forum.io/ --------- What are the best Python forums to hang out in? : Python - Reddit https://www.reddit.com/.../Python/.../what_are_the_best_python_forums_to_ hang_out_in/ --------- Python Forum | Dream.In.Code www.dreamincode.net/forums/forum/29-python/ --------- Python For Beginners Forum | Codecademy https://www.codecademy.com/en/forums/python-for-beginners --------- Python Syntax Forum | Codecademy https://www.codecademy.com/.../forums/introduction-to-python-6WeG3 --------- Nytt Norsk Python Forum - Scriptingspråk (Python, Perl, Ruby o.l ... https://www.diskusjon.no/index.php?showtopic=828151 --------- Python Programming - Dev Shed Forums forums.devshed.com/python-programming-11/ --------- Python - Raspberry Pi Forums https://www.raspberrypi.org/forums/viewforum.php?f=32 --------- Python - thenewboston Forum https://thenewboston.com/forum/category.php?id=15 ---------
Reply


Messages In This Thread
RE: How to use BeautifulSoup to parse google search results - by snippsat - Dec-22-2017, 10:23 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Unable to convert browser generated xml to parse in BeautifulSoup Nik1811 0 290 Mar-22-2024, 01:37 PM
Last Post: Nik1811
  Using BeautifulSoup And Getting -1 Results knight2000 10 3,025 Mar-07-2023, 02:42 PM
Last Post: snippsat
  Web scraping for search results JOE 7 3,306 May-14-2022, 01:19 PM
Last Post: JOE
  With Selenium create a google Search list in Incognito mode withe specific location, tsurubaso 3 3,288 Jun-15-2020, 12:34 PM
Last Post: tsurubaso
  Wrong number of google results in a date range Val 0 1,876 Mar-15-2020, 02:29 PM
Last Post: Val
  Project: “I’m Feeling Lucky” Google Search Truman 31 28,615 Jul-09-2019, 04:20 PM
Last Post: tab_lo_lo
  Outputing the results of search machine Emmanouil 4 5,091 Nov-07-2016, 05:20 PM
Last Post: nilamo

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020