Python Forum
How to use BeautifulSoup to parse google search results
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How to use BeautifulSoup to parse google search results
#1
I am trying to parse the first page of google search results. Specifically, the Title and the small Summary that is provided. Here is what I have so far:

from urllib.request import urlretrieve
import urllib.parse
from urllib.parse import urlencode, urlparse, parse_qs
import webbrowser
from bs4 import BeautifulSoup
import requests

address = 'https://google.com/#q='
# Default Google search address start
file = open( "OCR.txt", "rt" )
# Open text document that contains the question
word = file.read()
file.close()

myList = [item for item in word.split('\n')]
newString = ' '.join(myList)
# The question is on multiple lines so this joins them together with proper spacing

print(newString)

qstr = urllib.parse.quote_plus(newString)
# Encode the string

newWord = address + qstr
# Combine the base and the encoded query

print(newWord)

source = requests.get(newWord)

soup = BeautifulSoup(source.text, 'lxml')
The part I am stuck on now is going down the HTML path to parse the specific data that I want. Everything I have tried so far has just thrown an error saying that it has no attribute or it just gives back "[]".

I am new to Python and BeautifulSoup so I am not sure the syntax of how to get to where I want. I have found that these are the individual search results in the page:

https://ibb.co/jfRakR

Any help on what to add to parse the Title and Summary of each search result would be MASSIVELY appreciated.

Thank you!
Reply


Messages In This Thread
How to use BeautifulSoup to parse google search results - by DevinGP - Dec-21-2017, 04:25 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Unable to convert browser generated xml to parse in BeautifulSoup Nik1811 0 367 Mar-22-2024, 01:37 PM
Last Post: Nik1811
  Using BeautifulSoup And Getting -1 Results knight2000 10 3,121 Mar-07-2023, 02:42 PM
Last Post: snippsat
  Web scraping for search results JOE 7 3,409 May-14-2022, 01:19 PM
Last Post: JOE
  With Selenium create a google Search list in Incognito mode withe specific location, tsurubaso 3 3,331 Jun-15-2020, 12:34 PM
Last Post: tsurubaso
  Wrong number of google results in a date range Val 0 1,893 Mar-15-2020, 02:29 PM
Last Post: Val
  Project: “I’m Feeling Lucky” Google Search Truman 31 28,846 Jul-09-2019, 04:20 PM
Last Post: tab_lo_lo
  Outputing the results of search machine Emmanouil 4 5,128 Nov-07-2016, 05:20 PM
Last Post: nilamo

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020