Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Beautiful Soup find_all()
#1
Hi, I have problems doing webscrapping with this code from Google Search.

I am looking to get the top 10 titles from any search made in Google Search but all i get is the following:

I even tried the search string to h3 tag but code doesnt even run then.

Appreciate any help thanks.


My search results are:
Enter your search string : engineer
['engineer0 - Google Search']
['engineer1 - Google Search']
['engineer2 - Google Search']
['engineer3 - Google Search']
['engineer4 - Google Search']
['engineer5 - Google Search']
['engineer6 - Google Search']
['engineer7 - Google Search']
['engineer8 - Google Search']
['engineer9 - Google Search']

titles = soup.find_all("title")
for title in titles:
    print(title.contents)
    title_list.append(title)
Reply
#2
If you look at data you get back when using Requests and Beautiful Soup.
You will see that's is a big mess,this is because Google search use JavaScript heavily.
You will not find h3 tag or title(only for search word) in output.

I have done this task for before i 2-3 year ago for this question then did i used PhantomJS.
Now today is Selenium with Chrome and FireFox web-driver what's used.
Look at Web-scraping part-2.
Under:
Quote:God dammit JavaScript, why do i not get all content

Test my old code rewrite to use Selenium,search for python forum
I only post output,give it some effort and try to use these tool yourself.
Output:
Forums | Python.org https://www.python.org/community/forums/ --------- Python Forum https://python-forum.io/ --------- What are some active Python forums for beginners? - Quora https://www.quora.com/What-are-some-active-Python-forums-fo... --------- Python for beginner - Python - The freeCodeCamp Forum https://www.freecodecamp.org/forum/t/python-for.../169628 --------- What are the best Python forums to hang out in? : Python - Reddit https://www.reddit.com/.../Python/.../what_are_the_best_python_... --------- Python Forum | Dream.In.Code https://www.dreamincode.net/forums/forum/29-python/ --------- Python Forum - Youth4work https://www.youth4work.com › Talent › Python --------- Python Programming Help Forum | Go4Expert https://www.go4expert.com › Forums › Programming --------- Python - Raspberry Pi Forums https://www.raspberrypi.org/forums/viewforum.php?f=32 --------- Python Discussion Forum | Hackr.io https://hackr.io/forum/python
Reply
#3
Hi snippsat, thanks for the reply. I was using some chromedrive codes previously but did not know what it was for. I will look at your old post first. Thanks.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Beautiful Soup - access a rating value in a class KatMac 1 1,343 Apr-16-2021, 01:27 PM
Last Post: snippsat
  *Beginner* web scraping/Beautiful Soup help 7ken8 2 667 Jan-28-2021, 04:26 PM
Last Post: 7ken8
  Help: Beautiful Soup - Parsing HTML table ironfelix717 2 808 Oct-01-2020, 02:19 PM
Last Post: snippsat
  Beautiful Soup (suddenly) doesn't get full webpage html j.crater 8 1,966 Jul-11-2020, 04:31 PM
Last Post: j.crater
  Requests-HTML vs Beautiful Soup - How to Choose? robin73 0 1,772 Jun-23-2020, 02:53 PM
Last Post: robin73
  my soup.find_all is not finding anything: it runs into a empty-list apollo 1 2,146 May-05-2020, 07:25 PM
Last Post: snippsat
  looking for direction - scrappy, crawler, beautiful soup Sly_Corn 2 888 Mar-17-2020, 03:17 PM
Last Post: Sly_Corn
  Beautiful soup truncates results jonesjoz 4 1,373 Mar-09-2020, 06:04 PM
Last Post: jonesjoz
  Beautiful soup and tags starter_student 11 2,354 Jul-08-2019, 03:41 PM
Last Post: starter_student
  [split] Using beautiful soup to get html attribute value moski 6 3,281 Jun-03-2019, 04:24 PM
Last Post: moski

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020