Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 "I'm Feeling Lucky" script problem (again)
#1
I didn't get any replies to the other related thread; and, because one other person in that thread had the same problem as me with no answers I thought it appropriate to start a new thread.

Original question:

"Man, I'm having trouble with this project too. I'm getting no search results after passing the search terms into the run window."

My script returns this output:
"Googling...
None
Press any key to continue . . ."

My script:
#! python3
# lucky.py - Opens several Google search results.
 
import webbrowser, bs4, requests, sys
 
print('Googling...') # display text while downloading the Google page
res = requests.get('http://google.com/search?q=' + ' '.join(sys.argv[1:]))
print(res.raise_for_status())
 
# Retreive top search result links.
soup = bs4.BeautifulSoup(res.text, "html.parser")
 
# Open a browser tab for each result
linkElems = soup.select('.r a')
numOpen = min(5, len(linkElems))
for i in range(numOpen):
    webbrowser.open('http://google.com' + linkElems[i].get('href')) 
Quote
#2
print(linkElems)
# []
If you look at the response you're getting, there are no links nested under a .r class. The page that google sends you is dependent upon whether or not you have javascript enabled. So, inspect what you're getting, and narrow your selector appropriately.
Quote
#3
Hmmm. Wall

When I inspect the page I do see a .r
<div class="r"><a href="https://www.python.org/" for each link listed in the search: "python."

I added that code you posted to lines 19-20 of my script and I got the output:
Googling...
None
[]
Press any key to continue . . .

I tried just eliminating the .r class and I got the 5 browsers to open, but predictably I just got the first 5 links under the a class.

I checked in my Chrome settings and javascript was enabled

Thanks!!
Quote
#4
Look at Web-scraping part-2,under "God dammit JavaScript, why do i not get all content".
I have done this task before look at hint on this line browser.page_source.
soup = BeautifulSoup(browser.page_source, 'lxml')
If do like this will get JavaScript(how in it look Browser) and can parse out links from Google as the look when inspect in browser.
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  Project: “I’m Feeling Lucky” Google Search Truman 31 5,907 Jul-09-2019, 04:20 PM
Last Post: tab_lo_lo
  Problem With Simple Multiprocessing Script digitalmatic7 11 2,521 Apr-16-2018, 07:18 PM
Last Post: digitalmatic7

Forum Jump:


Users browsing this thread: 3 Guest(s)