Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 "I'm Feeling Lucky" script problem (again)
#1
I didn't get any replies to the other related thread; and, because one other person in that thread had the same problem as me with no answers I thought it appropriate to start a new thread.

Original question:

"Man, I'm having trouble with this project too. I'm getting no search results after passing the search terms into the run window."

My script returns this output:
"Googling...
None
Press any key to continue . . ."

My script:
#! python3
# lucky.py - Opens several Google search results.
 
import webbrowser, bs4, requests, sys
 
print('Googling...') # display text while downloading the Google page
res = requests.get('http://google.com/search?q=' + ' '.join(sys.argv[1:]))
print(res.raise_for_status())
 
# Retreive top search result links.
soup = bs4.BeautifulSoup(res.text, "html.parser")
 
# Open a browser tab for each result
linkElems = soup.select('.r a')
numOpen = min(5, len(linkElems))
for i in range(numOpen):
    webbrowser.open('http://google.com' + linkElems[i].get('href')) 
Quote
#2
print(linkElems)
# []
If you look at the response you're getting, there are no links nested under a .r class. The page that google sends you is dependent upon whether or not you have javascript enabled. So, inspect what you're getting, and narrow your selector appropriately.
Quote
#3
Hmmm. Wall

When I inspect the page I do see a .r
<div class="r"><a href="https://www.python.org/" for each link listed in the search: "python."

I added that code you posted to lines 19-20 of my script and I got the output:
Googling...
None
[]
Press any key to continue . . .

I tried just eliminating the .r class and I got the 5 browsers to open, but predictably I just got the first 5 links under the a class.

I checked in my Chrome settings and javascript was enabled

Thanks!!
Quote
#4
Look at Web-scraping part-2,under "God dammit JavaScript, why do i not get all content".
I have done this task before look at hint on this line browser.page_source.
soup = BeautifulSoup(browser.page_source, 'lxml')
If do like this will get JavaScript(how in it look Browser) and can parse out links from Google as the look when inspect in browser.
nilamo likes this post
Quote
#5
I don't get why the script given to the readers in the Automate the Boring Stuff just doesn't work and why the author hasn't tried to provide additional guidance? I appreciate your help help here. I'm frustrated that I'm still having problems. Ugh.

I get this message when I try to run the script with Selenium in it... This is the script from the "God dammit Javascript..." I tried to figure out how to add the Selenium piece and the browser.page_source to my script and got a similar message.
Error:
Traceback (most recent call last): File "C:\Users\lspeer\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\common\service.py", line 76, in start stdin=PIPE) File "C:\Users\lspeer\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 769, in __init__ restore_signals, start_new_session) File "C:\Users\lspeer\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 1172, in _execute_child startupinfo) FileNotFoundError: [WinError 2] The system cannot find the file specified During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:/Users/lspeer/MyPythonScripts/pwTester.py", line 6, in <module> browser = webdriver.Chrome() File "C:\Users\lspeer\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\chrome\webdriver.py", line 73, in __init__ self.service.start() File "C:\Users\lspeer\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\common\service.py", line 83, in start os.path.basename(self.path), self.start_error_message) selenium.common.exceptions.WebDriverException: Message: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home
Quote
#6
Quote: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/...river/home
You most have chromedriver.exe in your Windows Path.
Also in Environment Variables Path i have C:\cmder\bin.

Test code.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
import time

#--| Setup
chrome_options = Options()
#chrome_options.add_argument("--headless")
browser = webdriver.Chrome(executable_path=r'C:\cmder\bin\chromedriver.exe')
#--| Parse or automation
browser.get('https://duckduckgo.com')
input_field = browser.find_elements_by_css_selector('#search_form_input_homepage')
input_field[0].send_keys('car' + Keys.RETURN)
time.sleep(3)
images_link = browser.find_elements_by_link_text('Images') # Or name your country use
images_link[0].click()
time.sleep(5)
browser.quit()
Quote
#7
Bummer. I don't have admin rights to add this .exe to my PATH. I may just skip this project and move on to the next one. Thanks for your help on this one!
Quote
#8
(Jul-23-2019, 10:16 PM)tab_lo_lo Wrote: Bummer. I don't have admin rights to add this .exe to my PATH.
An other solution that work is to have chrome driver an python script in same folder.
browser = webdriver.Chrome(executable_path=r'chromedriver.exe')
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  Project: “I’m Feeling Lucky” Google Search Truman 31 6,609 Jul-09-2019, 04:20 PM
Last Post: tab_lo_lo
  Problem With Simple Multiprocessing Script digitalmatic7 11 2,717 Apr-16-2018, 07:18 PM
Last Post: digitalmatic7

Forum Jump:


Users browsing this thread: 1 Guest(s)