I didn't get any replies to the other related thread; and, because one other person in that thread had the same problem as me with no answers I thought it appropriate to start a new thread.
Original question:
"Man, I'm having trouble with this project too. I'm getting no search results after passing the search terms into the run window."
My script returns this output:
"Googling...
None
Press any key to continue . . ."
My script:
#! python3
# lucky.py - Opens several Google search results.
import webbrowser, bs4, requests, sys
print('Googling...') # display text while downloading the Google page
res = requests.get('http://google.com/search?q=' + ' '.join(sys.argv[1:]))
print(res.raise_for_status())
# Retreive top search result links.
soup = bs4.BeautifulSoup(res.text, "html.parser")
# Open a browser tab for each result
linkElems = soup.select('.r a')
numOpen = min(5, len(linkElems))
for i in range(numOpen):
webbrowser.open('http://google.com' + linkElems[i].get('href'))
print(linkElems)
# []
If you look at the response you're getting, there are no links nested under a .r class. The page that google sends you is dependent upon whether or not you have javascript enabled. So, inspect what you're getting, and narrow your selector appropriately.
Hmmm.
When I inspect the page I do see a .r
<div class="r"><a href="https://www.python.org/" for each link listed in the search: "python."
I added that code you posted to lines 19-20 of my script and I got the output:
Googling...
None
[]
Press any key to continue . . .
I tried just eliminating the .r class and I got the 5 browsers to open, but predictably I just got the first 5 links under the a class.
I checked in my Chrome settings and javascript was enabled
Thanks!!
Look at
Web-scraping part-2,under "God dammit JavaScript, why do i not get all content".
I have done this task before look at hint on this line
browser.page_source
.
soup = BeautifulSoup(browser.page_source, 'lxml')
If do like this will get JavaScript(how in it look Browser) and can parse out links from Google as the look when inspect in browser.
I don't get why the script given to the readers in the Automate the Boring Stuff just doesn't work and why the author hasn't tried to provide additional guidance? I appreciate your help help here. I'm frustrated that I'm still having problems. Ugh.
I get this message when I try to run the script with Selenium in it... This is the script from the "God dammit Javascript..." I tried to figure out how to add the Selenium piece and the browser.page_source to my script and got a similar message.
Error:
Traceback (most recent call last):
File "C:\Users\lspeer\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\common\service.py", line 76, in start
stdin=PIPE)
File "C:\Users\lspeer\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 769, in __init__
restore_signals, start_new_session)
File "C:\Users\lspeer\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 1172, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Users/lspeer/MyPythonScripts/pwTester.py", line 6, in <module>
browser = webdriver.Chrome()
File "C:\Users\lspeer\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\chrome\webdriver.py", line 73, in __init__
self.service.start()
File "C:\Users\lspeer\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\common\service.py", line 83, in start
os.path.basename(self.path), self.start_error_message)
selenium.common.exceptions.WebDriverException: Message: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home
Quote: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/...river/home
You most have
chromedriver.exe
in your Windows Path.
Also in
Environment Variables Path i have
C:\cmder\bin
.
Test code.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
import time
#--| Setup
chrome_options = Options()
#chrome_options.add_argument("--headless")
browser = webdriver.Chrome(executable_path=r'C:\cmder\bin\chromedriver.exe')
#--| Parse or automation
browser.get('https://duckduckgo.com')
input_field = browser.find_elements_by_css_selector('#search_form_input_homepage')
input_field[0].send_keys('car' + Keys.RETURN)
time.sleep(3)
images_link = browser.find_elements_by_link_text('Images') # Or name your country use
images_link[0].click()
time.sleep(5)
browser.quit()
Bummer. I don't have admin rights to add this .exe to my PATH. I may just skip this project and move on to the next one. Thanks for your help on this one!
(Jul-23-2019, 10:16 PM)tab_lo_lo Wrote: [ -> ]Bummer. I don't have admin rights to add this .exe to my PATH.
An other solution that work is to have chrome driver an python script in same folder.
browser = webdriver.Chrome(executable_path=r'chromedriver.exe')