Python Forum
Project: “I’m Feeling Lucky” Google Search - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html)
+--- Thread: Project: “I’m Feeling Lucky” Google Search (/thread-11674.html)

Pages: 1 2 3 4


Project: “I’m Feeling Lucky” Google Search - Truman - Jul-20-2018

This is web scraping project from Automate the Boring Stuff with Python. From some reason solution doesn't work and I have some doubts myself. The goal is to type a search term on the command line and have my computer automatically open a browser with all the top search results in new tabs.

#! python3
# lucky.py - Opens several Google search results.

import webbrowser, bs4, requests, sys

print('Googling...') # display text while downloading the Google page
res = requests.get('http://google.com/search?source=' + ' '.join(sys.argv[1:]))
print(res.raise_for_status())

# Retreive top search result links.
soup = bs4.BeautifulSoup(res.text, "html.parser")

# Open a browser tab for each result
linkElems = soup.select('.r a')
numOpen = min(5, len(linkElems))
for i in range(numOpen):
	webbrowser.open('http://google.com' + linkElems[i].get('href'))
I'm also not sure how to start this program. Should I type a search term in command prompt where I usually start my python files?


RE: Project: “I’m Feeling Lucky” Google Search - Larz60+ - Jul-21-2018

Quote:I'm also not sure how to start this program
you can create a function, for example named test_it() which executes a test case, then
start that with (untested):
#! python3
# lucky.py - Opens several Google search results.
 
import webbrowser
import bs4
import requests
import sys

def my_func(search_str) :
    print('Googling...') # display text while downloading the Google page
    res = requests.get('http://google.com/search?source=' + ' '.join(search_str))
    print(res.raise_for_status())
 
    # Retreive top search result links.
    soup = bs4.BeautifulSoup(res.text, "html.parser")
 
    # Open a browser tab for each result
    linkElems = soup.select('.r a')
    numOpen = min(5, len(linkElems))
    for i in range(numOpen):
        webbrowser.open('http://google.com' + linkElems[i].get('href'))

def test_it():
    my_func(sys.argv[1:])

if __name__ == '__main__':
    test_it()
this code will only run if the program is started from within an IDE, or from the command line, but will not run if your main code is called from another program.


RE: Project: “I’m Feeling Lucky” Google Search - Truman - Jul-21-2018

Error:
Traceback (most recent call last): File "C:\Python36\kodovi\lucky.py", line 4, in <module> import webbrowser, bs4, requests, sys ModuleNotFoundError: No module named 'bs4' >>> %Run lucky1.py Traceback (most recent call last): File "C:\Python36\kodovi\lucky1.py", line 5, in <module> import bs4 ModuleNotFoundError: No module named 'bs4'
Now I just feel completely lost. lucky is my code filename and I named lucky1 what you Larz wrote. I used Thonny IDE.
I installed bs4 of course.

and when I run code from Notepad++ with command prompt I get this:
Output:
C:\Python36\kodovi>lucky novak djokovic Googling... None
my search term in command line is "novak djokovic" and it's not giving any result obviously.
I'll try to find solution on stackoverflow or smth like that.


RE: Project: “I’m Feeling Lucky” Google Search - Larz60+ - Jul-22-2018

whenever you see: ModuleNotFoundError: No module named '...', you need to install a package.

In this case, you need to install BeautifulSoup4:

first test to see that pip version is correct one:
pip -V
You want to see something like:

Output:
pip 10.0.1 from ... (python 3.6)
if you don't see proper python version, substitute pip3 for pip below

pip install BeautifulSoup4



RE: Project: “I’m Feeling Lucky” Google Search - Truman - Jul-22-2018

Output:
C:\Users\user>pip -V pip 10.0.1 from c:\python36\lib\site-packages\pip (python 3.6) C:\Users\user>pip3 install BeautifulSoup4 Requirement already satisfied: BeautifulSoup4 in c:\python36\lib\site-packages ( 4.6.0)
this is what I get...From some reason I still receive the same Traceback error with Thonny and the same output with command prompt.


RE: Project: “I’m Feeling Lucky” Google Search - Larz60+ - Jul-22-2018

that's good, that means you can use:
pip install BeautifulSoup4

I see it's already installed.
Then make the following changes on your code (by line number)
In reverse order (so line numbers remain constant until done)
line 15
# from:
    soup = bs4.BeautifulSoup(res.text, "html.parser")
# to:
    soup = BeautifulSoup(res.text, "html.parser")
line 5
# From:
import bs4
# to:
from bs4 import BeautifulSoup



RE: Project: “I’m Feeling Lucky” Google Search - Truman - Jul-22-2018

I edited my last post in the meantime. Now the question is how to start this program.


RE: Project: “I’m Feeling Lucky” Google Search - Larz60+ - Jul-22-2018

run from command line
python myprogramname.py



RE: Project: “I’m Feeling Lucky” Google Search - Truman - Jul-22-2018

Output:
C:\Python36\kodovi>python lucky1.py Googling... None
The point of this program, if I understand it correctly, is to add search word in command line and immediately get 5 open tasks with first 5 search results. I tried with adding search words too after filename but the outcome is the same.


RE: Project: “I’m Feeling Lucky” Google Search - Larz60+ - Jul-22-2018

It appears to have run
you are getting none from:
print(res.raise_for_status())
after line 18, add:
for link in linkElems:
    print(f'link: {link}')
so you can see if the soup.select (which is css select by the way) is finding anything

also, you need to supply a value to the command line e.g.
python -m myprog.py searchvalue