Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Searching yahoo with selenium
#1
The last time I was doing an automatization of search ( on google ) it was with bs4 and webbrowser only. Now I'm trying to use selenium first. So, I want to achieve this:
1. open a browser
2. open yahoo search page
3. search for term 'seleniumhq'
4. open first 5 results in 5 different tabs

in later stage when I improve the algorithm planning to bundle it into functions and with input() but first want my basic code to work. Help is appreciated.

code:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import bs4

browser = webdriver.Firefox()
browser.get('http://www.yahoo.com')
assert 'Yahoo' in browser.title

elem = browser.find_element_by_name('p') # find the search box
res = elem.send_keys('seleniumhq' + Keys.RETURN)

soup = bs4.BeautifulSoup(res.text, "html.parser")
linkElems = soup.select('a')
for link in linkElems:
	print(f'link: {link}')
numOpen = min(5, len(linkElems))
for i in range(numOpen):
	webbrowser.open(linkElems[i].get('href'))

browser.quit
error:
Error:
Traceback (most recent call last): File "C:\Python36\kodovi\sel4.py", line 12, in <module> soup = bs4.BeautifulSoup(res.text, "html.parser") AttributeError: 'NoneType' object has no attribute 'text'
According to this res is nontype object. Don't know why and don't know how to make it type so that code can be fully conducted ( although not sure that the rest is good ).
Reply


Messages In This Thread
Searching yahoo with selenium - by Truman - Oct-10-2018, 11:54 PM
RE: Searching yahoo with selenium - by Larz60+ - Oct-11-2018, 02:53 AM
RE: Searching yahoo with selenium - by Truman - Oct-11-2018, 09:09 PM
RE: Searching yahoo with selenium - by nilamo - Oct-11-2018, 09:20 PM
RE: Searching yahoo with selenium - by Truman - Oct-11-2018, 09:36 PM
RE: Searching yahoo with selenium - by nilamo - Oct-11-2018, 09:37 PM
RE: Searching yahoo with selenium - by Truman - Oct-11-2018, 09:57 PM
RE: Searching yahoo with selenium - by nilamo - Oct-11-2018, 10:02 PM
RE: Searching yahoo with selenium - by Truman - Oct-11-2018, 10:10 PM
RE: Searching yahoo with selenium - by snippsat - Oct-12-2018, 02:23 AM
RE: Searching yahoo with selenium - by Truman - Oct-12-2018, 09:50 PM
RE: Searching yahoo with selenium - by Larz60+ - Oct-12-2018, 11:00 PM
RE: Searching yahoo with selenium - by Truman - Oct-12-2018, 11:13 PM
RE: Searching yahoo with selenium - by metulburr - Oct-12-2018, 11:02 PM
RE: Searching yahoo with selenium - by snippsat - Oct-12-2018, 11:09 PM
RE: Searching yahoo with selenium - by snippsat - Oct-13-2018, 12:50 AM
RE: Searching yahoo with selenium - by Truman - Oct-13-2018, 09:45 PM
RE: Searching yahoo with selenium - by snippsat - Oct-13-2018, 10:44 PM
RE: Searching yahoo with selenium - by Truman - Oct-13-2018, 10:59 PM
RE: Searching yahoo with selenium - by snippsat - Oct-13-2018, 11:56 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Web Scraping with Yahoo Finance miloellison 1 2,151 Jul-03-2020, 11:12 PM
Last Post: Larz60+
  getting financial data from yahoo finance asiaphone12 7 7,266 Jun-15-2020, 05:49 AM
Last Post: mick_g
  Scrap Yahoo Finance using BS4 mr_byte31 7 6,438 Aug-24-2018, 02:50 PM
Last Post: Larz60+
  webscraping yahoo data - custom date implementation Jens89 4 5,292 Jun-19-2018, 08:02 AM
Last Post: Jens89
  Error in Selenium: CRITICAL:root:Selenium module is not installed...Exiting program. AcszE 1 3,799 Nov-03-2017, 08:41 PM
Last Post: metulburr

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020