Hi
I need to lunch firefox
with the module anonymous. The following code lunchs firefox without the module anonymous. How can I do it?
regards
from selenium import webdriver
import webbrowser
import requests
from bs4 import BeautifulSoup
from urllib.request import urlopen
browser = webdriver.Firefox()
url = 'http://want-to-go-in/'
browser.get(url)
soup = BeautifulSoup(urlopen(url), 'html.parser')
images = soup.findAll('img')
I guess you mean headless as it called.
from selenium.webdriver.firefox.options import Options
#-- Setup
options = Options()
options.add_argument("--headless")
browser = webdriver.Firefox(firefox_options=options, executable_path=r"path to geckodriver")
#-- Parse
browser.get('url')
title = browser.find_element_by_xpath('//title')
print(t.text)
#browser.quit()So code over use selenium to parse,if use BeautifulSoup to parse you send code from selenium,you don't use
urlopen(url),
but
browser.page_source.
Otherwise there is no point to use Selenium(which should be used if need eg JavaScript,otherwise not).
# Give source code to BeautifulSoup
soup = BeautifulSoup(browser.page_source, 'lxml')
welcome = soup.select('head > title')
print(welcome.text)
No luck my friend
Here is my problem
I can access one website only with firefox and anonymous extension actived otherwise I got blocked with <script src='https://www.google.com/recaptcha/api.js'></script>
If I use the following program I can't reach the website and I got
Error:
Traceback (most recent call last):
File "C:/Users/ASUS PC/AppData/Local/Programs/Python/Python36/gg.py", line 14, in <module>
print(welcome.text)
AttributeError: 'list' object has no attribute 'text'
Here is my code:
import requests
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
#-- Setup
options = Options()
options.add_argument("--headless")
browser = webdriver.Firefox(firefox_options=options)
#-- Parse
browser.get('http://I want to enter')
soup = BeautifulSoup(browser.page_source, 'lxml')
welcome = soup.select('head > title')
print(welcome.text)
No luck with
welcome = soup.select('head > title')
print(welcome[0].text)either. Got error
Error:
Traceback (most recent call last):
File "C:/Users/ASUS PC/AppData/Local/Programs/Python/Python36/gg.py", line 14, in <module>
print(welcome[0].text)
IndexError: list index out of range
(Feb-10-2018, 08:54 PM)wavic Wrote: [ -> ]He means in private mode.
Not sure about that,i think he use a 3 party extension.
mariolopes Wrote:I can access one website only with firefox and anonymous extension actived
Every time you start FireFox through selenium webdriver it creates a brand new anonymous profile,
so you are actually browsing privately.
Can also force it headless with.
chrome_options.add_argument("--incognito")mariolopes Wrote:either. Got error
Can not use the exact simple XPath/CSS selector examples as i use.
You have to look at source code of site of are parsing and find correct ones for what you want to parse.
mariolopes Wrote:anonymous extension actived
What extension is this?
with the fpollowing extension installed on forefox I can access the website
https://addons.mozilla.org/pt-PT/firefox.../anonymox/
Everything works fine.
When I run my program it launches firefox
without the anonymous extension therefore I can't access the website.
In this case the website gets my ip and block the access it with the <script src='https://www.google.com/recaptcha/api.js'></script> (I think this is the script that blocks my ip). The captcha allways work in cycle and I can't access to website.
In this case the following code:
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
#--setup
options=Options()
options.add_argument("--headless")
import webbrowser
import requests
from bs4 import BeautifulSoup
from urllib.request import urlopen
browser = webdriver.Firefox(firefox_options=options)
url = 'http://let me in please/'
browser.get(url)I got the error
Error:
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: Reached error page: about:neterror?e=redirectLoop&u=http%3A//[b]website I want[/b]
Therefore the Firefox doesn't lunch the anonymous extension and I got the redirectLoop