Python Forum
how to add proxy authentication selenium
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
how to add proxy authentication selenium
#1
I did lot of google on this topics and also tried lot of solution from stack overflow but none of them didn't work. Currently I am using paid proxy so that I can avoid block during web scraping. My proxy needs authentication by user name and password with proxy port. Assume my proxy user_name:"x",proxy_password:"abc",proxy_server: "abcdfr.com" port_port:80 how I will use them in selenium
Reply
#2
(Sep-10-2020, 01:56 PM)farhan275 Wrote: and also tried lot of solution from stack overflow
maybe you should show what you have tried and not worked so that people here don't waste time to suggest the same.
post some code that should work, but did not and what you get as error
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply
#3
here is my full code
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from bs4 import BeautifulSoup
import time

#argument for incognito Chrome
option = Options()
option.add_argument("--incognito")


browser = webdriver.Chrome(options=option)

for page_num in range(1,20): # change the range as you want to
    url = "https://www.usine-digitale.fr/annuaire-start-up/?page={}".format(page_num)
    browser.get(url)
    print("page url : ",url)
    time.sleep(3)
    # Wait 20 seconds for page to load
    timeout = 20
    try:
      WebDriverWait(browser, timeout).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='texteContenu3']")))
    except TimeoutException:
      print("Timed out waiting for page to load")
      browser.quit()

    soup = BeautifulSoup(browser.page_source, "html.parser")
    product_items = soup.find_all("a",{"class":"contenu"})
    for item in product_items:
      item_url = item.get('href')
      print("https://www.usine-digitale.fr"+item_url)
      browser.get("https://www.usine-digitale.fr"+item_url)
      time.sleep(3)
      itm_soup = BeautifulSoup(browser.page_source, "html.parser")
      container = itm_soup.find_all("div",{"class":"contenuPage"})
      for contain in container:
        name_element = contain.find("h1", class_="titreFicheStartUp")
        name =  name_element.get_text() if  name_element else "No name found"
        description_element = contain.find("div",{"itemprop":"description"})
        description =  description_element.get_text() if  description_element else "No description found"
        product_element = contain.find("div",{"itemprop":"makesOffer"})
        product =  product_element.get_text() if  product_element else "No product found"
        creators_element = contain.find("div",{"itemprop":"founders"})
        creators = creators_element.get_text() if  creators_element else "No creators found"
        domain_element = contain.find("a",{"itemprop":"sameAs"})
        domain =  creators_element.get_text() if  creators_element else "No domain found"
        telephone_element  = contain.find("p",{"itemprop":"telephone"})
        telephone = telephone_element.get_text() if telephone_element else "No number found"
        email_element = contain.find("p",{"itemprop":"email"})
        email = email_element.get_text() if email_element else "No number found"
        time.sleep(3)
        print(name,description,product,creators,domain,telephone,email)

        from csv import writer
        def AddToCSV(List):

            with open("Output.csv", "a+", newline='') as output_file:
                csv_writer = writer(output_file)
                csv_writer.writerow(List)

        # this can be used within your for loop
        row_list = [name,description,product,creators,domain,telephone,email]
        AddToCSV(row_list)
    
browser.quit()
Reply
#4
you said that you tried a lot of solutions but they didn't work
I asked to show what you have tried and did not work, so that people don't waste time to suggest again the same thing. You don't show what you have tried.
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply
#5
I tried this.

from seleniumwire import webdriver
from selenium import webdriver

proxy= "username:password@ip:port"
options = {'proxy': {'http': proxy, 'https': proxy, 'no_proxy': 'localhost,127.0.0.1,dev_server:8080'}}

driver = webdriver.Chrome(options=chrome_options, executable_path="path of chrome driver", seleniumwire_options=options)
Reply
#6
try without this line from selenium import webdriver
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply
#7
tried but didn't work
Reply
#8
Looking at the docs - I think you need to have https/http in proxy string
from seleniumwire import webdriver

options = {'proxy':{'http':"http://username:password@ip:port",
                    'https':"https://username:password@ip:port",
                    'no_proxy':'localhost,127.0.0.1,dev_server:8080'}}
 
driver = webdriver.Chrome(options=chrome_options, executable_path="path of chrome driver", seleniumwire_options=options)
note, you pass chrome_options but it is not present in your code

also, post full traceback in error tags if you get any
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply
#9
AttributeError: 'dict' object has no attribute 'to_capabilities'

Process finished with exit code 1
Reply
#10
post the code that produce the error as well as full traceback in error tags, not the last line

the code I suggest is exactly as in the docs https://github.com/wkeeling/selenium-wire#proxies
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  saml2 idp sso authentication tiagome 1 2,085 Apr-08-2022, 02:38 PM
Last Post: snippsat
  Proxy Variable in Selenium wont work with FireFox Profile Proxy Setting. MIPython 0 8,491 Jul-13-2018, 05:43 PM
Last Post: MIPython
  Error in Selenium: CRITICAL:root:Selenium module is not installed...Exiting program. AcszE 1 3,588 Nov-03-2017, 08:41 PM
Last Post: metulburr
  Python+Selenium+PhantomJS with proxy list technoir 0 8,647 Jan-17-2017, 05:22 AM
Last Post: technoir

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020