+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: Clone with pywebcopy with proxy (/thread-30647.html)
Clone with pywebcopy with proxy - MeMyselfAndI - Oct-29-2020
Im struggling with pywebcopy and proxies. Its no problem to scrape a Onion site with BeautifulSoup(not the below code).
But im trying to use pywebcopy and cant figure it out, any help would be appreciated. It works fine with "ordinary sites" but not with Onion urls.
from pywebcopy import WebPage
from pywebcopy import save_webpage
import requests
import os
proxies = {
'http': 'socks5h://localhost:9150',
'https': 'socks5h://localhost:9150',
}
proxy = 'socks5h://localhost:9150'
os.environ['http_proxy'] = proxy
os.environ['HTTP_PROXY'] = proxy
os.environ['https_proxy'] = proxy
os.environ['HTTPS_PROXY'] = proxy
save_webpage(
url='http://qrmfuxwgyzk5jdjz.onion',
project_folder='C:/temp/PyClone',
)When trying with Onion url i get the following:Error:Exception has occurred: AccessError
Access is not allowed by the site of url http://qrmfuxwgyzk5jdjz.onion
File "C:\temp\PyClone\testreq2.py", line 20, in <module>
project_folder='C:/temp/PyClone',RE: Clone with pywebcopy with proxy - Larz60+ - Oct-29-2020
As the error shows.
the specified URL does not exist, or is down.
also, you are not showing where you execute the code in your listing.