Python Forum

Full Version: [Hlep]Scrap webiste
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Hi All,

I have a website that I need to collect some info from it.
I tried to use simple code like this :
import urllib.request
headers = {}
headers['User-Agent'] = "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:48.0) Gecko/20100101 Firefox/48.0"
url= ''
x  = urllib.request.Request(url,headers=headers)
html = urllib.request.urlopen(x,timeout=10).read()
it didn't work. they python program hang !

I tried this as well :

import requests
url= ''
url_get = requests.get(url)
it also didn't work !!!

any idea what is the problem ?
You can use the requests module. Requests
For an advanced scrapping I suggest you to use the beatifulsoup module. BeautifulSoup

To see the content of the requests in your second script, use .text:

import requests
url = ''
url_get = requests.get(url)