Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
[Hlep]Scrap webiste
#1
Hi All,

I have a website that I need to collect some info from it.
I tried to use simple code like this :
import urllib.request
headers = {}
headers['User-Agent'] = "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:48.0) Gecko/20100101 Firefox/48.0"
url= 'https://www.biopharmcatalyst.com/calendars/historical-catalyst-calendar'
x  = urllib.request.Request(url,headers=headers)
html = urllib.request.urlopen(x,timeout=10).read()
it didn't work. they python program hang !

I tried this as well :

import requests
url= 'https://www.biopharmcatalyst.com/calendars/historical-catalyst-calendar'
url_get = requests.get(url)
it also didn't work !!!

any idea what is the problem ?
Reply
#2
You can use the requests module. Requests
For an advanced scrapping I suggest you to use the beatifulsoup module. BeautifulSoup

To see the content of the requests in your second script, use .text:

import requests
url = 'https://www.biopharmcatalyst.com/calendars/historical-catalyst-calendar'
url_get = requests.get(url)
print(url_get.text)
Reply
#3
for further reading:
Web Scraping part1
Web Scraping part2
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Web scrap --Need help Lizardpython 4 953 Oct-01-2023, 11:37 AM
Last Post: Lizardpython
  I tried every way to scrap morningstar financials data without success so far sparkt 2 8,171 Oct-20-2020, 05:43 PM
Last Post: sparkt
  Web scrap multiple pages anilacem_302 3 3,783 Jul-01-2020, 07:50 PM
Last Post: mlieqo
  Need logic on how to scrap 100K URLs goodmind 2 2,569 Jun-29-2020, 09:53 AM
Last Post: goodmind
  Scrap a dynamic span hefaz 0 2,659 Mar-07-2020, 02:56 PM
Last Post: hefaz
  scrap by defining 3 functions zarize 0 1,833 Feb-18-2020, 03:55 PM
Last Post: zarize
  Skipping anti-scrap zarize 0 1,853 Jan-17-2020, 11:51 AM
Last Post: zarize
  Cannot get selenium to scrap past the first two pages newbie_programmer 0 4,133 Dec-12-2019, 06:19 AM
Last Post: newbie_programmer
  Scrap data from not standarized page? zarize 4 3,242 Nov-25-2019, 10:25 AM
Last Post: zarize
  page impossible to scrap? :O zarize 2 3,882 Oct-03-2019, 02:44 PM
Last Post: zarize

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020