Python Forum
How do I avoid Beautiful Soup redirects?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How do I avoid Beautiful Soup redirects?
#1

import bs4 as bs
import urllib.request
sauce = urllib.request.urlopen('https://globenewswire.com/Search/NewsSearch?lang=en&exchange=NYSE').read()
soup = bs.BeautifulSoup(sauce,'lxml')
list = []
for div in soup.find_all('div', class_='results-link', limit=10):
	initialglobenewsnyseurls = ('https://globenewswire.com' + div.h1.a['href'])
	list.append(initialglobenewsnyseurls)
	a, b, c, d, e, f, g, h, i, j = list
so far this works. The only problem is I have the exchange set to NYSE, but when I enter the url as such, NYSE is removed from it, as the url is automatically redirected to:
https://globenewswire.com/NewsRoom

(if you copy and paste the original url into chrome(the one in the code), it will redirect you to the main newsroom, and remove any criteria you previously selected. How can I keep this from happening?
Reply


Messages In This Thread
How do I avoid Beautiful Soup redirects? - by HiImNew - Nov-29-2017, 12:28 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Soup('A') new_coder_231013 6 3,076 Aug-12-2023, 10:55 AM
Last Post: Pubfonts
  Beautiful Soup - access a rating value in a class KatMac 1 3,608 Apr-16-2021, 01:27 PM
Last Post: snippsat
  *Beginner* web scraping/Beautiful Soup help 7ken8 2 2,783 Jan-28-2021, 04:26 PM
Last Post: 7ken8
  Help: Beautiful Soup - Parsing HTML table ironfelix717 2 2,867 Oct-01-2020, 02:19 PM
Last Post: snippsat
  Beautiful Soup (suddenly) doesn't get full webpage html j.crater 8 18,110 Jul-11-2020, 04:31 PM
Last Post: j.crater
  Requests-HTML vs Beautiful Soup - How to Choose? robin73 0 3,922 Jun-23-2020, 02:53 PM
Last Post: robin73
  looking for direction - scrappy, crawler, beautiful soup Sly_Corn 2 2,611 Mar-17-2020, 03:17 PM
Last Post: Sly_Corn
  Beautiful soup truncates results jonesjoz 4 4,182 Mar-09-2020, 06:04 PM
Last Post: jonesjoz
  Beautiful soup and tags starter_student 11 6,695 Jul-08-2019, 03:41 PM
Last Post: starter_student
  Beautiful Soup find_all() kirito85 2 3,550 Jun-14-2019, 02:17 AM
Last Post: kirito85

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020