Python Forum
How do I avoid Beautiful Soup redirects?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How do I avoid Beautiful Soup redirects?
#1

import bs4 as bs
import urllib.request
sauce = urllib.request.urlopen('https://globenewswire.com/Search/NewsSearch?lang=en&exchange=NYSE').read()
soup = bs.BeautifulSoup(sauce,'lxml')
list = []
for div in soup.find_all('div', class_='results-link', limit=10):
	initialglobenewsnyseurls = ('https://globenewswire.com' + div.h1.a['href'])
	list.append(initialglobenewsnyseurls)
	a, b, c, d, e, f, g, h, i, j = list
so far this works. The only problem is I have the exchange set to NYSE, but when I enter the url as such, NYSE is removed from it, as the url is automatically redirected to:
https://globenewswire.com/NewsRoom

(if you copy and paste the original url into chrome(the one in the code), it will redirect you to the main newsroom, and remove any criteria you previously selected. How can I keep this from happening?
Reply


Messages In This Thread
How do I avoid Beautiful Soup redirects? - by HiImNew - Nov-29-2017, 12:28 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Soup('A') new_coder_231013 6 3,096 Aug-12-2023, 10:55 AM
Last Post: Pubfonts
  Beautiful Soup - access a rating value in a class KatMac 1 3,613 Apr-16-2021, 01:27 PM
Last Post: snippsat
  *Beginner* web scraping/Beautiful Soup help 7ken8 2 2,785 Jan-28-2021, 04:26 PM
Last Post: 7ken8
  Help: Beautiful Soup - Parsing HTML table ironfelix717 2 2,875 Oct-01-2020, 02:19 PM
Last Post: snippsat
  Beautiful Soup (suddenly) doesn't get full webpage html j.crater 8 18,145 Jul-11-2020, 04:31 PM
Last Post: j.crater
  Requests-HTML vs Beautiful Soup - How to Choose? robin73 0 3,924 Jun-23-2020, 02:53 PM
Last Post: robin73
  looking for direction - scrappy, crawler, beautiful soup Sly_Corn 2 2,618 Mar-17-2020, 03:17 PM
Last Post: Sly_Corn
  Beautiful soup truncates results jonesjoz 4 4,188 Mar-09-2020, 06:04 PM
Last Post: jonesjoz
  Beautiful soup and tags starter_student 11 6,699 Jul-08-2019, 03:41 PM
Last Post: starter_student
  Beautiful Soup find_all() kirito85 2 3,552 Jun-14-2019, 02:17 AM
Last Post: kirito85

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020