Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Beautiful soup truncates results
#1
When scraping a long web page the printed results get cut truncated. Any advice?
import requests
from bs4 import BeautifulSoup
URL = 'https://www.mobileread.com/forums/showthread.php?t=285771'
page = requests.get(URL)
soup = BeautifulSoup(page.content, 'html.parser')
print(soup.text)
Reply
#2
Quote:printed results get cut truncated
How so?
Reply
#3
(Mar-08-2020, 03:52 AM)Larz60+ Wrote:
Quote:printed results get cut truncated
How so?

The code will print the first 992 lines of text which is not the entirety. There is still more to follow.

<sample output>
<many lines>
I have updated the plugin in the first post so that it manages angular brackets that contain recognised HTML tags.

I tested it with variations of the following text file:

This is a line of text
This is <HELLO> <i>another</i>
<end sample>
Reply
#4
It's there, you just cant see it because no formatting.
change line 6 to
print(soup.prettify())
Reply
#5
Strangely enough, on a different computer the original code works fine. .prettyfy() did make more tags visible than just .text
Thank you all. - JJ
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Beautiful Soup - access a rating value in a class KatMac 1 3,419 Apr-16-2021, 01:27 PM
Last Post: snippsat
  *Beginner* web scraping/Beautiful Soup help 7ken8 2 2,560 Jan-28-2021, 04:26 PM
Last Post: 7ken8
  Help: Beautiful Soup - Parsing HTML table ironfelix717 2 2,622 Oct-01-2020, 02:19 PM
Last Post: snippsat
  Beautiful Soup (suddenly) doesn't get full webpage html j.crater 8 16,381 Jul-11-2020, 04:31 PM
Last Post: j.crater
  Requests-HTML vs Beautiful Soup - How to Choose? robin73 0 3,780 Jun-23-2020, 02:53 PM
Last Post: robin73
  looking for direction - scrappy, crawler, beautiful soup Sly_Corn 2 2,401 Mar-17-2020, 03:17 PM
Last Post: Sly_Corn
  Beautiful soup and tags starter_student 11 6,047 Jul-08-2019, 03:41 PM
Last Post: starter_student
  Beautiful Soup find_all() kirito85 2 3,311 Jun-14-2019, 02:17 AM
Last Post: kirito85
  [split] Using beautiful soup to get html attribute value moski 6 6,220 Jun-03-2019, 04:24 PM
Last Post: moski
  Using beautiful soup to get html attribute value graham23s 2 17,977 Apr-23-2019, 09:21 PM
Last Post: graham23s

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020