Python Forum
Beautiful Soup (suddenly) doesn't get full webpage html
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Beautiful Soup (suddenly) doesn't get full webpage html
#4
Thank you both for answers.

@HarleyQuin
The code I ran months ago was same as I posted here, but result was not same. As stated, on my first attempt I got all the HTML contents, while this time I didn't. Also, replacing the parser for lxml parser didn't make a difference. Do you have any idea, from experience, why such difference?

@snippsat
Your code returns all the HTML contents of the page, if I print the soup. Is the main factor here 2 seconds sleep, which allows the Javascript to execute completely before parsing the HTML? However, to reiterate, my original run of the code returned complete HTML contents. Could it be that website render just got slower for some reason, since my last attempt at parsing (few months ago)?
Reply


Messages In This Thread
RE: Beautiful Soup (suddenly) doesn't get full webpage html - by j.crater - Jul-11-2020, 11:52 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Selenium suddenly fails to find element Pavel_47 3 6,197 Sep-04-2022, 11:06 AM
Last Post: Pavel_47
Lightbulb Python Obstacles | Kung-Fu | Full File HTML Document Scrape and Store it in MariaDB BrandonKastning 5 2,817 Dec-29-2021, 02:26 AM
Last Post: BrandonKastning
  Beautiful Soup - access a rating value in a class KatMac 1 3,419 Apr-16-2021, 01:27 PM
Last Post: snippsat
  HTML multi select HTML listbox with Flask/Python rfeyer 0 4,529 Mar-14-2021, 12:23 PM
Last Post: rfeyer
  *Beginner* web scraping/Beautiful Soup help 7ken8 2 2,560 Jan-28-2021, 04:26 PM
Last Post: 7ken8
  Help: Beautiful Soup - Parsing HTML table ironfelix717 2 2,622 Oct-01-2020, 02:19 PM
Last Post: snippsat
  Requests-HTML vs Beautiful Soup - How to Choose? robin73 0 3,780 Jun-23-2020, 02:53 PM
Last Post: robin73
  Python3 + BeautifulSoup4 + lxml (HTML -> CSV) - How to loop to next HTML/new CSV Row BrandonKastning 0 2,328 Mar-22-2020, 06:10 AM
Last Post: BrandonKastning
  looking for direction - scrappy, crawler, beautiful soup Sly_Corn 2 2,402 Mar-17-2020, 03:17 PM
Last Post: Sly_Corn
  Beautiful soup truncates results jonesjoz 4 3,794 Mar-09-2020, 06:04 PM
Last Post: jonesjoz

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020