Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
requests-html + Beautifulsoup
#1
Hello,

I am trying to scrape a dynamic web page. The code I use is:

from requests_html import HTMLSession
from bs4 import BeautifulSoup

session = HTMLSession()

url = 'someurlhere'

r = session.get(url)
r.html.render(keep_page=True, sleep=1,timeout=10,wait=2)
Question 1: How do I save the rendered HTML to file?
Question 2: How do I feed the rendered HTML to Beautifulsoup?

Thank you!
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Getting a URL from Amazon using requests-html, or beautifulsoup aaander 1 1,662 Nov-06-2022, 10:59 PM
Last Post: snippsat
  POST requests - different requests return the same response Default_001 3 1,938 Mar-10-2022, 11:26 PM
Last Post: Default_001
  HTML multi select HTML listbox with Flask/Python rfeyer 0 4,621 Mar-14-2021, 12:23 PM
Last Post: rfeyer
Smile Extracting the Address tag from multiple HTML files using BeautifulSoup Dredd 8 4,881 Jan-25-2021, 12:16 PM
Last Post: Dredd
  Requests-HTML vs Beautiful Soup - How to Choose? robin73 0 3,820 Jun-23-2020, 02:53 PM
Last Post: robin73
  Python3 + BeautifulSoup4 + lxml (HTML -> CSV) - How to loop to next HTML/new CSV Row BrandonKastning 0 2,362 Mar-22-2020, 06:10 AM
Last Post: BrandonKastning
  requests post/get to HTML form mrdominikku 1 2,325 Nov-03-2019, 07:12 PM
Last Post: Larz60+
  BeautifulSoup: Error while extracting a value from an HTML table kawasso 3 3,219 Aug-25-2019, 01:13 AM
Last Post: kawasso
  How to clean html content using BeautifulSoup in Python 3.6? PrateekG 5 10,344 Apr-27-2018, 01:14 PM
Last Post: snippsat

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020