Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Problem parsing website html file
#1
Hi Guys,

I am trying to parse this web link (http://insight.toba.fun/address/TVAWF6eD...sigfWK2oRB) using Python3 and requests. When we click the link it loads the details of the account. However, when calling from requests, it does not generate the HTML page that can be obtained by clicking the link. I tried putting a delay but didn't work. I'm unable to get the account the info using python. Can anyone point out a solution for this?
I tried using selenium webdriver, but I'm unable to find the search box in the html.

import requests
url='http://insight.toba.fun/address/TVAWF6eDfW9MRdLitCb7ag41sigfWK2oRB'
r = requests.get(url,verify=False)
print(r.text)
Reply
#2
Quote:However, when calling from requests, it does not generate the HTML page that can be obtained by clicking the link.
You are not saving the file.
add after line 3:
with open('yourfilename', 'w') as f:
    f.write(r.text)
(of course, replace 'yourfilername' with a valid .html file name)
Reply
#3
(May-01-2018, 11:04 AM)Larz60+ Wrote:
Quote:However, when calling from requests, it does not generate the HTML page that can be obtained by clicking the link.
You are not saving the file.
add after line 3:
with open('yourfilename', 'w') as f:
    f.write(r.text)
(of course, replace 'yourfilername' with a valid .html file name)

The OP probably meant - page is not rendered properly because of Javascript. Try https://html.python-requests.org/ which can render the page with Chromium.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Problem with scrapping Website giddyhead 1 1,582 Mar-08-2024, 08:20 AM
Last Post: AhanaSharma
Lightbulb Python Obstacles | Kung-Fu | Full File HTML Document Scrape and Store it in MariaDB BrandonKastning 5 2,821 Dec-29-2021, 02:26 AM
Last Post: BrandonKastning
  show csv file in flask template.html rr28rizal 8 34,532 Apr-12-2021, 09:24 AM
Last Post: adamabusamra
  HTML multi select HTML listbox with Flask/Python rfeyer 0 4,536 Mar-14-2021, 12:23 PM
Last Post: rfeyer
  Login and download an exported csv file within a ribbon/button in a website Alekhya 0 2,617 Feb-26-2021, 04:15 PM
Last Post: Alekhya
  Parsing html page and working with checkbox (on a captcha) straannick 17 11,066 Feb-04-2021, 02:54 PM
Last Post: snippsat
  Help: Beautiful Soup - Parsing HTML table ironfelix717 2 2,623 Oct-01-2020, 02:19 PM
Last Post: snippsat
  Problem with logging in on website - python w/ requests GoldeNx 6 5,214 Sep-25-2020, 10:52 AM
Last Post: snippsat
  Open and read a tab delimited file from html using python cgi luffy 2 2,633 Aug-24-2020, 06:25 AM
Last Post: luffy
  Python3 + BeautifulSoup4 + lxml (HTML -> CSV) - How to loop to next HTML/new CSV Row BrandonKastning 0 2,329 Mar-22-2020, 06:10 AM
Last Post: BrandonKastning

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020