Python Forum
Requests module get() incomplete download
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Requests module get() incomplete download
#1
With requests.get() the file downloaded is only 3/4 of the file I get by bringing up the page in Firefox and doing a "Save As". It lacks a data item that is visible when reading the page on the screen.
Reply
#2
What exactly do you expect, without showing any code?
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply
#3
Your reproach is justified. I understand nothing about Web communications, and do not know where to start. For what it is worth, I took a sample case. I downloaded a page with Requests using this program:
#!/usr/bin/python
import requests
url='https://www.wsj.com/market-data/quotes/IBM/financials/annual/balance-sheet'
r=requests.get(url,allow_redirects=True)
open('ibmrqst.html','wb').write(r.content)
Then I called up the page with Firefox, right clicked on the page and did "Save Page As..." with "Web Page,complete" and then with "Web Page,HTML only". This obtained the following:
Requests download:
-rw-r--r-- 1 boba boba 710985 Feb 17 11:19 ibmrqst.html
Firefox download complete:
-rw-rw-r-- 1 boba boba 527413 Feb 17 11:21 ibmffx.html
drwxr-xr-x 2 boba boba 4096 Feb 17 11:21 ibmffx_files
Firefox download HTML only:
-rw-rw-r-- 1 boba boba 414414 Feb 17 11:23 ibmffxhonly.html
I do not know whether any significant values on the page are missing or not; it is just that the sizes of the files are different, even though they are all HTML. I had hoped there was a generally known explanation for such cases.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  POST requests - different requests return the same response Default_001 3 1,952 Mar-10-2022, 11:26 PM
Last Post: Default_001
  requests module is not working varsh 3 3,836 Sep-10-2020, 03:53 PM
Last Post: buran
  ModuleNotFoundError: No module named "requests" BushFacts 1 5,145 Jan-13-2020, 06:49 AM
Last Post: perfringo
  help about requests download ggbaby 1 2,271 Sep-18-2018, 03:44 AM
Last Post: wavic
  An Error in Requests Module pratheep 3 11,931 Feb-06-2018, 05:17 PM
Last Post: pratheep
  Using python requests module and BS4 to login on an Wordpress based website apollo 1 9,430 Feb-06-2018, 01:31 AM
Last Post: metulburr
  disable proxy with requests module metulburr 2 9,560 Apr-05-2017, 02:54 AM
Last Post: metulburr

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020