Python Forum
Can't read url when specifying more than a specific number of bytes
Thread Rating:
  • 1 Vote(s) - 4 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Can't read url when specifying more than a specific number of bytes
#1
This works

from urllib.request import urlopen

data = urlopen("https://cinnamon-spices.linuxmint.com/json/applets.json").read(60000)

print(data)
This hangs

from urllib.request import urlopen

data = urlopen("https://cinnamon-spices.linuxmint.com/json/applets.json").read(70000)

print(data)
or without a specific number of bytes, it hangs too


from urllib.request import urlopen

data = urlopen("https://cinnamon-spices.linuxmint.com/json/applets.json").read()

print(data)
It only happens for this specific site (that I know of).

Any ideas on what might be going on? I'm totally new to python.
Reply
#2
It do not hang for me.
Anyway should not use urllib,use Requests
Requests has build in JSON decoder.
import requests

response = requests.get("https://cinnamon-spices.linuxmint.com/json/applets.json")
data = response.json()
print(data)
Example parse:
>>> data['location-detection@heimdall']
{'author_id': '0',
 'author_user': 'none',
 'created': 1408356497,
 'description': 'Shows the geographic location of your public IP and '
                'country-flag.',
 'file': '/files/applets/[email protected]',
 'icon': '/files/applets/[email protected]',
 'last_edited': 1504832706,
 'name': 'Location Detection + Flags',
 'score': 2,
 'screenshot': '/git/applets/location-detection@heimdall/screenshot.png',
 'spices-id': 196,
 'uuid': 'location-detection@heimdall'}

>>> data['location-detection@heimdall']['screenshot']
'/git/applets/location-detection@heimdall/screenshot.png'
Reply
#3
Thank you @snippsat!

Using requests did the trick and worked just fine. My program was only a minimal reproduction of a problem I'm having with one library. Have a look at https://github.com/linuxmint/Cinnamon/is...-411201291 and the following messages.

Knowing that this works with requests is useful. It sounds like something where the standard library could do better so I guess it might be worth reporting to python?
Reply
#4
Just to report this theads "resolution". I ended up reporting to python here, and the culprit was the "Connection: close" header added by the standard library by default. The best solution for now (other than switching to the requests library) seems to use the underlying "http.client" directly so I can customize the headers sent and avoid sending the "Connection: close" one. Although apparently it should just work with the default headers and the issue is due to some server misconfiguration.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Read directory listing of files and parse out the highest number? cubangt 5 2,376 Sep-28-2022, 10:15 PM
Last Post: Larz60+
Question How to understand the received bytes of ser.read? pf2022 3 1,995 Mar-24-2022, 11:37 AM
Last Post: pf2022
  Remove Specific Columns when the number of columns is greater than a specific value CuriousOne 0 1,326 Sep-09-2021, 09:17 PM
Last Post: CuriousOne
  [Solved] Trying to read specific lines from a file Laplace12 7 3,569 Jun-21-2021, 11:15 AM
Last Post: Laplace12
  TypeError: int() argument must be a string, a bytes-like object or a number, not 'Non Anldra12 2 5,231 May-02-2021, 03:45 PM
Last Post: Anldra12
  csv.reader(): Limit the number of columns read in Windows Pedroski55 9 5,245 Jan-23-2021, 01:03 AM
Last Post: pjfarley3
  read logfile between two specific strings FelixReiter 6 2,941 Jan-04-2021, 02:26 PM
Last Post: FelixReiter
  Read Multiples Text Files get specific lines based criteria zinho 5 3,142 May-19-2020, 12:30 PM
Last Post: zinho
  adding elements to a list that are more than a specific number Olavv 2 2,207 Mar-19-2020, 06:05 PM
Last Post: Olavv
  Delete specific lines contain specific words mannyi 2 4,149 Nov-04-2019, 04:50 PM
Last Post: mannyi

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020