Python Forum
extracting data from json on website - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: extracting data from json on website (/thread-8173.html)



extracting data from json on website - larry2311 - Feb-08-2018

I am trying to extract two data items from a json web page, frustratingly without much joy. Wall
Being fairly new to json, it is proving difficult for me as I'm probably not fully understanding the Python libraries for doing this.

When I run the following code in Python 3.5, I get the following output:
Error:
Python Version: 3.5.2 URL Return Code: 200 Content Type: text/plain; charset=utf-8 Traceback (most recent call last): File "test5.py", line 30, in <module> for key, value in data.items(): AttributeError: 'list' object has no attribute 'items'
Can anyone suggest where I am going wrong.
#! /usr/bin/env python3
import platform
import json
import requests

class JSONObject:
	def __init__(self, d):
		self.__dict__ = d

print("Python Version: %s" % platform.python_version())

# for testing, this url has the appropriate data formats
url="https://raw.githubusercontent.com/kovidgoyal/build-calibre/master/scripts/sources.json"
urlFile = requests.get(url)
print('URL Return Code: ', urlFile.status_code)
print('Content Type:    ', urlFile.headers["content-type"])

#json_data = urlFile.json	# NOT WORKING
#data = json.loads(urlFile.content.decode('utf-8'))	#WORKS
data = requests.get(url).json()	#WORKS

### Print "name" and "unix-->filename" from json data
#for key, value in json_data.iteritems():
# something???
#	print("%s: %s", name, filename)

### later - store in python list and process

count = 3	# for testing just print first 3
for key, value in data.items():
	print('key: {}, value: {}'.format(key, value))
	count -= 1
	if count == 0:
		break



RE: extracting data from json on website - buran - Feb-08-2018

this json is array of objects, or converted to python terms - list of dicts

import requests
url="https://raw.githubusercontent.com/kovidgoyal/build-calibre/master/scripts/sources.json"
data = requests.get(url).json()
print data[:3]
print '\n\n'
for obj in data[:3]:
    for key, value in obj.items():
        print('{} --> {}'.format(key, value))
    print('')
Output:
[{u'unix': {u'hash': u'sha256:4ff941449631ace0d4d203e3483be9dbc9da454084111f97ea 0a2114e19bf066', u'urls': [u'http://zlib.net/{filename}'], u'filename': u'zlib-1 .2.11.tar.xz'}, u'name': u'zlib'}, {u'unix': {u'hash': u'md5:00b516f4704d4a7cb50 a1d97e6e8e15b', u'urls': [u'http://www.bzip.org/1.0.6/{filename}'], u'filename': u'bzip2-1.0.6.tar.gz'}, u'name': u'bzip2'}, {u'unix': {u'hash': u'sha1:f4fa1c41 42b497356a3df505e1459c188f646fad', u'urls': [u'http://www.rarlab.com/rar/{filena me}'], u'filename': u'unrarsrc-5.5.8.tar.gz'}, u'name': u'unrar'}] unix --> {u'hash': u'sha256:4ff941449631ace0d4d203e3483be9dbc9da454084111f97ea0a 2114e19bf066', u'urls': [u'http://zlib.net/{filename}'], u'filename': u'zlib-1.2 .11.tar.xz'} name --> zlib unix --> {u'hash': u'md5:00b516f4704d4a7cb50a1d97e6e8e15b', u'urls': [u'http://w ww.bzip.org/1.0.6/{filename}'], u'filename': u'bzip2-1.0.6.tar.gz'} name --> bzip2 unix --> {u'hash': u'sha1:f4fa1c4142b497356a3df505e1459c188f646fad', u'urls': [u 'http://www.rarlab.com/rar/{filename}'], u'filename': u'unrarsrc-5.5.8.tar.gz'} name --> unrar >>>



RE: extracting data from json on website - larry2311 - Feb-09-2018

Thank you, that got past the list, dict addressing issues and the looping thru them.