Python Forum
extracting data from json on website
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
extracting data from json on website
#1
I am trying to extract two data items from a json web page, frustratingly without much joy. Wall
Being fairly new to json, it is proving difficult for me as I'm probably not fully understanding the Python libraries for doing this.

When I run the following code in Python 3.5, I get the following output:
Error:
Python Version: 3.5.2 URL Return Code: 200 Content Type: text/plain; charset=utf-8 Traceback (most recent call last): File "test5.py", line 30, in <module> for key, value in data.items(): AttributeError: 'list' object has no attribute 'items'
Can anyone suggest where I am going wrong.
#! /usr/bin/env python3
import platform
import json
import requests

class JSONObject:
	def __init__(self, d):
		self.__dict__ = d

print("Python Version: %s" % platform.python_version())

# for testing, this url has the appropriate data formats
url="https://raw.githubusercontent.com/kovidgoyal/build-calibre/master/scripts/sources.json"
urlFile = requests.get(url)
print('URL Return Code: ', urlFile.status_code)
print('Content Type:    ', urlFile.headers["content-type"])

#json_data = urlFile.json	# NOT WORKING
#data = json.loads(urlFile.content.decode('utf-8'))	#WORKS
data = requests.get(url).json()	#WORKS

### Print "name" and "unix-->filename" from json data
#for key, value in json_data.iteritems():
# something???
#	print("%s: %s", name, filename)

### later - store in python list and process

count = 3	# for testing just print first 3
for key, value in data.items():
	print('key: {}, value: {}'.format(key, value))
	count -= 1
	if count == 0:
		break
Reply
#2
this json is array of objects, or converted to python terms - list of dicts

import requests
url="https://raw.githubusercontent.com/kovidgoyal/build-calibre/master/scripts/sources.json"
data = requests.get(url).json()
print data[:3]
print '\n\n'
for obj in data[:3]:
    for key, value in obj.items():
        print('{} --> {}'.format(key, value))
    print('')
Output:
[{u'unix': {u'hash': u'sha256:4ff941449631ace0d4d203e3483be9dbc9da454084111f97ea 0a2114e19bf066', u'urls': [u'http://zlib.net/{filename}'], u'filename': u'zlib-1 .2.11.tar.xz'}, u'name': u'zlib'}, {u'unix': {u'hash': u'md5:00b516f4704d4a7cb50 a1d97e6e8e15b', u'urls': [u'http://www.bzip.org/1.0.6/{filename}'], u'filename': u'bzip2-1.0.6.tar.gz'}, u'name': u'bzip2'}, {u'unix': {u'hash': u'sha1:f4fa1c41 42b497356a3df505e1459c188f646fad', u'urls': [u'http://www.rarlab.com/rar/{filena me}'], u'filename': u'unrarsrc-5.5.8.tar.gz'}, u'name': u'unrar'}] unix --> {u'hash': u'sha256:4ff941449631ace0d4d203e3483be9dbc9da454084111f97ea0a 2114e19bf066', u'urls': [u'http://zlib.net/{filename}'], u'filename': u'zlib-1.2 .11.tar.xz'} name --> zlib unix --> {u'hash': u'md5:00b516f4704d4a7cb50a1d97e6e8e15b', u'urls': [u'http://w ww.bzip.org/1.0.6/{filename}'], u'filename': u'bzip2-1.0.6.tar.gz'} name --> bzip2 unix --> {u'hash': u'sha1:f4fa1c4142b497356a3df505e1459c188f646fad', u'urls': [u 'http://www.rarlab.com/rar/{filename}'], u'filename': u'unrarsrc-5.5.8.tar.gz'} name --> unrar >>>
Reply
#3
Thank you, that got past the list, dict addressing issues and the looping thru them.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  encrypt data in json file help jacksfrustration 1 268 Mar-28-2024, 05:16 PM
Last Post: deanhystad
  Scraping Data from Website melkaray 3 815 Sep-22-2023, 12:41 PM
Last Post: melkaray
  Extracting Data into Columns using pdfplumber arvin 17 5,672 Dec-17-2022, 11:59 AM
Last Post: arvin
  Read nested data from JSON - Getting an error marlonbown 5 1,408 Nov-23-2022, 03:51 PM
Last Post: snippsat
  Reading Data from JSON tpolim008 2 1,116 Sep-27-2022, 06:34 PM
Last Post: Larz60+
  Code to retrieve data from a website charlie13255 0 997 Jul-07-2022, 07:53 PM
Last Post: charlie13255
  Is this possible to write a script for checking data from website? WanW 2 1,139 Jun-02-2022, 02:31 AM
Last Post: Larz60+
  Convert nested sample json api data into csv in python shantanu97 3 2,893 May-21-2022, 01:30 PM
Last Post: deanhystad
  Struggling with Juggling JSON Data SamWatt 7 1,940 May-09-2022, 02:49 AM
Last Post: snippsat
  json api data parsing elvis 0 946 Apr-21-2022, 11:59 PM
Last Post: elvis

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020