Python Forum
API JSON response missing list gives keyerror
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
API JSON response missing list gives keyerror
#1
Python newbie - please be gentle ...
System: Windows 10 + Winpython + Spyder

I have a list of searchqueries that I use to perform search requests against an API which returns JSON.
For each response I print the values of the keys 'hitcount' and 'title' of the first object.

Some of the responses don't contain a searchresult resulting in:
"KeyError; 'searchResult' in the terminal.
NB. See full terminal output below.

I have been trying to print the values from a number of requests in a for loop and it works great until the keyerror blocks the loop.

I have tried ".get()", defaultdict and Try Except to no avail, but may just be using them incorrectly.

The missing JSON 'searchResult' structure looks like this:
NB. See full REST URL's below for full JSON response.

[...]
Output:
{ "searchResponse": { "result": { "hitCount": { "$": "103554" }, "searchResult": [ { "collection": { "resultPosition": { "$": "1" }, "numberOfObjects": { "$": "1" }, "object": [ { "record": { "title": [ { "$": "Peter Nicolaisen", "@": "dc" }, ] } } ] } }, {}, {}, {}, {} ], }, }, }
[...]

Query that works
https://oss-services.dbc.dk/opensearch/5...tType=json

Query that doesnt work
https://oss-services.dbc.dk/opensearch/5...tType=json

Script

#-*- coding: utf-8 -*-
"""
Created on Sat Mar 21 17:21:16 2020

@author: Rolf
"""
import json
import requests

def getsearchresult(query):
    
    querystring = f"\"{query}\""

    parameters = {
        'action':'search',
        'query':querystring,
        'agency':'100200',
        'profile':'test',
        'start':'1',
        'stepValue':'5',
        'outputType':'json'
    }
    
    endpoint = 'https://oss-services.dbc.dk/opensearch/5.2/?'
    
    response = requests.get(
        url=endpoint, 
        params=parameters,
        )
    print(response.url)
    response_json = response.json()
    return response_json

#query = 'peter' #works
query = 'verdens 100 mærkeligste dyr' #doesnt work

response_dict = getsearchresult(query)

hitcount = response_dict['searchResponse']['result']['hitCount']['$']
title = response_dict['searchResponse']['result']['searchResult'][0]['collection']['object'][0]['record']['title'][0]['$']

print(f"A search for the query '{query}' has {hitcount} hits, and the first title is {title}!")
My suspicion is that I simply dont understand how nested dictionaries combined with lists work, but after days of scouring the web for solutions I am throwing in the towel.

Question: How do I handle the keyerror so it doesnt block a for loop posting values from a number of responses.

I hope you can help! :-)

Terminal output

Error:
runfile('C:/Users/**** ******/Dropbox/Development/Python/opensearch/testscript_python-forum_io.py', wdir='C:/Users/**** ******/Dropbox/Development/Python/opensearch') https://oss-services.dbc.dk/opensearch/5.2/?action=search&query=%22verdens+100+m%C3%A6rkeligste+dyr%22&agency=100200&profile=test&start=1&stepValue=5&outputType=json Traceback (most recent call last): File "<ipython-input-19-badb5bb4e229>", line 1, in <module> runfile('C:/Users/**** ******/Dropbox/Development/Python/opensearch/testscript_python-forum_io.py', wdir='C:/Users/Rolf Madsen/Dropbox/Development/Python/opensearch') File "C:\Users\**** ******\Dropbox\Development\WPy64-3810\python-3.8.1.amd64\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 827, in runfile execfile(filename, namespace) File "C:\Users\**** ******\Dropbox\Development\WPy64-3810\python-3.8.1.amd64\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 110, in execfile exec(compile(f.read(), filename, 'exec'), namespace) File "C:/Users/**** ******/Dropbox/Development/Python/opensearch/testscript_python-forum_io.py", line 40, in <module> title = response_dict['searchResponse']['result']['searchResult'][0]['collection']['object'][0]['record']['title'][0]['$'] KeyError: 'searchResult'
Reply
#2
one way is to check the hitcount value:
hitcount = response_dict['searchResponse']['result']['hitCount']['$']
if hitcount == '0':
    print(f"A search for the query '{query}' has {hitcount} hits")
else:
    title = response_dict['searchResponse']['result']['searchResult'][0]['collection']['object'][0]['record']['title'][0]['$']
    print(f"A search for the query '{query}' has {hitcount} hits, and the first title is {title}!")
another ways is use .get() as you tried

hitcount = response_dict['searchResponse']['result']['hitCount']['$']
search_result = response_dict['searchResponse']['result'].get('searchResult')
if search_result:
    title = search_result[0]['collection']['object'][0]['record']['title'][0]['$']
    print(f"A search for the query '{query}' has {hitcount} hits, and the first title is {title}!")
else:
    print('No search result found')
another option is to error-handle the KeyError:
hitcount = response_dict['searchResponse']['result']['hitCount']['$']
try:
    title = response_dict['searchResponse']['result']['searchResult'][0]['collection']['object'][0]['record']['title'][0]['$']
    print(f"A search for the query '{query}' has {hitcount} hits, and the first title is {title}!")
except KeyError:
    print(f"A search for the query '{query}' has {hitcount} hits")
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply
#3
@buran thank you so much!

The .get() example was missing the hitcount variable and the underscore in "search response", but after fixing that all three examples worked as a charm.

Thank you again for helping me learn, and aside from this keyerror problem Python has been a joy to work with after a number of tries with JavaScript/Node.
Reply
#4
(Mar-28-2020, 10:08 AM)rolfmadsen Wrote: The .get() example was missing the hitcount variable
It was presumably kept from original code, maybe I was unclear, sorry. The other was just typo :-)
I will edit my post
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Response.json list indices must be integers or slices, not str [SOLVED] AlphaInc 4 6,192 Mar-24-2023, 08:34 AM
Last Post: fullytotal
  Name not found in response json NewbiePyt 4 1,020 Dec-29-2022, 11:12 AM
Last Post: buran
  geojson to json --missing multiple row output yoshi 9 2,657 Mar-06-2022, 08:34 PM
Last Post: snippsat
Question convert unlabeled list of tuples to json (string) masterAndreas 4 7,356 Apr-27-2021, 10:35 AM
Last Post: masterAndreas
  Code failed JSON KeyError popps 3 2,364 Jan-15-2021, 03:45 PM
Last Post: buran
  JSON response from REST service get nested value nl2ttl 2 2,489 Nov-30-2020, 09:34 PM
Last Post: nl2ttl
  parser json response absolut 4 2,784 Sep-15-2020, 12:10 PM
Last Post: buran
  Compare response and name list in experiment knoxvillerailgrind 3 2,174 Jul-26-2020, 12:23 PM
Last Post: deanhystad
  Empty response to request causing .json() to error t4keheart 1 9,953 Jun-26-2020, 08:35 PM
Last Post: bowlofred
  json.dumps list output qurr 12 5,076 Apr-08-2020, 10:13 PM
Last Post: micseydel

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020