May-26-2018, 12:35 PM
Hi,
The 2 things I see there are that when you put pagination = True you enter in an endless loop and that you overwrite the results of the previous query.
I have not way to try it, specially confirm that the json object returned is a list of results and there is not more wrappers... but try this:
The 2 things I see there are that when you put pagination = True you enter in an endless loop and that you overwrite the results of the previous query.
I have not way to try it, specially confirm that the json object returned is a list of results and there is not more wrappers... but try this:
def get_data(userid, api_key, action, limit=500): timeformat = datetime.datetime.now().replace(microsecond=0).isoformat() + '+08:00' endpoint = 'https://example.com' page_json = {} # set required parameters for this api parameters = { 'UserID': userid, 'Version': '1.0', 'Action': action, 'Format': 'JSON', 'Timestamp': timeformat } if limit > 0: parameters['Limit'] = limit page = 0 while True: parameters['Offset'] = limit * page # set the required cryptographic signature concatenated = urllib.parse.urlencode(sorted(parameters.items())) parameters['Signature'] = HMAC(api_key, concatenated.encode('utf-8'), sha256).hexdigest() try: response = requests.get(endpoint, params=parameters) except requests.exceptions.ConnectionError: print("Connection refused!") sleep(5) # As the page was not recovered, just try it again, # do not increase the page counter continue # Ok, one page recovered, append to the total result page += 1 # Here I assume the answer is a dictionary of entries like: # { 501: value, 502: value, ...} # you might need to look for the right element to accumulate. # i.e.: if response is something like: # {'result': [value, value, value...]} # The following lines will not be valid delta = response.json() page_json.update(delta) # This might be the end, no more pages... but this criteria again depends # on the api you are using if len(delta) % limit == 0: # Here we enter only if the number of results are 0 or 500 break else: try: concatenated = urllib.parse.urlencode(sorted(parameters.items())) # set the required cryptographic signature parameters['Signature'] = HMAC(api_key, concatenated.encode('utf-8'), sha256).hexdigest() response = requests.get(endpoint, params=parameters) page_json = response.json() except requests.exceptions.ConnectionError: print("Connection refused!") sleep(5) return page_jsonNotice that I changed your pagination flag with the limit number (so you can request 50 by 50, 100 by 100...) use limit=0 for the equivalent of your pagination=False.