Bottom Page

Thread Rating:
  • 1 Vote(s) - 4 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 Need help to correct my python function for fetching full data!
#1
I am trying to fetch product data from an api using Python3.6.

By default this api returns 20 products and in a single request the api can return max 500 products if we use api's parameter Limit=500.

So for fetching all products we need to use one more parameter with Limit- Offset(Number of products to skip).

I have written following function to achieve this but in case of full data my function is not working well and it's giving me error like- Login failed, Signature mismatching.

def get_data(userid, api_key, action, pagination=True):
timeformat = datetime.datetime.now().replace(microsecond=0).isoformat() + '+08:00'
endpoint = 'https://example.com'
page_json = {}
# set required parameters for this api
parameters = {
  'UserID': userid,
  'Version': '1.0',
  'Action': action,
  'Format': 'JSON',
  'Timestamp': timeformat
}
if pagination:
    page = 0
    parameters['Limit'] = 500
    while True:
        parameters['Offset'] = 500 * page
        # set the required cryptographic signature
        concatenated = urllib.parse.urlencode(sorted(parameters.items()))
        parameters['Signature'] = HMAC(api_key, concatenated.encode('utf-8'), sha256).hexdigest()
        page += 1
        try:
            response = requests.get(endpoint, params=parameters)
            page_json = response.json()
        except requests.exceptions.ConnectionError:
            print("Connection refused!")
            sleep(5)
else:
    try:
        concatenated = urllib.parse.urlencode(sorted(parameters.items()))
        # set the required cryptographic signature
        parameters['Signature'] = HMAC(api_key, concatenated.encode('utf-8'), sha256).hexdigest()
        response = requests.get(endpoint, params=parameters)
        page_json = response.json()
    except requests.exceptions.ConnectionError:
        print("Connection refused!")
        sleep(5)

return page_json
It looks like I am not fitting my signature parameter line correctly in case of full data.

Can you please look into my function and help me to find out what I have written wrong and what it should be like?
Quote
#2
Hi,

The 2 things I see there are that when you put pagination = True you enter in an endless loop and that you overwrite the results of the previous query.

I have not way to try it, specially confirm that the json object returned is a list of results and there is not more wrappers... but try this:
def get_data(userid, api_key, action, limit=500):
timeformat = datetime.datetime.now().replace(microsecond=0).isoformat() + '+08:00'
endpoint = 'https://example.com'
page_json = {}
# set required parameters for this api
parameters = {
  'UserID': userid,
  'Version': '1.0',
  'Action': action,
  'Format': 'JSON',
  'Timestamp': timeformat
}
if limit > 0:
    parameters['Limit'] = limit
    page = 0
    while True:
        parameters['Offset'] = limit * page
        # set the required cryptographic signature
        concatenated = urllib.parse.urlencode(sorted(parameters.items()))
        parameters['Signature'] = HMAC(api_key, concatenated.encode('utf-8'), sha256).hexdigest()
        try:
            response = requests.get(endpoint, params=parameters)
        except requests.exceptions.ConnectionError:
            print("Connection refused!")
            sleep(5)
            # As the page was not recovered, just try it again,
            # do not increase the page counter
            continue
        # Ok, one page recovered, append to the total result
        page += 1
        # Here I assume the answer is a dictionary of entries like:
        #  { 501: value, 502: value, ...}
        # you might need to look for the right element to accumulate.
        # i.e.: if response is something like:
        #   {'result': [value, value, value...]}
        # The following lines will not be valid
        delta = response.json()
        page_json.update(delta)
        # This might be the end, no more pages... but this criteria again depends
        # on the api you are using
        if len(delta) % limit == 0:
            # Here we enter only if the number of results are 0 or 500
            break
else:
    try:
        concatenated = urllib.parse.urlencode(sorted(parameters.items()))
        # set the required cryptographic signature
        parameters['Signature'] = HMAC(api_key, concatenated.encode('utf-8'), sha256).hexdigest()
        response = requests.get(endpoint, params=parameters)
        page_json = response.json()
    except requests.exceptions.ConnectionError:
        print("Connection refused!")
        sleep(5)
 
return page_json
Notice that I changed your pagination flag with the limit number (so you can request 50 by 50, 100 by 100...) use limit=0 for the equivalent of your pagination=False.
PrateekG likes this post
Quote
#3
Thanks killerrex!
I am able to find issue in my code.
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  Not quite getting the correct Output from a function twinpiques 3 217 Aug-04-2019, 11:53 PM
Last Post: twinpiques
  Fetching data from Sklearn zaki424160 2 574 Jul-09-2019, 05:20 AM
Last Post: ThomasL
  Help with finding correct topic in Python learning yahya01 1 336 Jun-06-2019, 05:01 PM
Last Post: buran
  Error while fetching data from PostgreSQL linu 3 444 May-13-2019, 02:38 PM
Last Post: rxndy
  Error while fetching data from PostgreSQL tuple indices must be integers or slices, n Sandy777 6 658 May-12-2019, 11:41 AM
Last Post: Sandy777
  numpynot associted with correct python version ErnestTBass 17 1,090 Jan-16-2019, 02:47 PM
Last Post: ErnestTBass
  Function not returning correct value ActualNoob 3 408 Jan-11-2019, 12:35 AM
Last Post: stullis
  Function to skip meta data in a .csv file using Python ajgardev 1 530 Jul-22-2018, 12:53 PM
Last Post: Larz60+
  Run function in parallel but inherite dynamic data from the previous function atizva 4 834 Jul-11-2018, 06:39 AM
Last Post: volcano63
  Delay in fetching kerberos tokens : python gssapi securitycontext unni10286 1 704 Apr-30-2018, 04:12 PM
Last Post: micseydel

Forum Jump:


Users browsing this thread: 1 Guest(s)