Python Forum
With Python I cannot calculate an AWS signature for Rest APIs
Thread Rating:
  • 1 Vote(s) - 4 Average
  • 1
  • 2
  • 3
  • 4
  • 5
With Python I cannot calculate an AWS signature for Rest APIs
#1
I have never been able to get Rest APIs to completely work with AWS. The error messages I have seen have been about the time not being correct or the command not being recognized (e.g., list-users). I have verified the "version" was appropriate for the corresponding command with AWS's website documentation.

I am trying to use curl with Linux to list the users or instances in my AWS account. I have a problem when I run it. My current error, that I would like to focus on, is "request signatures calculated does not match the signature provided." I went through the process of creating a signature carefully. It wasn't that surprising that it did not work given the hours of trouble and the seemingly many potential pitfalls in the tedious task of creating a signature.

I used this link to generate the hexadecimal string for the signature: http://docs.aws.amazon.com/general/lates...les-python

Here is the code:
def sign(key, msg):
    return hmac.new(key, msg.encode("utf-8"), hashlib.sha256).digest()

def getSignatureKey(key, dateStamp, regionName, serviceName):
    kDate = sign(("AWS4" + key).encode("utf-8"), dateStamp)
    kRegion = sign(kDate, regionName)
    kService = sign(kRegion, serviceName)
    kSigning = sign(kService, "aws4_request")
    return kSigning
I analyzed the output of the signatureKey using a modification of the Python code in the above link. The result is a string that is not hexadecimal nor alphanumeric. The result is a combination of special non-alphabet, non-numeric symbols and very few alphabet letters. I tried to work around this problem by using import binascii and binascii.hexlify (to convert the garbled string into a hexadecimal). I was able to get a hexadecimal string from otherwise strictly adhering to the sample of Python code from the above link. I tend to think my signatureKey is not right because of this binascii work that I had to do. But what did I do wrong? How is that Python code in the link above supposed to calculate a hexadecimal signature? For me it create something very different.

Within 10 minutes of posting, I did not have the opportunity to edit this post. The hyperlink that is on the first three paragraphs should be totally deleted.  Can the moderators do this?
Reply
#2
please print the return value.
don't forget that utf-8 utilizes from 1 to four 8 bit bytes to represent each character

see the following https://en.wikipedia.org/wiki/UTF-8
Reply
#3
For various reasons, I will not be able to post the return value. The return value (before I change it to a hexadecimal) has circumflexes etc. It appears on a few different lines not just one.

Moderators: Can the linked, first two words on the original post be removed from the original post?
Reply
#4
i use botocore for AWS.  works great for me.  here is my list-instances command. this is a newer script that should have no tabs to mess up the indenting but just in case i also put a copy on my website at:

http://stratusrelay.com/phil/list-instances.py.txt
http://stratusrelay.com/phil/list-instances.py (c0a9dbaad45c44a2896e013390130228)

this script also uses multiprocessing to get info from each region in parallel (faster that way).  the command line arguments are the regions to be listed, defaulting to all regions.  if you want to actually run this script you will need the aws_regions module and set up your own .boto file.

http://stratusrelay.com/phil/aws_regions.py.txt
http://stratusrelay.com/phil/aws_regions.py (369f3bd36c65529f5fcbcbb12ed2d826)

#!/usr/bin/env python
#-----------------------------------------------------------------------------
# Copyright (C) 2016, by Phil D. Howard - all other rights reserved
# 
# Permission to use, copy, modify, and/or distribute this software for any
# purpose with or without fee is hereby granted, provided that the above
# copyright notice and this permission notice appear in all copies.
# 
# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
# WHATSOEVER RESULTING FROM LOSS OF USE, DATA, OR PROFITS, WHETHER IN AN
# ACTION OF CONTRACT, NEGLIGENCE, OR OTHER TORTIOUS ACTION, ARISING OUT OF
# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
# 
# The author may be contacted using the short first name, middle initial,
# and last name, separated by periods, with gmail dot com as the host part
# of an email address.
#-----------------------------------------------------------------------------
# audience      linux and unix users
# script        list-instances.py
# purpose       show instances
# syntax        python list-instances.py [ region ... ]
# alternate     python list-running-instances.py [ region ... ]
# note          copy or symlink so the command for list-running-instances.py runs list-instances.py to list only running (and pending) instances
#-----------------------------------------------------------------------------
from __future__ import division, print_function
"""
show instances
"""

import botocore.session

from aws_regions     import aws_regions_list_long, aws_regions_any_to_long
from multiprocessing import Pipe, Process
from os              import getpid, remove
from pickle          import dump, load
from pyutils         import eprint, plural

pipe   = None
region = None
run    = None

#-----------------------------------------------------------------------------
# function      print_instance
# purpose       print one line describing one instance
#-----------------------------------------------------------------------------
def print_instance( region, instance ):
    if instance == None:
        return None
    s = region + ' ' + instance['InstanceId']
    n = region + '_'
    if 'State' in instance:
        s += ' '
        s += instance['State']['Name']
    if 'ImageId' in instance:
        s += ' '
        s += instance['ImageId']
        n += instance['ImageId']
    if 'Placement' in instance:
        if 'AvailabilityZone' in instance['Placement']:
            s += ' '
            s += instance['Placement']['AvailabilityZone'][-1]
    if 'InstanceType' in instance:
        s += ' '
        s += instance['InstanceType']
    if 'VpcId' in instance:
        s += ' '
        s += instance['VpcId']
    if 'PublicIpAddress' in instance:
        s += ' '
        s += instance['PublicIpAddress']
    if 'PrivateIpAddress' in instance:
        s += ' ( '
        s += instance['PrivateIpAddress']
        s += ' )'
    if 'Tags' in instance:
        tags = sorted([(tag['Key'], tag['Value']) for tag in instance['Tags']],cmptag)
        for tag in tags:
            s += ' '
            s += tag[0]
            s += '='
            s += repr(tag[1])
    try:
        print( s )
    except IOError:
        pass
    return s

#-----------------------------------------------------------------------------
# function      cmptag
# purpose       compare logic for sort() so 'Name' collates lowest
#-----------------------------------------------------------------------------
def cmptag(a,b):
    if a == 'Name':
        return -1
    if b == 'Name':
        return 1
    if a < b:
        return -1
    if a > b:
        return 1
    return 0

#-----------------------------------------------------------------------------
# function    lookup
# purpose     run as a child process per region to get instance info
#             pickle the info from the describe_instances method
#             pipe this info to the parent using pickle format
# note        the pipe is inherited from the parent
#-----------------------------------------------------------------------------
def lookup():
    session = botocore.session.get_session()
    client = session.create_client( 'ec2', region_name=region )
    paginator = client.get_paginator( 'describe_instances' )
    page_iterator = paginator.paginate()
    pipe[0].close()
    for page in page_iterator:
        for resv in page[ 'Reservations' ]:
            for instance in resv[ 'Instances' ]:
                if run and 'State' in instance:
                   if instance['State']['Name'][1:] != 'unning':
                       continue
                pipe[1].send( instance )
    pipe[1].send( None )
    pipe[1].close() # send EOF
    return

#-----------------------------------------------------------------------------
# function      main
#-----------------------------------------------------------------------------
def main( args ):
    global pipe, region, run # passing as globals instead of args

    errors = 0

    # arguments are regions
    region_list = []
    
    if 'run' in args[0]:
        run = True
    else:
        run = False

#-----------------------------------------------------------------------------
    for arg in args[1:]:
        if arg in ('-r','+r','--run','++run'):
            run = True
            continue
        if arg.lower() in aws_regions_any_to_long:
            region = aws_regions_any_to_long[ arg.lower() ]
            if region in region_list:
                print( 'region', region, 'already specified', file=stderr )
                errors += 1
            else:
                region_list += [ region ]
        else:
            print( 'argument', repr(arg), 'is not a valid region', file=stderr )
            errors += 1

    if errors > 0:
        return 'aborting due to '+repr( errors )+'error'+plural( errors )

    if len( region_list ) == 0:
        region_list = list( aws_regions_list_long )

    region_list.sort()
    region_count = len( region_list )

#-----------------------------------------------------------------------------
# setup, record, and start all the processes
#-----------------------------------------------------------------------------
    plist = []
    for region in region_list:
        pipe = Pipe( False )
        process = Process( target = lookup ) # child inherits pipe and region, so no args needed.
        plist += [ ( region, process, pipe ) ]
        process.start()

#-----------------------------------------------------------------------------
# read all the results of each process.
# so what if we delay on the first ones, we have to wait on all the data.
#-----------------------------------------------------------------------------
    for this_region, this_process, this_pipe in plist:
        pipe[1].close() # close the parent copy of the send side of the pipe.
        while True:
            try:
                if print_instance( this_region, this_pipe[0].recv() ) == None: break
            except EOFError:
                break

#-----------------------------------------------------------------------------
# wait up to 6 minutes for all of them to finish
#-----------------------------------------------------------------------------
    for this_region, this_process, this_pipe in plist:
        this_process.join( 360 )

#-----------------------------------------------------------------------------
# all done
#-----------------------------------------------------------------------------
    return 0

#-----------------------------------------------------------------------------
if __name__ == '__main__':
    from sys import argv, stderr, stdout
    result = main( argv )
    stdout.flush()
    try:
        exit( int( result ) )
    except ValueError:
        print( str( result ), file=stderr )
        exit( 1 )
    except TypeError:
        if result == None:
            exit( 0 )
        exit( 254 )
    except:
        exit( 255 )

#-----------------------------------------------------------------------------
# EOF

and i attached those files

oh... and this also needs pyutils

http://stratusrelay.com/free/pyutils.py.txt
http://stratusrelay.com/free/pyutils.py (76d7e080645a69fceb38526eee192db1)

nice... separate attachments are merged.  i hope they transfer correctly.

i downloaded the attachments and the md5 checksums match, so it looks like attachments do norget corrupted.

Attached Files

.py   list-instances.py (Size: 8.02 KB / Downloads: 324)
.py   aws_regions.py (Size: 5.3 KB / Downloads: 349)
.py   pyutils.py (Size: 13.48 KB / Downloads: 322)
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply
#5
Moderators: Can the linked, first two words on the original post be removed from the original post?
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Byte Error when working with APIs Oshadha 2 980 Jul-05-2022, 05:23 AM
Last Post: deanhystad
  Calling Oracle REST SQL from Python johnjacob 2 1,993 Nov-05-2020, 04:19 AM
Last Post: johnjacob
  Parse a REST API call using Python GKT 1 1,874 May-07-2020, 04:15 AM
Last Post: buran
  Remove Email Signature NewBeie 4 9,247 Jan-01-2020, 06:44 PM
Last Post: PythonPaul2016
  Connecting 2 APIs kamaleon 2 2,007 Dec-27-2019, 08:06 AM
Last Post: kamaleon
  Raspberry Pi Python Rest Post madaxe 0 1,329 Nov-01-2019, 07:30 PM
Last Post: madaxe
  How to get valid error message while invoking REST API through Python gollupraveen 0 2,049 Oct-04-2019, 07:15 PM
Last Post: gollupraveen
  [cryptography.io] How to convert DER signature to ECDSA fstefanov 1 2,986 Jul-04-2019, 08:59 AM
Last Post: fstefanov
  Signature verification saisankalpj 19 8,008 Nov-22-2018, 01:55 PM
Last Post: saisankalpj
  Signature verification saisankalpj 8 5,163 Nov-20-2018, 09:32 AM
Last Post: saisankalpj

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020