With Python I cannot calculate an AWS signature for Rest APIs - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: General Coding Help (https://python-forum.io/forum-8.html) +--- Thread: With Python I cannot calculate an AWS signature for Rest APIs (/thread-332.html) |
With Python I cannot calculate an AWS signature for Rest APIs - Johno - Oct-06-2016 I have never been able to get Rest APIs to completely work with AWS. The error messages I have seen have been about the time not being correct or the command not being recognized (e.g., list-users). I have verified the "version" was appropriate for the corresponding command with AWS's website documentation. I am trying to use curl with Linux to list the users or instances in my AWS account. I have a problem when I run it. My current error, that I would like to focus on, is "request signatures calculated does not match the signature provided." I went through the process of creating a signature carefully. It wasn't that surprising that it did not work given the hours of trouble and the seemingly many potential pitfalls in the tedious task of creating a signature. I used this link to generate the hexadecimal string for the signature: http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html#signature-v4-examples-python Here is the code: def sign(key, msg): return hmac.new(key, msg.encode("utf-8"), hashlib.sha256).digest() def getSignatureKey(key, dateStamp, regionName, serviceName): kDate = sign(("AWS4" + key).encode("utf-8"), dateStamp) kRegion = sign(kDate, regionName) kService = sign(kRegion, serviceName) kSigning = sign(kService, "aws4_request") return kSigningI analyzed the output of the signatureKey using a modification of the Python code in the above link. The result is a string that is not hexadecimal nor alphanumeric. The result is a combination of special non-alphabet, non-numeric symbols and very few alphabet letters. I tried to work around this problem by using import binascii and binascii.hexlify (to convert the garbled string into a hexadecimal). I was able to get a hexadecimal string from otherwise strictly adhering to the sample of Python code from the above link. I tend to think my signatureKey is not right because of this binascii work that I had to do. But what did I do wrong? How is that Python code in the link above supposed to calculate a hexadecimal signature? For me it create something very different. Within 10 minutes of posting, I did not have the opportunity to edit this post. The hyperlink that is on the first three paragraphs should be totally deleted. Can the moderators do this? RE: With Python I cannot calculate an AWS signature for Rest APIs - Larz60+ - Oct-06-2016 please print the return value. don't forget that utf-8 utilizes from 1 to four 8 bit bytes to represent each character see the following https://en.wikipedia.org/wiki/UTF-8 RE: With Python I cannot calculate an AWS signature for Rest APIs - Johno - Oct-06-2016 For various reasons, I will not be able to post the return value. The return value (before I change it to a hexadecimal) has circumflexes etc. It appears on a few different lines not just one. Moderators: Can the linked, first two words on the original post be removed from the original post? RE: With Python I cannot calculate an AWS signature for Rest APIs - Skaperen - Oct-06-2016 i use botocore for AWS. works great for me. here is my list-instances command. this is a newer script that should have no tabs to mess up the indenting but just in case i also put a copy on my website at: http://stratusrelay.com/phil/list-instances.py.txt http://stratusrelay.com/phil/list-instances.py (c0a9dbaad45c44a2896e013390130228) this script also uses multiprocessing to get info from each region in parallel (faster that way). the command line arguments are the regions to be listed, defaulting to all regions. if you want to actually run this script you will need the aws_regions module and set up your own .boto file. http://stratusrelay.com/phil/aws_regions.py.txt http://stratusrelay.com/phil/aws_regions.py (369f3bd36c65529f5fcbcbb12ed2d826) #!/usr/bin/env python #----------------------------------------------------------------------------- # Copyright (C) 2016, by Phil D. Howard - all other rights reserved # # Permission to use, copy, modify, and/or distribute this software for any # purpose with or without fee is hereby granted, provided that the above # copyright notice and this permission notice appear in all copies. # # THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES # WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF # MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR # ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES # WHATSOEVER RESULTING FROM LOSS OF USE, DATA, OR PROFITS, WHETHER IN AN # ACTION OF CONTRACT, NEGLIGENCE, OR OTHER TORTIOUS ACTION, ARISING OUT OF # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. # # The author may be contacted using the short first name, middle initial, # and last name, separated by periods, with gmail dot com as the host part # of an email address. #----------------------------------------------------------------------------- # audience linux and unix users # script list-instances.py # purpose show instances # syntax python list-instances.py [ region ... ] # alternate python list-running-instances.py [ region ... ] # note copy or symlink so the command for list-running-instances.py runs list-instances.py to list only running (and pending) instances #----------------------------------------------------------------------------- from __future__ import division, print_function """ show instances """ import botocore.session from aws_regions import aws_regions_list_long, aws_regions_any_to_long from multiprocessing import Pipe, Process from os import getpid, remove from pickle import dump, load from pyutils import eprint, plural pipe = None region = None run = None #----------------------------------------------------------------------------- # function print_instance # purpose print one line describing one instance #----------------------------------------------------------------------------- def print_instance( region, instance ): if instance == None: return None s = region + ' ' + instance['InstanceId'] n = region + '_' if 'State' in instance: s += ' ' s += instance['State']['Name'] if 'ImageId' in instance: s += ' ' s += instance['ImageId'] n += instance['ImageId'] if 'Placement' in instance: if 'AvailabilityZone' in instance['Placement']: s += ' ' s += instance['Placement']['AvailabilityZone'][-1] if 'InstanceType' in instance: s += ' ' s += instance['InstanceType'] if 'VpcId' in instance: s += ' ' s += instance['VpcId'] if 'PublicIpAddress' in instance: s += ' ' s += instance['PublicIpAddress'] if 'PrivateIpAddress' in instance: s += ' ( ' s += instance['PrivateIpAddress'] s += ' )' if 'Tags' in instance: tags = sorted([(tag['Key'], tag['Value']) for tag in instance['Tags']],cmptag) for tag in tags: s += ' ' s += tag[0] s += '=' s += repr(tag[1]) try: print( s ) except IOError: pass return s #----------------------------------------------------------------------------- # function cmptag # purpose compare logic for sort() so 'Name' collates lowest #----------------------------------------------------------------------------- def cmptag(a,b): if a == 'Name': return -1 if b == 'Name': return 1 if a < b: return -1 if a > b: return 1 return 0 #----------------------------------------------------------------------------- # function lookup # purpose run as a child process per region to get instance info # pickle the info from the describe_instances method # pipe this info to the parent using pickle format # note the pipe is inherited from the parent #----------------------------------------------------------------------------- def lookup(): session = botocore.session.get_session() client = session.create_client( 'ec2', region_name=region ) paginator = client.get_paginator( 'describe_instances' ) page_iterator = paginator.paginate() pipe[0].close() for page in page_iterator: for resv in page[ 'Reservations' ]: for instance in resv[ 'Instances' ]: if run and 'State' in instance: if instance['State']['Name'][1:] != 'unning': continue pipe[1].send( instance ) pipe[1].send( None ) pipe[1].close() # send EOF return #----------------------------------------------------------------------------- # function main #----------------------------------------------------------------------------- def main( args ): global pipe, region, run # passing as globals instead of args errors = 0 # arguments are regions region_list = [] if 'run' in args[0]: run = True else: run = False #----------------------------------------------------------------------------- for arg in args[1:]: if arg in ('-r','+r','--run','++run'): run = True continue if arg.lower() in aws_regions_any_to_long: region = aws_regions_any_to_long[ arg.lower() ] if region in region_list: print( 'region', region, 'already specified', file=stderr ) errors += 1 else: region_list += [ region ] else: print( 'argument', repr(arg), 'is not a valid region', file=stderr ) errors += 1 if errors > 0: return 'aborting due to '+repr( errors )+'error'+plural( errors ) if len( region_list ) == 0: region_list = list( aws_regions_list_long ) region_list.sort() region_count = len( region_list ) #----------------------------------------------------------------------------- # setup, record, and start all the processes #----------------------------------------------------------------------------- plist = [] for region in region_list: pipe = Pipe( False ) process = Process( target = lookup ) # child inherits pipe and region, so no args needed. plist += [ ( region, process, pipe ) ] process.start() #----------------------------------------------------------------------------- # read all the results of each process. # so what if we delay on the first ones, we have to wait on all the data. #----------------------------------------------------------------------------- for this_region, this_process, this_pipe in plist: pipe[1].close() # close the parent copy of the send side of the pipe. while True: try: if print_instance( this_region, this_pipe[0].recv() ) == None: break except EOFError: break #----------------------------------------------------------------------------- # wait up to 6 minutes for all of them to finish #----------------------------------------------------------------------------- for this_region, this_process, this_pipe in plist: this_process.join( 360 ) #----------------------------------------------------------------------------- # all done #----------------------------------------------------------------------------- return 0 #----------------------------------------------------------------------------- if __name__ == '__main__': from sys import argv, stderr, stdout result = main( argv ) stdout.flush() try: exit( int( result ) ) except ValueError: print( str( result ), file=stderr ) exit( 1 ) except TypeError: if result == None: exit( 0 ) exit( 254 ) except: exit( 255 ) #----------------------------------------------------------------------------- # EOF and i attached those files oh... and this also needs pyutils http://stratusrelay.com/free/pyutils.py.txt http://stratusrelay.com/free/pyutils.py (76d7e080645a69fceb38526eee192db1) nice... separate attachments are merged. i hope they transfer correctly. i downloaded the attachments and the md5 checksums match, so it looks like attachments do norget corrupted. RE: With Python I cannot calculate an AWS signature for Rest APIs - Johno - Oct-06-2016 Moderators: Can the linked, first two words on the original post be removed from the original post? |