Python Forum
Python function returns inconsistent results
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Python function returns inconsistent results
#1
I wrote a script that lists EC2 instances in Amazon Web Services. It writes the results to confluence. But it's behaving oddly.

I'm on windows 10.

The first time you run it from the command line it reports the correct total of servers in EC2 (can be verified in the AWS console):

    ----------------------------------------------------------
    There are: 51 EC2 instances in AWS Account: company-lab.
    ----------------------------------------------------------
The next time it's run with ABSOLUTELY NOTHING changed it reports this total that doubles the amount:

    ----------------------------------------------------------
    There are: 102 EC2 instances in AWS Account: company-lab.
    ----------------------------------------------------------
You literally just up arrow the command and it doubles the results. When it writes to confluence you can see duplicate servers listed. It does that each time you up arrow to run it again, with the same incorrect total (102 servers).

If you close the powershell command line and open again it's back to reporting the correct result (51 servers) which corresponds to what you see in the AWS console. Run it again in the new powershell and it doubles the amount and repeats that each subsequent time you run it.

This is the function that lists the instances and this is where I think the trouble is, but I'm having trouble finding it:

def list_instances(aws_account,aws_account_number, interactive, regions, fieldnames, show_details):
    today, aws_env_list, output_file, output_file_name, fieldnames = initialize(interactive, aws_account)
    options = arguments()
    instance_list = ''
    session = ''
    ec2 = ''
    account_found = ''
    PrivateDNS = None
    block_device_list = None
    instance_count = 0
    account_type_message = ''
    profile_missing_message = ''
    region = ''
    # Set the ec2 dictionary
    ec2info = {}
    # Write the file headers
    if interactive == 1:
        with open(output_file, mode='w+') as csv_file:
            writer = csv.DictWriter(csv_file, fieldnames=fieldnames, delimiter=',', lineterminator='\n')
            writer.writeheader()
    if 'gov' in aws_account and not 'admin' in aws_account:
        try:
            session = boto3.Session(profile_name=aws_account,region_name=region)
            account_found = 'yes'
        except botocore.exceptions.ProfileNotFound as e:
            message = f"An exception has occurred: {e}"
            account_found = 'no'
            banner(message)
    else:
        try:
            session = boto3.Session(profile_name=aws_account,region_name=region)
            account_found = 'yes'
        except botocore.exceptions.ProfileNotFound as e:
            message = f"An exception has occurred: {e}"
            account_found = 'no'
            banner(message)

    print(Fore.CYAN)      
    report_gov_or_comm(aws_account, account_found)
    print(Fore.RESET)
    for region in regions:
        if 'gov' in aws_account and not 'admin' in aws_account:
            try:
                session = boto3.Session(profile_name=aws_account,region_name=region)
            except botocore.exceptions.ProfileNotFound as e:
                profile_missing_message = f"An exception has occurred: {e}"
                account_found = 'no'
                pass
        else:
            try:
                session = boto3.Session(profile_name=aws_account,region_name=region)
                account_found = 'yes'
            except botocore.exceptions.ProfileNotFound as e:
                profile_missing_message = f"An exception has occurred: {e}"
                pass
        try:
            ec2 = session.client("ec2")
        except Exception as e:
            pass
        # Loop through the instances
        try:
            instance_list = ec2.describe_instances()
        except Exception as e:
                pass
        try:
            for reservation in instance_list["Reservations"]:
                for instance in reservation.get("Instances", []):
                    instance_count = instance_count + 1
                    launch_time = instance["LaunchTime"]
                    launch_time_friendly = launch_time.strftime("%B %d %Y")
                    tree = objectpath.Tree(instance)
                    block_devices = set(tree.execute('$..BlockDeviceMappings[\'Ebs\'][\'VolumeId\']'))
                    if block_devices:
                        block_devices = list(block_devices)
                        block_devices = str(block_devices).replace('[','').replace(']','').replace('\'','')
                    else:
                        block_devices = None
                    private_ips =  set(tree.execute('$..PrivateIpAddress'))
                    if private_ips:
                        private_ips_list = list(private_ips)
                        private_ips_list = str(private_ips_list).replace('[','').replace(']','').replace('\'','')
                    else:
                        private_ips_list = None
                    type(private_ips_list)
                    public_ips =  set(tree.execute('$..PublicIp'))
                    if len(public_ips) == 0:
                        public_ips = None
                    if public_ips:
                        public_ips_list = list(public_ips)
                        public_ips_list = str(public_ips_list).replace('[','').replace(']','').replace('\'','')
                    else:
                        public_ips_list = None
                    if 'KeyName' in instance:
                        key_name = instance['KeyName']
                    else:
                        key_name = None
                    name = None
                    if 'Tags' in instance:
                        try:
                            tags = instance['Tags']
                            name = None
                            for tag in tags:
                                if tag["Key"] == "Name":
                                    name = tag["Value"]
                                if tag["Key"] == "Engagement" or tag["Key"] == "Engagement Code":
                                    engagement = tag["Value"]
                        except ValueError:
                           # print("Instance: %s has no tags" % instance_id)
                           pass
                    if 'VpcId' in instance:
                        vpc_id = instance['VpcId']
                    else:
                        vpc_id = None
                    if 'PrivateDnsName' in instance:
                        private_dns = instance['PrivateDnsName']
                    else:
                        private_dns = None
                    if 'Platform' in instance:
                        platform = instance['Platform']
                    else:
                        platform = None
                    print(f"Platform: {platform}")
                    ec2info[instance['InstanceId']] = {
                        'AWS Account': aws_account,
                        'Account Number': aws_account_number,
                        'Name': name,
                        'Instance ID': instance['InstanceId'],
                        'Volumes': block_devices,
                        'Private IP': private_ips_list,
                        'Public IP': public_ips_list,
                        'Private DNS': private_dns,
                        'Availability Zone': instance['Placement']['AvailabilityZone'],
                        'VPC ID': vpc_id,
                        'Type': instance['InstanceType'],
                        'Platform': platform,
                        'Key Pair Name': key_name,
                        'State': instance['State']['Name'],
                        'Launch Date': launch_time_friendly
                    }
                    with open(output_file,'a') as csv_file:
                        writer = csv.DictWriter(csv_file, fieldnames=fieldnames, delimiter=',', lineterminator='\n')
                        writer.writerow({'AWS Account': aws_account, "Account Number": aws_account_number, 'Name': name, 'Instance ID': instance["InstanceId"], 'Volumes': block_devices,  'Private IP': private_ips_list, 'Public IP': public_ips_list, 'Private DNS': private_dns, 'Availability Zone': instance['Placement']['AvailabilityZone'], 'VPC ID': vpc_id, 'Type': instance["InstanceType"], 'Platform': platform, 'Key Pair Name': key_name, 'State': instance["State"]["Name"], 'Launch Date': launch_time_friendly})

                    if show_details == 'y' or show_details == 'yes':
                        for instance_id, instance in ec2info.items():
                            if account_found == 'yes':
                                print(Fore.RESET + "-------------------------------------")
                                for key in [
                                    'AWS Account',
                                    'Account Number',
                                    'Name',
                                    'Instance ID',
                                    'Volumes',
                                    'Private IP',
                                    'Public IP',
                                    'Private DNS',
                                    'Availability Zone',
                                    'VPC ID',
                                    'Type',
                                    'Platform',
                                    'Key Pair Name',
                                    'State',
                                    'Launch Date'
                                ]:
                                    print(Fore.GREEN + f"{key}: {instance.get(key)}")
                                print(Fore.RESET + "-------------------------------------")
                        else:
                            pass
                    ec2info = {}
                    with open(output_file,'a') as csv_file:
                        csv_file.close()
        except Exception as e:
            pass
    if profile_missing_message == '*':
        banner(profile_missing_message)
    print(Fore.GREEN)
    report_instance_stats(instance_count, aws_account, account_found)
    print(Fore.RESET + '\n')
    return output_file
This is a paste of the whole code for context: aws_ec2_list_instances.py


What the heck is happening, here? Why is it doing that and how do I correct the problem?
Reply
#2
If you run it a third time, does it triple? What happens if you wait a couple minutes before running it again? The only way I see it doubling the number of instances rest on line 66. If there are multiple, identical Reservations; it would count duplicate instances. Trying printing the instance_list before the loop and run the script a couple of times.

On a side note, there are some code blocks here that can be simplified. In several places, the following pattern occurs:

if 'KeyName' in instance:
    key_name = instance['KeyName']
else:
    key_name = None
This effectively does the same thing as dict.get() with a default value included:

key_name = instance.get('KeyName', None)
There are others too, but that one stood out the most.
Reply
#3
(Dec-20-2019, 04:00 PM)stullis Wrote: If you run it a third time, does it triple? What happens if you wait a couple minutes before running it again? The only way I see it doubling the number of instances rest on line 66. If there are multiple, identical Reservations; it would count duplicate instances. Trying printing the instance_list before the loop and run the script a couple of times.

On a side note, there are some code blocks here that can be simplified. In several places, the following pattern occurs:

if 'KeyName' in instance:
    key_name = instance['KeyName']
else:
    key_name = None
This effectively does the same thing as dict.get() with a default value included:

key_name = instance.get('KeyName', None)
There are others too, but that one stood out the most.

Ok thanks for your response! No, running it a third time, it reports the same number as before. It does that each time it's run. The first time it's run it reports correctly that there are 51 instances. Each time you up arrow and run it again it reports there are 102 (doubled).

I tried printing out the instance_list and I see that it does print out twice. Does that mean that there are two identical reservations? If so how can I deduplicate those results? Thanks for the tip on how to simplify my code I will definitely do that!
Reply
#4
Hmm... I'll have to review the entire code again later to see if there's anything else that could be causing it.

To be clear, instance_list increased in size the second time the script ran, as compared to its size the first time?
Reply
#5
So, I'm not seeing anything in the code that would cause doubling except the data the comes from the server. There isn't a way for the count to carry over from one run of the script to the next.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
Bug New to coding, Using the zip() function to create Diret and getting weird results Shagamatula 6 1,362 Apr-09-2023, 02:35 PM
Last Post: Shagamatula
  Read csv file with inconsistent delimiter gracenz 2 1,140 Mar-27-2023, 08:59 PM
Last Post: deanhystad
  Inconsistent loop iteration behavior JonWayn 2 954 Dec-10-2022, 06:49 AM
Last Post: JonWayn
  function returns dataframe as list harum 2 1,336 Aug-13-2022, 08:27 PM
Last Post: rob101
  ValueError: Found input variables with inconsistent numbers of samples saoko 0 2,432 Jun-16-2022, 06:59 PM
Last Post: saoko
  function accepts infinite parameters and returns a graph with those values edencthompson 0 812 Jun-10-2022, 03:42 PM
Last Post: edencthompson
  Loop Dict with inconsistent Keys Personne 1 1,576 Feb-05-2022, 03:19 AM
Last Post: Larz60+
  Inconsistent counting / timing with threading rantwhy 1 1,719 Nov-24-2021, 04:04 AM
Last Post: deanhystad
  Inconsistent behaviour in output - web scraping Steve 6 2,445 Sep-20-2021, 01:54 AM
Last Post: Larz60+
  Found input variables with inconsistent numbers of samples: [1000, 200] jenya56 2 2,806 Sep-15-2021, 12:48 PM
Last Post: jenya56

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020