Python Forum
Importing a CSV loops to infinity in python
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Importing a CSV loops to infinity in python
#1
I've accidentally created an infinite loop importing a CSV file using python and importing it into MySQL. I don't see where the error is!

This is my code:

import os
import mysql.connector
import csv
import pandas
source = os.path.join('source_files', 'aws_bills', 'march-bill-original-2019.csv')
destination = os.path.join('output_files', 'aws_bills', 'aws-bill-2019-03.csv')
mydb = mysql.connector.connect(user='xxxx', password='xxxxx',
                            host='xxxxx',
                            database='aws_bill')
cursor = mydb.cursor()
    try:
        with open(source) as csv_file:
            csv_reader = csv.reader(csv_file, delimiter=',')
            next(csv_reader)
            sql = "INSERT INTO billing_info (InvoiceId, PayerAccountId, LinkedAccountId, RecordType, RecordId, ProductName, RateId, SubscriptionId, PricingPlanId, UsageType, Operation, AvailabilityZone, ReservedInstance, ItemDescription, UsageStartDate, UsageEndDate, UsageQuantity, BlendedRate, BlendedCost, UnBlendedRate, UnBlendedCost, ResourceId, Engagement, Name, Owner, Parent) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)"
            for row in csv_reader:
                print(row)
                try:
                    cursor.execute(sql.format(row))
                except Exception as e:
                    print("MySQL Exception: ", e)
                mydb.commit()
            print("Done importing data.")      
    except Exception as e:
        print("Exception:", e)
        mydb.rollback()
    finally:
        mydb.close()
The main field that I'm using to identify what is being inserted into the database is the one called LinkedAccountId.

It's a set of unique numbers that identify AWS accounts.

I'm verifying that the data is being inserted by logging into the database and watching it with this query:
select ID,LinkedAccountId from billing_info ORDER BY ID DESC LIMIT 25;
After a while of importing, I see that it starts again from the top of the CSV file. The LinkedAccountIds that were reported before start appearing again. Indicating that it has started over from the beginning of the file.

Why is this code looping infinitely???
Reply
#2
(May-03-2019, 03:19 PM)bluethundr Wrote: Why is this code looping infinitely???
it's not looping infinitely, but every time you run it, it starts from the top of the file (i.e. imports records alraedy imported in previous runs.
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply
#3
(May-03-2019, 03:25 PM)buran Wrote:
(May-03-2019, 03:19 PM)bluethundr Wrote: Why is this code looping infinitely???
it's not looping infinitely, but every time you run it, it starts from the top of the file (i.e. imports records alraedy imported in previous runs.
To clarify, these are not separate runs of the script. I run the script once. It takes a while to import the data. Then after a while I start seeing the account ID's from the top of the file get inserted into the database again. Also the script is printing out each row as it inserts. And I see the IDs repeating there too.

I need to find out why that happens and correct it.

Sigh... it just happened again! I was running the script (once) and waiting for the file to import. Then after a while I start seeing this number that was input into the database previously: 051170381115. It's very frustrating! Just don't see why this file is looping this way.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  python standard way of importing library mg24 1 872 Nov-15-2022, 01:41 AM
Last Post: deanhystad
  Why both loops don't work in python? nau 3 1,055 Sep-21-2022, 02:17 PM
Last Post: rob101
Star Infinity loop edek121 0 18,109 Jun-05-2022, 12:51 PM
Last Post: edek121
  int infinity Skaperen 2 1,816 Sep-13-2021, 04:42 AM
Last Post: Skaperen
  Why does Python not use parenthesis to contain loops? davidorlow 3 3,399 Jun-09-2021, 06:33 AM
Last Post: Gribouillis
  importing a list of numbers into python script barrypyth 8 4,436 Aug-22-2020, 09:10 PM
Last Post: barrypyth
  Importing python data to Textfile or CSV yanDvator 0 1,720 Aug-02-2020, 06:58 AM
Last Post: yanDvator
  importing a CSV file into Python russoj5 1 2,918 Aug-02-2020, 12:03 AM
Last Post: scidam
  Python for loops Kristenl2784 3 41,874 Jun-16-2020, 06:01 PM
Last Post: Yoriz
  For loops in python Boinbo 3 68,523 Apr-18-2020, 01:23 AM
Last Post: buran

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020