Python Forum
Dividing certain columns in multiple CSV files
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Dividing certain columns in multiple CSV files
#1
I'm trying to divide all data in certain columns from multiple CSV files using python. I want to divide all columns with certain values except the first one.

This is what I wrote so far:

import os
import csv
import argparse

ap = argparse.ArgumentParser()
ap.add_argument("-i", "--input", required=True)
ap.add_argument("-o", "--output", required=True)
args = vars(ap.parse_args())


if os.path.exists(args["output"]) and os.path.isdir(args["output"]):
        print("Writing to {}".format(args["output"]))
else:
        print("Cannot write to directory {}".format(args["output"]))
        exit()

for file in os.listdir(args["input"]):
    if file.endswith(".csv"):
        print("{} ...".format(file))
        with open(os.path.join(args["input"],file), 'r', newline='') as infile, open(os.path.join(args["output"], file), 'w', newline='') as outfile:
            reader = csv.reader(infile)
            writer = csv.writer(outfile)
            rows = list(reader)
            data = rows[0:]
            def div(this_row):
                return [this_row[0], float(this_row[1])/1920, float(this_row[2])/560, float(this_row[3])/1920, float(this_row[4])/560]
            new_data = [div(row) for row in data]
            new_rows = new_data
            for row in new_rows:
                writer.writerow(row)
I get an error:
Error:
IndexError: list index out of range
Is it problem in different number of rows in CSV files?
Reply
#2
Always show the complete unaltered error traceback as it contains valuable debugging information.
cannot determine line number with this limited error.
Reply
#3
Each row in each file needs to have at least 5 items in it. Otherwise you will get that error. Also, next time please post the full text of the error message. There is useful information in the part you didn't post.
Craig "Ichabod" O'Brien - xenomind.com
I wish you happiness.
Recommended Tutorials: BBCode, functions, classes, text adventures
Reply
#4
I had empty rows in some csv files. Removed them with this code:
import os
import argparse

ap = argparse.ArgumentParser()
ap.add_argument("-i", "--input", required=True)
ap.add_argument("-o", "--output", required=True)
args = vars(ap.parse_args())

if os.path.exists(args["output"]) and os.path.isdir(args["output"]):
        print("Writing to {}".format(args["output"]))
else:
        print("Cannot write to directory {}".format(args["output"]))
        exit()

for file in os.listdir(args["input"]):
    if file.endswith(".csv"):
        print("{} ...".format(file))
        with open(os.path.join(args["input"],file), 'r') as infile, open(os.path.join(args["output"], file), 'w') as outfile:
            non_blank = (line for line in infile if line.strip())
            outfile.writelines(non_blank)
I tried it again and it worked. SOLVED
Reply
#5
You could do that at the same time you are processing the files:

new_data = [div(row) for row in data if row.strip()]
Craig "Ichabod" O'Brien - xenomind.com
I wish you happiness.
Recommended Tutorials: BBCode, functions, classes, text adventures
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  python convert multiple files to multiple lists MCL169 6 1,436 Nov-25-2023, 05:31 AM
Last Post: Iqratech
  splitting file into multiple files by searching for string AlphaInc 2 816 Jul-01-2023, 10:35 PM
Last Post: Pedroski55
  How to check multiple columns value within range SamLiu 2 1,103 Mar-13-2023, 09:32 AM
Last Post: SamLiu
  Merging multiple csv files with same X,Y,Z in each Auz_Pete 3 1,087 Feb-21-2023, 04:21 AM
Last Post: Auz_Pete
  unittest generates multiple files for each of my test case, how do I change to 1 file zsousa 0 918 Feb-15-2023, 05:34 PM
Last Post: zsousa
  Find duplicate files in multiple directories Pavel_47 9 2,926 Dec-27-2022, 04:47 PM
Last Post: deanhystad
  How to move multiple columns to initial position SriRajesh 4 1,375 Jul-02-2022, 10:34 AM
Last Post: deanhystad
  df column aggregate and group by multiple columns SriRajesh 0 977 May-06-2022, 02:26 PM
Last Post: SriRajesh
  Extract parts of multiple log-files and put it in a dataframe hasiro 4 2,022 Apr-27-2022, 12:44 PM
Last Post: hasiro
  Search multiple CSV files for a string or strings cubangt 7 7,842 Feb-23-2022, 12:53 AM
Last Post: Pedroski55

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020