Python Forum
CSV file created is huge in size. How to reduce the size?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
CSV file created is huge in size. How to reduce the size?
#1
I have the below code used to create a CSV file:

import csv
srcFile = open("C:\Users\XXXX\Expenses.csv","r")
data = srcFile.read()

srcList = data.split("\n")

colNames = srcList[0].split(",")
header = srcList[1]

sheetData = srcList[2:]
final_data = []
expense_data = []

for row in sheetData:
    split_row = row.split(",")
    final_data.append(split_row)

for row in range(0, len(final_data)-1):
    for col in range(3, 30):
        loc = final_data[row][0]
        opsheet = final_data[row][1]
        rowNum = final_data[row][2]
        colName = colNames[col]

        str1 = "='\\\xxxxxxx\xxxx\xxx\[APR_"
        str2 = "_DATA.xlsm]"
        str3 = "\'!"
        str4 = "$"
        finalStr = str1 + loc + str2 + opsheet + str3 + colName + str4 + rowNum
        final_data[row][col] = ""

with open("c:\users\xxxxx\new_expenses.csv", "wb") as f:
    fwriter = csv.writer(f)
    fwriter.writerows(final_data)
I'm a beginner in Python and I know this code is not up to the standards or a real way to code. But this code works fine and creates a CSV file.

Now the problem is this file is already created by someone else and I'm replicating it for other purposes with new data and everything. The existing file, created earlier, was only 30MB in size. But the file that is created in this method is 180MB and takes an enormous amount of time to open.

The original file is nothing but a single worksheet with 51,000 rows and 20 Columns each cell referencing to some cell in a workbook in a network location.

Any idea on how to reduce the size? or What is causing the huge?
Also as a beginner, any suggestions/feedback to improving the code is really appreciated. Thanks in advance!!
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  File is not being created with 'w' argument CAD79 3 412 Mar-14-2024, 12:05 PM
Last Post: snippsat
  Python code for alignment and font size 1418 0 303 Jan-14-2024, 03:56 AM
Last Post: 1418
Question [PyMuPDF] Grab all strings of a given size? Winfried 3 663 Dec-26-2023, 07:39 AM
Last Post: Pedroski55
  howto get size of a ctk image? janeik 2 839 Oct-03-2023, 03:49 AM
Last Post: janeik
  IndexError: index 10 is out of bounds for axis 0 with size 10 Mehboob 11 2,067 Sep-14-2023, 06:54 AM
Last Post: Mehboob
  How to do "fixed size" (wrapping) math in Python? AlexanderWulf 13 1,836 Jul-19-2023, 04:13 PM
Last Post: deanhystad
  Converted EXE file size is too large Rajasekaran 0 1,499 Mar-30-2023, 11:50 AM
Last Post: Rajasekaran
  reduce nested for-loops Phaze90 11 1,860 Mar-16-2023, 06:28 PM
Last Post: ndc85430
  help how to get size of pandas dataframe into MB\GB mg24 1 2,332 Jan-28-2023, 01:23 PM
Last Post: snippsat
  Adding values with reduce() function from the list of tuples kinimod 10 2,626 Jan-24-2023, 08:22 AM
Last Post: perfringo

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020