Python Forum
CSV file created is huge in size. How to reduce the size?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
CSV file created is huge in size. How to reduce the size?
#1
I have the below code used to create a CSV file:

import csv
srcFile = open("C:\Users\XXXX\Expenses.csv","r")
data = srcFile.read()

srcList = data.split("\n")

colNames = srcList[0].split(",")
header = srcList[1]

sheetData = srcList[2:]
final_data = []
expense_data = []

for row in sheetData:
    split_row = row.split(",")
    final_data.append(split_row)

for row in range(0, len(final_data)-1):
    for col in range(3, 30):
        loc = final_data[row][0]
        opsheet = final_data[row][1]
        rowNum = final_data[row][2]
        colName = colNames[col]

        str1 = "='\\\xxxxxxx\xxxx\xxx\[APR_"
        str2 = "_DATA.xlsm]"
        str3 = "\'!"
        str4 = "$"
        finalStr = str1 + loc + str2 + opsheet + str3 + colName + str4 + rowNum
        final_data[row][col] = ""

with open("c:\users\xxxxx\new_expenses.csv", "wb") as f:
    fwriter = csv.writer(f)
    fwriter.writerows(final_data)
I'm a beginner in Python and I know this code is not up to the standards or a real way to code. But this code works fine and creates a CSV file.

Now the problem is this file is already created by someone else and I'm replicating it for other purposes with new data and everything. The existing file, created earlier, was only 30MB in size. But the file that is created in this method is 180MB and takes an enormous amount of time to open.

The original file is nothing but a single worksheet with 51,000 rows and 20 Columns each cell referencing to some cell in a workbook in a network location.

Any idea on how to reduce the size? or What is causing the huge?
Also as a beginner, any suggestions/feedback to improving the code is really appreciated. Thanks in advance!!
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  How can I write formatted (i.e. bold, italic, change font size, etc.) text to a file? JohnJSal 12 27,705 Feb-13-2025, 04:48 AM
Last Post: tomhansky
Question [redistribution] Reduce number + size of dependencies? Winfried 2 562 Jan-31-2025, 10:17 PM
Last Post: snippsat
  Changing client.get() method type based on size of data... dl0dth 1 517 Jan-02-2025, 08:30 PM
Last Post: dl0dth
  Python 3.13 : def setup size HansieB 6 1,273 Dec-02-2024, 11:12 AM
Last Post: buran
  IndexError: index 31 is out of bounds for axis 0 with size 31 YL789 1 753 Sep-21-2024, 09:46 AM
Last Post: Gribouillis
  File is not being created with 'w' argument CAD79 3 1,458 Mar-14-2024, 12:05 PM
Last Post: snippsat
  Python code for alignment and font size 1418 0 908 Jan-14-2024, 03:56 AM
Last Post: 1418
Question [PyMuPDF] Grab all strings of a given size? Winfried 3 1,529 Dec-26-2023, 07:39 AM
Last Post: Pedroski55
  howto get size of a ctk image? janeik 2 2,043 Oct-03-2023, 03:49 AM
Last Post: janeik
  IndexError: index 10 is out of bounds for axis 0 with size 10 Mehboob 11 8,612 Sep-14-2023, 06:54 AM
Last Post: Mehboob

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020