Python Forum
reading in csv row by row
Thread Rating:
  • 1 Vote(s) - 2 Average
  • 1
  • 2
  • 3
  • 4
  • 5
reading in csv row by row
#5
The csv-module is implemented in C. It's even faster (I guess).
But this does not matter. At the end it's nearly the same code.

For example, the use of the method readlines is not good.
What happens if the csv-file is 16 GiB big and you have only 8 GiB memory?
(of course by this size, you can't put the results in a list)

Here the last example:

data = open(csv_file, "r").readlines()
for row in data:
    row_list=row.strip().split(",")
    # code
Modified to consume lesser memory:

data = open(csv_file, "r")
for row in data:
    row_list=row.strip().split(",")
    # code
Now with a context manager:

with open(csv_file, "r") as data:
    for row in data:
        row_list=row.strip().split(",")
        # code
Now the code with csv.reader:


with open(csv_file, "r") as data:
    reader = csv.reader(data, delimiter=',')
    # header = next(reader) # this skips the first line
    for row in reader:
        # code
The difference is, that the csv.reader splits the row for you. It does more, but it's not always needed.

The fileobject is an Iterator. Iterating over the iterator yields line by line with line ending (\n).
The csv-object is an Iterator. Iterating over the iterator yields row by row, where the row is already splitted into columns.

The rest is just index access and appending to a list.
If you're unsure, what the code does, just print the row to console inside the loop.
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply


Messages In This Thread
reading in csv row by row - by SchroedingersLion - Jan-08-2019, 07:01 PM
RE: reading in csv row by row - by woooee - Jan-08-2019, 07:18 PM
RE: reading in csv row by row - by woooee - Jan-08-2019, 07:55 PM
RE: reading in csv row by row - by DeaD_EyE - Jan-08-2019, 10:17 PM
RE: reading in csv row by row - by Gribouillis - Jan-09-2019, 07:28 AM
RE: reading in csv row by row - by perfringo - Jan-09-2019, 07:42 AM
RE: reading in csv row by row - by woooee - Jan-09-2019, 06:26 PM
RE: reading in csv row by row - by perfringo - Jan-10-2019, 12:49 PM

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020