Python Forum
reading in csv row by row
Thread Rating:
  • 1 Vote(s) - 2 Average
  • 1
  • 2
  • 3
  • 4
  • 5
reading in csv row by row
#1
Good evening,

can someone help me with the syntax in saving the rows of a csv files with M rows and N columns into a list?
I do not want the last column, but all of the M rows.
Thus, the list should look like an M x (N-1) matrix.
The last column should be saved to a Y list.

In my case, N=82 and M=26000.

import csv
import os

#give path to training / test data files
data_path='XXX'

#READ IN DATA 
x_training = []
y_training = []
print(os.getcwd())
with open(data_path) as csv_file:
    data = csv.reader(csv_file, delimiter=",")
    for row in data:
        for k in range(81):
            # now what? How do I write it :(
        y_training.append(float(row[-1]))
Regards!
Reply
#2
Using csv.reader only complicates this IMHO
## obviously we don't have this file, so this is untested code

x_training = []
y_training = []

data = open(csv_file, "r").readlines()
for row in data:
    row_list=row.strip().split(",")
    x_training.append(row_list[:-1])
    y_training.append(row_list[-1])  
Reply
#3
(Jan-08-2019, 07:18 PM)woooee Wrote: Using csv.reader only complicates this IMHO
## obviously we don't have this file, so this is untested code

x_training = []
y_training = []

data = open(csv_file, "r").readlines()
for row in data:
    row_list=row.strip().split(",")
    x_training.append(row_list[:-1])
    y_training.append(row_list[-1])  

Thank you, but could we do it with csv reader? I have learned to use it to read in stuff and I don't want to confuse myself even more with a different method.
Reply
#4
Quote:I don't want to confuse myself even more with a different method.
How does this make sense? You are confused with csv.reader, not the code that I submitted. If this is homework and you are required to use csv.reader then you should state so.
Reply
#5
The csv-module is implemented in C. It's even faster (I guess).
But this does not matter. At the end it's nearly the same code.

For example, the use of the method readlines is not good.
What happens if the csv-file is 16 GiB big and you have only 8 GiB memory?
(of course by this size, you can't put the results in a list)

Here the last example:

data = open(csv_file, "r").readlines()
for row in data:
    row_list=row.strip().split(",")
    # code
Modified to consume lesser memory:

data = open(csv_file, "r")
for row in data:
    row_list=row.strip().split(",")
    # code
Now with a context manager:

with open(csv_file, "r") as data:
    for row in data:
        row_list=row.strip().split(",")
        # code
Now the code with csv.reader:


with open(csv_file, "r") as data:
    reader = csv.reader(data, delimiter=',')
    # header = next(reader) # this skips the first line
    for row in reader:
        # code
The difference is, that the csv.reader splits the row for you. It does more, but it's not always needed.

The fileobject is an Iterator. Iterating over the iterator yields line by line with line ending (\n).
The csv-object is an Iterator. Iterating over the iterator yields row by row, where the row is already splitted into columns.

The rest is just index access and appending to a list.
If you're unsure, what the code does, just print the row to console inside the loop.
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply
#6
Guys, please...

I want to use csv reader to read in my data as stated. I am in a deep hurry and I don't have the time to think about the equivalency of different implementation methods. I have learned to use the .csv reader and I will stick to it for now.


with open(csv_file, "r") as data:
    reader = csv.reader(data, delimiter=',')
    # header = next(reader) # this skips the first line
    for row in reader:
        # code
So what then? The code stops at the part where I did not know what to do...
Reply
#7
SchroedingersLion Wrote:The code stops at the part where I did not know what to do...
Use woooee's last two lines.
Reply
#8
woooee has posted code what to do with rows in order to get needed data (slicing and appending).


PS - if you overstated your knowledge ("I have learned to use the .csv reader and I will stick to it for now") then you should refresh your memory by reading documentation csv.reader

Depending on your needs (do you want skip first row or not) you can iterate over reader object directly as (per documentation): "Return a reader object which will iterate over lines in the given csvfile. csvfile can be any object which supports the iterator protocol and returns a string each time its __next__() method is called — file objects and list objects are both suitable."

You can iterate over reader object directly:

>>> x_training = list()
>>> y_training = list()
>>> with open(csv_file, "r") as data:
...     for row in csv.reader(data, delimiter=','):
...         x_training.append(row[:-1])
...         y_training.append(row[-1])
...
I also suggest reading DeaD_EyE post carefully. Without fundamentals it's hard to write code which delivers results you want.
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Reply
#9
I don't think this OP is interested in learning fundamentals or writing any kind of code, but instead getting someone to do their homework for them, which is what you just did. A "regular" poster would not care if the csv module is used or not. {sucker emoji}.
Reply
#10
(Jan-09-2019, 06:26 PM)woooee Wrote: I don't think this OP is interested in learning fundamentals or writing any kind of code, but instead getting someone to do their homework for them, which is what you just did. A "regular" poster would not care if the csv module is used or not. {sucker emoji}.

I am afraid that you are right about current state of OP. But my experience in teaching kids and managing adults is that you should nudge them consistently in right direction. You can't expect them change overnight. If even one from hundred changes attitude and will be truly interested then it's tiny local contribution for better global world Smile
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Reply


Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020