Load specific set of Rows CSV - Printable Version

Load specific set of Rows CSV - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Data Science (https://python-forum.io/forum-44.html)
+--- Thread: Load specific set of Rows CSV (/thread-11266.html)

Load specific set of Rows CSV - Marcuslang - Jun-30-2018

Hello,

I have a CSV file that looks something like this:

strings
strings
data (float)
data (float)
data (float)
strings
strings
strings
data (float)
data (float)
...
etc

The file contains many rows and two columns and I would like to load e.g. all the columns in row 1 to 5 in a numpy array. Additionally, I would like to store e.g. row 6-144 in another array (all those rows are data) etc. Is there a way to do this? I tried things like np.loadtxt, csv.reder and pandas.read_csv but I must be missing something. I would prefer to do it without using a loop due to the time consumption when the files get very large.

If someone use MATLAB and know the python way to do this, here's a hint:
MATRIX = dlmread('FILE.csv',',',[5 0 144 1]);
This stores row 5 to 144 and colum 0 to 1 if a matlab code.

Thanks in advance!

RE: Load specific set of Rows CSV - Larz60+ - Jun-30-2018

Sounds like an assignment.
Give it your best shot and we will be glad to help when you get stuck.

RE: Load specific set of Rows CSV - Marcuslang - Jul-01-2018

Hello,

It is not an assignment. I'm doing my second years in the applied mechanics masters and we've been using MATLAB the first year. All the companies/industries in our city are using python so I'm learning python during the summers and that's why I try to convert my own MATLAB scripts to python to get a skeleton with hints which I can use later on in life. I'm getting a lot of data files for which I can't control the exact output when doing the simulations which are a tricky part.

I managed to get it to work by doing this:

wholelist = list(csv.reader(open('FILE.csv')))
set1 = np.array(wholelist[5:144],dtype=float)

But that is in two lines and I cant control it as much as I would like. Would be great if someone has another way of doing it.

Thanks!

RE: Load specific set of Rows CSV - Larz60+ - Jul-01-2018

you can read the data line by line. It's very difficult to attempt without a data file,

but something like this (Totally untested as I have no data file):

import csv

with open('File.csv') as fp:
    reader = csv.reader(fp)
    fields = None
    fields = [field for field in reader]
    # process fields here
    next(reader)

RE: Load specific set of Rows CSV - buran - Jul-01-2018

it's late in the evening for Larz and maybe he is tired so he suggest something that will raise exception

import csv
 
with open('File.csv') as fp:
    reader = csv.reader(fp)
    fields = None # no need of this
    fields = [field for field in reader] # you will read the whole file here
    # process fields here
    next(reader) # this will rise an exception because reader is exhausted - you are at the end of the file

better

import csv
 
with open('File.csv') as fp:
    reader = csv.reader(fp)
    for line in reader: # iterate over lines in the file
        # process line here

RE: Load specific set of Rows CSV - Larz60+ - Jul-01-2018

Buran is correct, it was about 4:00 A.M. and I haden't slept yet.
With his solution, line will be a list of a single csv row.

RE: Load specific set of Rows CSV - Marcuslang - Jul-01-2018

It seems to do the job well, thanks!