Python Forum
Load specific set of Rows CSV
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Load specific set of Rows CSV
#1
Hello,

I have a CSV file that looks something like this:

strings
strings
data (float)
data (float)
data (float)
strings
strings
strings
data (float)
data (float)
...
etc

The file contains many rows and two columns and I would like to load e.g. all the columns in row 1 to 5 in a numpy array. Additionally, I would like to store e.g. row 6-144 in another array (all those rows are data) etc. Is there a way to do this? I tried things like np.loadtxt, csv.reder and pandas.read_csv but I must be missing something. I would prefer to do it without using a loop due to the time consumption when the files get very large.

If someone use MATLAB and know the python way to do this, here's a hint:
MATRIX = dlmread('FILE.csv',',',[5 0 144 1]);
This stores row 5 to 144 and colum 0 to 1 if a matlab code.

Thanks in advance!
Reply
#2
Sounds like an assignment.
Give it your best shot and we will be glad to help when you get stuck.
Reply
#3
Hello,

It is not an assignment. I'm doing my second years in the applied mechanics masters and we've been using MATLAB the first year. All the companies/industries in our city are using python so I'm learning python during the summers and that's why I try to convert my own MATLAB scripts to python to get a skeleton with hints which I can use later on in life. I'm getting a lot of data files for which I can't control the exact output when doing the simulations which are a tricky part.

I managed to get it to work by doing this:

wholelist = list(csv.reader(open('FILE.csv')))
set1 = np.array(wholelist[5:144],dtype=float)

But that is in two lines and I cant control it as much as I would like. Would be great if someone has another way of doing it.

Thanks!
Reply
#4
you can read the data line by line. It's very difficult to attempt without a data file,

but something like this (Totally untested as I have no data file):
import csv

with open('File.csv') as fp:
    reader = csv.reader(fp)
    fields = None
    fields = [field for field in reader]
    # process fields here
    next(reader)
Reply
#5
it's late in the evening for Larz and maybe he is tired so he suggest something that will raise exception

import csv
 
with open('File.csv') as fp:
    reader = csv.reader(fp)
    fields = None # no need of this
    fields = [field for field in reader] # you will read the whole file here
    # process fields here
    next(reader) # this will rise an exception because reader is exhausted - you are at the end of the file
better
import csv
 
with open('File.csv') as fp:
    reader = csv.reader(fp)
    for line in reader: # iterate over lines in the file
        # process line here
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply
#6
Buran is correct, it was about 4:00 A.M. and I haden't slept yet.
With his solution, line will be a list of a single csv row.
Reply
#7
It seems to do the job well, thanks!
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Write specific rows from pandas dataframe to csv file pradeepkumarbe 3 5,427 Oct-18-2018, 09:33 PM
Last Post: volcano63
  Removing rows at random based on the value of a specific column Mr_Keystrokes 4 5,528 Aug-24-2018, 11:15 AM
Last Post: Mr_Keystrokes
  How to filter specific rows from large data file Ariane 7 8,141 Jun-29-2018, 02:43 PM
Last Post: gontajones

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020