Solution for Python CSV>"Array">Simple +-*/

[Bass]

Hi,

Bashing head against wall about this question and *where* to post it.

1. I would really appreciate some straight forward thoughts on this, looking for a solution that works rather than needing the most elegant, modern etc. * The file format is at the botom of the email. *

2. New to Python but competant with systems from '80's 90's etc.

3. I would like to initially read in 4 CSV/Excel files and use simple maths (+-/*) on some of the fields.  This read routine will be replaced by an API in the next month or so (Just waiting on the tech spec of the API.)

4. What is a good starting point for reading in the file and performing a "simple" conversion to an array. It would be great to access the array by refering to array, column and row.

For example Numpy, CSV seem to have some useful functions, but which ones?
The reading is not really a problem, but certain read functions, seem to lend themselves better to creating certain types of arrays, and would like to bear i mind that the read will be replaced by an 'as yet unspecified' API.

Really appreciate any helpful feedback!

Thanks

Bass

* The File Format*

Currently I wish to access 4 excel CSV files.
The files have no headers and there should be commas!
Files 1 & 2 - First two rows are shown below - Total rows = 24
Files 3 & 4 - Exactly the same format (just 1 minute time frame rather than 1 hour), but Total rows  ~ 1440
Column 1 - contains Date DD.MM.YY (space1) HH:MM:SS.sss
Column 2-5 - contains numeric - usually 2-3 N :decimal point: 0-3 N
Column 6 - not important as unlikely to be used.
I have refereed to the fields as columns (from excel) but the intention is to access them as a CSV file. I have added the [col 1..] indicators  and * End of file format example *. (Hope the formatting remains)

[col 1..............................]  [col 2...] [col 3...] [col 4...]  [col 5...] [col 6...]
02.03.2017 00:00:00.000 140.042 140.144 139.96   140.081 6995.11
02.03.2017 00:00:01.000 140.0     140.144 139.962 140.081 6995.11
*End of file format example *

[zivoni]

You probably should look into pandas - its data processing library built on top of numpy. With something like

import pandas as pd
data = pd.read_csv("my.csv", sep=" ")

you can read your csv file into 2-D data structure that can have columns of different types. Dataframes can be accessed by column/"row" name or by indexing with position(s), slicing or boolean vectors. And they provide tons of data manipulation/processing features (grouping, aggregating, rolling windows, applying functions column/row based, regexp etc).

[Bass]

Zivoni,

Thank you for your help.

I starting looking as DF's last night, they certainly seem to fit the task. I just had visions of spending weeks trawling through a never ending list of libraries and functions!

*checks spec, checks function, that works*

Thats decided then -  pandas and data frames it is.

Best wishes,

Bass