Bottom Page

Thread Rating:
  • 2 Vote(s) - 1.5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 Solution for Python CSV>"Array">Simple +-*/
#1
Hi,

Bashing head against wall about this question and *where* to post it.

1. I would really appreciate some straight forward thoughts on this, looking for a solution that works rather than needing the most elegant, modern etc. * The file format is at the botom of the email. *

2. New to Python but competant with systems from '80's 90's etc.

3. I would like to initially read in 4 CSV/Excel files and use simple maths (+-/*) on some of the fields.  This read routine will be replaced by an API in the next month or so (Just waiting on the tech spec of the API.)

4. What is a good starting point for reading in the file and performing a "simple" conversion to an array. It would be great to access the array by refering to array, column and row.

For example Numpy, CSV seem to have some useful functions, but which ones?
The reading is not really a problem, but certain read functions, seem to lend themselves better to creating certain types of arrays, and would like to bear i mind that the read will be replaced by an 'as yet unspecified' API.

Really appreciate any helpful feedback!

Thanks

Bass

* The File Format*

Currently I wish to access 4 excel CSV files.
The files have no headers and there should be commas!
Files 1 & 2 - First two rows are shown below - Total rows = 24
Files 3 & 4 - Exactly the same format (just 1 minute time frame rather than 1 hour), but Total rows  ~ 1440
Column 1 - contains Date DD.MM.YY (space1) HH:MM:SS.sss
Column 2-5 - contains numeric - usually 2-3 N :decimal point: 0-3 N
Column 6 - not important as unlikely to be used.
I have refereed to the fields as columns (from excel) but the intention is to access them as a CSV file. I have added the [col 1..] indicators  and * End of file format example *. (Hope the formatting remains)

[col 1..............................]  [col 2...] [col 3...] [col 4...]  [col 5...] [col 6...]
02.03.2017 00:00:00.000 140.042 140.144 139.96   140.081 6995.11
02.03.2017 00:00:01.000 140.0     140.144 139.962 140.081 6995.11
*End of file format example *
Quote
#2
You probably should look into pandas - its data processing library built on top of numpy. With something like
import pandas as pd
data = pd.read_csv("my.csv", sep=" ")
you can read your csv file into 2-D data structure that can have columns of different types. Dataframes can be accessed by column/"row" name or by indexing with position(s), slicing or boolean vectors. And they provide tons of data manipulation/processing features (grouping, aggregating, rolling windows, applying functions column/row based, regexp etc).
snippsat likes this post
Quote
#3
Zivoni,

Thank you for your help.

I starting looking as DF's last night, they certainly seem to fit the task. I just had visions of spending weeks trawling through a never ending list of libraries and functions!

*checks spec, checks function, that works*

Thats decided then -  pandas and data frames it is.

Best wishes,

Bass
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  How convert multidimensional array to two dimensional array tkkhan44 1 337 Feb-20-2019, 05:00 AM
Last Post: scidam
  Converting a VideoFrame to a byte array using python synthex 0 266 Feb-06-2019, 04:04 PM
Last Post: synthex
  creating an empty NumPy array in Python sandeepatel 2 615 Jan-16-2019, 09:24 AM
Last Post: rinu
  To draw a polynomial solution pianistseb 2 427 Nov-13-2018, 01:45 PM
Last Post: pianistseb
  To put a solution of an equation into a function pianistseb 6 527 Nov-11-2018, 08:22 PM
Last Post: pianistseb
  python odeint keeps giving me size of array error kiyoshi7 1 604 Nov-01-2018, 02:03 AM
Last Post: j.crater
  Newbie at using python and tensorflow getting error when running simple code FeatherineAu 0 873 Sep-28-2018, 02:09 PM
Last Post: FeatherineAu
  Array input with time module (Python 3) alrm31015 4 889 Jul-26-2018, 08:52 PM
Last Post: alrm31015
  Simple Python function help needed Imran23 2 648 Jun-07-2018, 12:20 AM
Last Post: Imran23
  python array/cell/indexing python 1 768 Feb-08-2018, 11:18 AM
Last Post: DeaD_EyE

Forum Jump:


Users browsing this thread: 1 Guest(s)