numpy dtype anomaly

[bluefrog]

I'm attempting to load 2 arrays from 2 columns read from a file . The file is delimited and I'm using numpy's loadtxt() function to load the arrays, like so:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45

#!/usr/bin/python3   import sys import numpy as np import os.path as op from datetime import datetime, date, time from io import StringIO   sample_data = StringIO("AAPL,28-01-2011, ,344.17,344.4,333.53,336.1,21144800\n\ AAPL,31-01-2011, ,335.8,340.04,334.3,339.32,13473000\n\ AAPL,01-02-2011, ,341.3,345.65,340.98,345.03,15236800\n\ AAPL,02-02-2011, ,344.45,345.25,343.55,344.32,9242600\n\ AAPL,03-02-2011, ,343.8,344.24,338.55,343.44,14064100\n\ AAPL,04-02-2011, ,343.61,346.7,343.51,346.5,11494200")   def usage():     print("usage: {} {}".format(op.basename(sys.argv[0], 'filename')))   def get_weekday(date_str):     return datetime.strptime(date_str, "%d-%m-%Y").date().weekday()   def load_arrays(data_file, *col_tuple):     a1 = a2 = None     rec_type = np.dtype([('stock_code', '|S4'), ('cob_date', '|S10'), ('filler', '|S1'),                         ('low_price', 'f4'), ('high_price', 'f4'), ('close_price', 'f4'),                         ('valuation', 'f4'), ('volume', 'uint') ])       try:         a1, a2 = np.loadtxt(data_file, dtype=rec_type, usecols=col_tuple, delimiter=',', unpack=True)         # a1, a2 = np.loadtxt(data_file, usecols=col_tuple, delimiter=',', unpack=True)     except IOError as e:         usage() # failed to open file     except Exception as e: print(e)       return a1, a2   try:     # data_file = sys.argv[0]     data_file = sample_data     c, v = load_arrays(data_file, 5, 6) except IndexError:     usage()   print("Closing price array:\n{}".format(c)) print("\nValuation array:\n{}".format(v))

When I attempt to load the arrays without any data types defined then the load is successfull,
i.e. using

1	a1, a2 = np.loadtxt(data_file, usecols=col_tuple, delimiter=',', unpack=True)

but when I attempt to apply data types, by specifying

1	a1, a2 = np.loadtxt(data_file, dtype=rec_type, usecols=col_tuple, delimiter=',', unpack=True)

I get the following output

1 2 3 4 5 6

list index out of range Closing price array: None   Valuation array: None

Can anybody suggest why the difference or what I am specifying incorrectly as part of the data type specification?

[stullis]

Hmm... If the list index is out of range, that suggests to me that your arrays have a different length than your rec_type. I would guess that rec_type has more fields than the arrays and the interpreter cannot find a corresponding index in your arrays to match up to the rec_type.

[bluefrog]

I've reduced the number of columns to 4, and still the error occurs if usecols is specified. If not, it succeeds in loading each column into an array. I've taken out the missing column, which is column 2 in the previous post.
I've also included a decode('ascii') on the byte string for the date, which is the 2nd column.

So without "usecols", it works fine, as follows:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

import sys import numpy as np import os.path as op from datetime import datetime, date, time from io import StringIO   sample_data = StringIO( "AAPL,28-01-2011,344.17,344.4\n\ AAPL,31-01-2011,335.8,340.04\n\ AAPL,01-02-2011,341.3,345.65\n\ AAPL,02-02-2011,344.45,345.25\n\ AAPL,03-02-2011,343.8,344.24\n\ AAPL,04-02-2011,343.61,346.7")   def usage():     print("usage: {} {}".format(op.basename(sys.argv[0], 'filename')))   def get_weekday(date_str):     return datetime.strptime(date_str.decode('ascii'), "%d-%m-%Y").date().weekday()   def load_arrays(data_file, *col_tuple):     a1 = a2 = a3 = a4 = None     #  rec_type = np.dtype([('stock_code', 'S4'), ('cob_date', 'S10'), ('close_price', 'f4')])     try:         a1, a2, a3, a4 = np.loadtxt(data_file,                                 dtype={'names': ('stock_code','cob_date','high_price','low_price'),                                        'formats': ('S4', 'S10', 'f4', 'f4')},                                 converters={1: get_weekday}, delimiter=',',                                 unpack=True)       except IOError as e:         usage() # failed to open file     except Exception as e: print(e)       return a1, a2, a3, a4   try:     data_file = sample_data     s, d, h, l = load_arrays(data_file) except IndexError:     usage()   print("Stock code array:\n{}".format(s)) print("\nClose of Business date array:\n{}".format(d)) print("\nHigh price array:\n{}".format(h)) print("\nLow price array:\n{}".format(l))

But when I want to only load say column 0 and 2, then I get

1	list index out of range

code appears as:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

def load_arrays(data_file, *col_tuple):     a1 = a2 = None       try:         a1, a2 = np.loadtxt(data_file,                                 dtype={'names': ('stock_code','cob_date','high_price','low_price'),                                        'formats': ('S4', 'S10', 'f4', 'f4')},                                 converters={1: get_weekday}, delimiter=',',                                 usecols=(0,2), unpack=True)       except IOError as e:         usage() # failed to open file     except Exception as e: print(e)       return a1, a2   try:     data_file = sample_data     s,  h= load_arrays(data_file) except IndexError:     usage()

The number of columns matches what is specified in dtype.

[stullis]

According to the documentation, the usecols parameter does this:

usecols : int or sequence, optional

Which columns to read, with 0 being the first. For example, usecols = (1,4,5) will extract the 2nd, 5th and 6th columns. The default, None, results in all columns being read.

So, you're telling it to use column 0 and column 2 only but your dtype has four columns listed. Have you tried usecols with the same number of columns as the dtype?

[bluefrog]

thanks. it now works.
so basically one has to specify your record column names and data types each time you extract an arbitrary set of column number(s).
mmmm, bit clunky.