Python Forum
pandas.read_sas with chunksize: IndexError list index out of range
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
pandas.read_sas with chunksize: IndexError list index out of range
#1
Good morning,

I have read the rules about posting but i cannot attach a sample of my data or reproduce the entire error message as the data i am working on is located on a server without access to internet. I apologise for this inconvenience. I'll try to reproduce most of what is requested however below.

I am working with very big sas files (data on each job, hence millions of lines) and got memory error when i was trying to simple read them (they open fine in R or stata strangely). Therefore i searched and find the pandas.read_sas option to work with chunks of the data. My code is now the following:

df_chunk = pd.read_sas(r'file.sas7bdat', chunksize=500)

for chunk in df_chunk:  
    chunk_list.append(chunk)


At this point i get the following error (I am reproducing it here manually as i cannot copy paste):

Error:
line 660, in _chunk_to_dataframe if self.column_formats[j] in const.sas_date_formats: IndexError: list index out of range
Looking deeper in the error message, the issue seems to be in the underlying function "
_chunk_to_dataframe(self)
" in the following line :
if self.column_formats[j] in const.sas_date_formats 
Many thanks for your help,
Axelle
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Cycle through Numpy range within another range(?) Zero01 0 393 Jul-31-2020, 02:37 PM
Last Post: Zero01
  [pandas] How to reshape the list Mekala 6 552 Jul-26-2020, 12:49 AM
Last Post: Mekala
  IndexError: index 0 is out of bounds for axis 0 with size 0 tmhsa 0 835 Apr-24-2020, 10:00 AM
Last Post: tmhsa
  Getting Index Error - list index out of range RahulSingh 2 1,383 Feb-03-2020, 07:17 AM
Last Post: RahulSingh
  itertuples, datetime, pandas, groupby, in range karlito 0 861 Nov-29-2019, 11:35 AM
Last Post: karlito
  iterate over index and define each range as a day karlito 7 840 Nov-19-2019, 06:37 AM
Last Post: karlito
  How to speed up work with pandas index? AlekseyPython 1 517 Oct-16-2019, 02:06 PM
Last Post: AlekseyPython
  IndexError: index out of bounds LeoGER 3 4,146 Sep-05-2019, 02:05 PM
Last Post: LeoGER
  Applying operation to a pandas multi index dataframe subgroup Nuovoq 1 903 Sep-04-2019, 10:04 PM
Last Post: Nuovoq
  pandas: can we look for the index of a string paul18fr 2 780 Jul-31-2019, 08:25 AM
Last Post: paul18fr

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020