Python Forum
Thread Rating:
  • 1 Vote(s) - 4 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Memory Error
#1
I manage a dataset of AMEX, NASDAQ and AMEX stocks over last 40 years. I encountered a memory error shown as follows:
File "C:\Users\lokac\Anaconda3\lib\site-packages\pandas\core\algorithms.py", line 1379, in take_nd
out = np.empty(out_shape, dtype=dtype)

MemoryError

Many thanks for helping.
Reply
#2
Post the full error traceback message in error tags, and if possible also relevant piece of code in Python code tags. You can find help here.
Reply
#3
If I reduce the sample length to 10 years, the python code works well without any memory error.
Here is the tracking record:
Error:
Traceback (most recent call last): File "<ipython-input-1-4d852114e435>", line 1, in <module> runfile('C:/Users/lokac/Desktop/Data/Momentum wrds.py', wdir='C:/Users/lokac/Desktop/Data') File "C:\Users\lokac\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 705, in runfile execfile(filename, namespace) File "C:\Users\lokac\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile exec(compile(f.read(), filename, 'exec'), namespace) File "C:/Users/lokac/Desktop/Data/Momentum wrds.py", line 89, in <module> port = pd.merge(_tmp_ret, umd, on=['permno'], how='inner') File "C:\Users\lokac\Anaconda3\lib\site-packages\pandas\core\reshape\merge.py", line 58, in merge return op.get_result() File "C:\Users\lokac\Anaconda3\lib\site-packages\pandas\core\reshape\merge.py", line 596, in get_result concat_axis=0, copy=self.copy) File "C:\Users\lokac\Anaconda3\lib\site-packages\pandas\core\internals.py", line 5203, in concatenate_block_managers concatenate_join_units(join_units, concat_axis, copy=copy), File "C:\Users\lokac\Anaconda3\lib\site-packages\pandas\core\internals.py", line 5332, in concatenate_join_units for ju in join_units] File "C:\Users\lokac\Anaconda3\lib\site-packages\pandas\core\internals.py", line 5332, in <listcomp> for ju in join_units] File "C:\Users\lokac\Anaconda3\lib\site-packages\pandas\core\internals.py", line 5632, in get_reindexed_values fill_value=fill_value) File "C:\Users\lokac\Anaconda3\lib\site-packages\pandas\core\algorithms.py", line 1379, in take_nd out = np.empty(out_shape, dtype=dtype) MemoryError
Reply
#4
You are obviously running out of memory.

pandas is a memory hog - see this article. Quoting the author

Quote:my rule of thumb for pandas is that you should have 5 to 10 times as much RAM as the size of your dataset

You probably should find a way to split your data into chunks and process it in smaller portions - or increase the amount of available RAM
Test everything in a Python shell (iPython, Azure Notebook, etc.)
  • Someone gave you an advice you liked? Test it - maybe the advice was actually bad.
  • Someone gave you an advice you think is bad? Test it before arguing - maybe it was good.
  • You posted a claim that something you did not test works? Be prepared to eat your hat.
Reply
#5
Iterating is what you need. You chunk the data into parts, that it fits into your memory. Then you processing them, saving the results and proceed with the next chunk.
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply
#6
It's better to use a generator. It's super memory efficient.
"As they say in Mexico 'dosvidaniya'. That makes two vidaniyas."
https://freedns.afraid.org
Reply
#7
http://www.crucial.com/
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Memory error while recursively adding np.arrays Afterdarkreader 0 3,989 Dec-22-2017, 04:02 PM
Last Post: Afterdarkreader
  Memory error in python 2.7 Afterdarkreader 4 6,445 Dec-20-2017, 02:26 AM
Last Post: Afterdarkreader
  Memory Error rajeev1729 1 27,418 Sep-26-2017, 09:50 AM
Last Post: metulburr

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020