Do you use Anaconda?
What kind system memory and do use 64-bit Python?
Some geral info about Anaconda.
Keep all updated.
G:\Anaconda3\Scripts
λ conda update conda
λ conda update anaconda
λ conda update python
# Check version
λ conda --version
conda 4.5.0
# Update pandas
λ conda update pandas
# Check version
>>> import pandas as pd
>>> pd.__version__
'0.22.0'
# 64bit python
G:\Anaconda3
λ python
Python 3.6.5 |Anaconda custom (64-bit)| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>
Now is all updated,this can be important because the have fixed memory issues in newer version.
32bit processes gets 2GB of memory max to play with by default.
Here a test with a large file
2.3GB CSV.
G:\Anaconda3
λ python -m ptpython
>>> import pandas as pd
>>> # Read 2.3GB loan.csv
>>> df = pd.read_csv('loans.csv')
# no error
Work arounds:
# Set memory parameter
df = pd.read_csv(my_file.csv, sep='|', low_memory=False)
# Read in chunks then concat
tp = pd.read_csv('file_name.csv', header=None, chunksize=1000)
df = pd.concat(tp, ignore_index=True)