Technical info on reading csv with python

[snippsat]

Do you use Anaconda?
What kind system memory and do use 64-bit Python?

Some geral info about Anaconda.
Keep all updated.

G:\Anaconda3\Scripts
λ conda update conda
λ conda update anaconda
λ conda update python

# Check version 
λ conda --version
conda 4.5.0

# Update pandas
λ conda update pandas

# Check version 
>>> import pandas as pd

>>> pd.__version__
'0.22.0'

# 64bit python
G:\Anaconda3
λ python
Python 3.6.5 |Anaconda custom (64-bit)| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

Now is all updated,this can be important because the have fixed memory issues in newer version.
32bit processes gets 2GB of memory max to play with by default.
Here a test with a large file 2.3GB CSV.

G:\Anaconda3
λ python -m ptpython
>>> import pandas as pd

>>> # Read 2.3GB loan.csv
>>> df = pd.read_csv('loans.csv')
# no error

Work arounds:

# Set memory parameter 
df = pd.read_csv(my_file.csv, sep='|', low_memory=False)

# Read in chunks then concat
tp = pd.read_csv('file_name.csv', header=None, chunksize=1000)
df = pd.concat(tp, ignore_index=True)