Python Forum
Technical info on reading csv with python
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Technical info on reading csv with python
#1
Hi.

I have tried to read 500 MB csv file with pandas in python, but it gives MemoryError. So, could you share your experience on what kind of technial parameters should have the computer in order to read such files.

thanks in advance
Reply
#2
Do you use Anaconda?
What kind system memory and do use 64-bit Python?

Some geral info about Anaconda.
Keep all updated.
G:\Anaconda3\Scripts
λ conda update conda
λ conda update anaconda
λ conda update python

# Check version 
λ conda --version
conda 4.5.0

# Update pandas
λ conda update pandas

# Check version 
>>> import pandas as pd

>>> pd.__version__
'0.22.0'

# 64bit python
G:\Anaconda3
λ python
Python 3.6.5 |Anaconda custom (64-bit)| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>
Now is all updated,this can be important because the have fixed memory issues in newer version.
32bit processes gets 2GB of memory max to play with by default.
Here a test with a large file 2.3GB CSV.
G:\Anaconda3
λ python -m ptpython
>>> import pandas as pd

>>> # Read 2.3GB loan.csv
>>> df = pd.read_csv('loans.csv')
# no error
Work arounds:
# Set memory parameter 
df = pd.read_csv(my_file.csv, sep='|', low_memory=False)
# Read in chunks then concat
tp = pd.read_csv('file_name.csv', header=None, chunksize=1000)
df = pd.concat(tp, ignore_index=True)
Reply
#3
Here is the version info.
I use sublime.

3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:04:45) [MSC v.1900 32 bit (Intel)]

but my PC is 64 bit.
Reply
#4
(Apr-12-2018, 05:57 AM)garikhgh0 Wrote: I use sublime.

3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:04:45) [MSC v.1900 32 bit (Intel)]

but my PC is 64 bit.
Okay your python is 32-bit so limited to max 2GB,OS is 64-bit but it's limited to what Python use.
You can look solution with Work around that i posted.

I also have Python 32-bit as my main Python(OS is 64-bit),
but i also have Anaconda where Python is 64-bit where in read file as posted over.
No problem to have both on OS,here my tutorial about Anaconda

Not cmd but cmder.
C:\Users\Tom
λ cd\

# My main Python that is set in Windows environment variables Path(work from anywhere/cmd/cmder)
C:\
λ python
Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:04:45) [MSC v.1900 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()

G:\
λ cd Anaconda3

# Now i use Anaconda Python 64-bit
G:\Anaconda3
λ python
Python 3.6.5 |Anaconda custom (64-bit)| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Help for technical choice jikail 3 2,197 Jan-10-2020, 12:13 PM
Last Post: jefsummers

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020