+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Data Science (https://python-forum.io/forum-44.html)
+--- Thread: Is there a way to save a CSV file as a python object (/thread-19827.html)
Is there a way to save a CSV file as a python object - amjass12 - Jul-16-2019
Hi all,
I was wondering if there is a way to save a CSV file as a python object in order to be read in to python quicker. The reason I am cocnerned about this is is that some of the CSV files i hold contain many thousands (10's to 100's of columns) plus many rows of features, ~30,000.
Now having the CSV file itself is not an issue.. but its bulky and takes some time to read in to python.
In r, these can be saved as an rds or r object.. which although they hold the same CSV file, the load time in to R is almost instant. Reading in the CSV file itself in to R takes a good 20-30 seconds.
Is there an equivalent for this in python as I will be reading in a lot of CSV files in to python, so having an object with CSV data would be useful if it means reading in is much faster.
Any advice would be appreciated. Thank you!
Amir
RE: Is there a way to save a CSV file as a python object - perfringo - Jul-16-2019
The question is too ambiguous to answer: "its bulky and takes some time to read in to python".
What specifically is bulky and what time performance is considered satisfactory? And the main question: into what datastructure you read the file and what would you do with aquired data.
RE: Is there a way to save a CSV file as a python object - amjass12 - Jul-16-2019
(Jul-16-2019, 11:12 AM)perfringo Wrote: The question is too ambiguous to answer: "its bulky and takes some time to read in to python". What specifically is bulky and what time performance is considered satisfactory? And the main question: into what datastructure you read the file and what would you do with aquired data.
the file size increases into a file that takes time to read in to python. Based on experience in R, reading an R object in to R is near instant and that is what i would consider satisfactory.
The object would be able to hold the table and the downstream processes would be manipulation of the data for later analysis of the dataframe. a pandas dataframe is the ultimate goal. which again, is absolutely fine reading in the CSV file itself. I was jsut wondering if there was a known python style object that holds the csv file...
RE: Is there a way to save a CSV file as a python object - perfringo - Jul-16-2019
(Jul-16-2019, 11:23 AM)amjass12 Wrote: Based on experience in R, reading an R object in to R is near instant and that is what i would consider satisfactory.
I read csv files with 200K rows into dictionary and it's 'near instant'. Access to dictionary keys is instant (O(1)).
RE: Is there a way to save a CSV file as a python object - amjass12 - Jul-16-2019
(Jul-16-2019, 11:48 AM)perfringo Wrote:(Jul-16-2019, 11:23 AM)amjass12 Wrote: Based on experience in R, reading an R object in to R is near instant and that is what i would consider satisfactory.I read csv files with 200K rows into dictionary and it's 'near instant'. Access to dictionary keys is instant (O(1)).
ok perfect! this was the kind of information i was looking for! if this is the quickest way to do it in python then great. indeed it is significantly faster in python than in R but i was thinking about the long term where i will be getting data sets of a similar size ~200k. they are essentially counts tables... in R these take a long time to read in!
thanks for the info! i shall continue reading in CSV files..