+- Python Forum (https://python-forum.io)
+-- Forum: General (https://python-forum.io/forum-1.html)
+--- Forum: News and Discussions (https://python-forum.io/forum-31.html)
+--- Thread: Handle 2 million rows return by oracle database (/thread-5407.html)
Handle 2 million rows return by oracle database - purnima1 - Oct-02-2017
Hi All,
I have one query that needs to be called from python script. That query will return 2 million records.Is there any document in python which can explain best way to handle /process huge data .
Basically I will prepare excel report with those records.
RE: Handle 2 million rows return by oracle database - Larz60+ - Oct-02-2017
Is this useful: http://code.activestate.com/recipes/577304-export-oracle-database-to-csv-using-cx_oracle/
RE: Handle 2 million rows return by oracle database - purnima1 - Oct-02-2017
Hi Larz ,
Appreciate your effort in providing documents.
As per this document what I am understanding is I should export my data in csv and then I should read the csv file and put whatever logic I want to put further.
When we read from csv we get data in list format and that will be easy to process
Please correct me if my understanding is incorrect
RE: Handle 2 million rows return by oracle database - Larz60+ - Oct-02-2017
I didn't read the entire document, just providing a link.
RE: Handle 2 million rows return by oracle database - buran - Oct-02-2017
No need to write to csv file. Here is Oracle article expalining best practices http://www.oracle.com/technetwork/articles/dsl/prez-python-queries-101587.html
Note the part about cursors being iterator objects.
And useful SO link too
https://stackoverflow.com/questions/594/cx-oracle-how-do-i-iterate-over-a-result-set
RE: Handle 2 million rows return by oracle database - nilamo - Oct-04-2017
You probably don't really want that many results. Normal humans can't consume that much data... so modify the query to return paginated results, and only display 100ish at a time?