Python Forum
Handle 2 million rows return by oracle database
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Handle 2 million rows return by oracle database
#1
Hi All,

I have one query that needs to be called from python script. That query will return 2 million records.Is there any document in python which can explain best way to handle /process huge data .

Basically I will prepare excel report with those records.
Reply
#2
Is this useful: http://code.activestate.com/recipes/5773...cx_oracle/
Reply
#3
Hi Larz ,
Appreciate your effort in providing documents.

As per this document what I am understanding is I should export my data in csv and then I should read the csv file and put whatever logic I want to put further.
When we read from csv we get data in list format and that will be easy to process

Please correct me if my understanding is incorrect
Reply
#4
I didn't read the entire document, just providing a link.
Reply
#5
No need to write to csv file. Here is Oracle article expalining best practices http://www.oracle.com/technetwork/articl...01587.html
Note the part about cursors being iterator objects.
And useful SO link too
https://stackoverflow.com/questions/594/...result-set
Reply
#6
You probably don't really want that many results.  Normal humans can't consume that much data... so modify the query to return paginated results, and only display 100ish at a time?
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  A million requests per second Larz60+ 1 3,193 Sep-13-2017, 05:57 AM
Last Post: nilamo

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020