Python Forum
Fastest Method for Querying SQL Server with Python Pandas
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Fastest Method for Querying SQL Server with Python Pandas
#1
I am querying a sql table with about 3.5 MM rows and 400 columns - I am estimating it is less than 5 GB in size.

If I use pd.read_sql to select * from this table, it takes many, many hours to execute. (Over 10 hours sometimes....)

I would like to read in the entire table and not cut it into pieces....I know folks will suggest otherwise, I get that. But I am still looking for a better solution that can handle reading in the entire table. We are not talking about terabytes here...

Also, even if I use chunks, the sum of the time the individual chunks take to process is similar amount of time to execute the entire table query.

Is there any alternative approach that would work faster?

Thanks,

Brian
Reply


Messages In This Thread
Fastest Method for Querying SQL Server with Python Pandas - by BuJayBelvin - Jul-30-2020, 04:02 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Fastest way tkinter Quatrixouuu 2 549 Feb-19-2024, 07:20 AM
Last Post: Danishhafeez
  What is the fastest way to get all the frames from a video file? glorsh66 3 1,390 May-26-2023, 04:41 AM
Last Post: Gribouillis
  Method works as expected on host machine but not on server gradlon93 4 1,236 Jan-05-2023, 10:41 AM
Last Post: DeaD_EyE
  Basic SQL query using Py: Inserting or querying sqlite3 database not returning data marlonbown 3 1,629 Nov-08-2022, 07:16 PM
Last Post: marlonbown
  Fastest Way of Writing/Reading Data JamesA 1 2,317 Jul-27-2021, 03:52 PM
Last Post: Larz60+
  How to take the tar backup files form remote server to local server sivareddy 0 2,033 Jul-14-2021, 01:32 PM
Last Post: sivareddy
  Recommendations: Querying CSV / Database?? ironfelix717 2 1,758 Apr-01-2020, 12:13 PM
Last Post: ndc85430
  Passing parameters to SQL Server using pandas Ilangos 8 16,853 Jun-03-2019, 11:25 AM
Last Post: Ilangos
  Fastest dict/map method when 'key' is already a hash? tasket 6 4,188 Apr-20-2019, 06:40 PM
Last Post: tasket
  fastest way to record values between quotes paul18fr 5 3,481 Apr-15-2019, 01:51 PM
Last Post: snippsat

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020