how to process 12 million record

Thread Rating:

0 Vote(s) - 0 Average
1
2
3
4
5

Thread Modes

how to process 12 million record

Calab
Not Blown Up Yet

Posts: 66

Threads: 28

Joined: Feb 2023

Reputation: 0

Jul-31-2024, 06:27 PM

(Jul-30-2024, 09:56 PM)pymach Wrote: I need to read 12 millions records from a SQL table to store in another database after pivoting it. It works, but its taking lot time, any idea to improve the performance would be a great help.

Looking forward to get some help to improve to performance.

If your data matches the table you are trying to update, look into the executemany method: ExecuteMany

Find

Messages In This Thread

how to process 12 million record - by pymach - Jul-30-2024, 09:56 PM

RE: how to process 12 million record - by Calab - Jul-31-2024, 06:27 PM

RE: how to process 12 million record - by Larz60+ - Aug-01-2024, 07:15 AM

RE: how to process 12 million record - by DeaD_EyE - Aug-01-2024, 07:28 AM

RE: how to process 12 million record - by Pedroski55 - Aug-01-2024, 07:58 AM

Possibly Related Threads…
Thread		Author	Replies	Views	Last Post
	Last record in file doesn't write to newline	gonksoup	3	1,788	Jan-22-2024, 12:56 PM Last Post: deanhystad
	How do I stream and record at the same time with arducam?	traderjoe	0	1,166	Oct-23-2023, 12:01 AM Last Post: traderjoe
	Only getting last record saved...Why	Milfredo	10	7,223	Sep-10-2020, 03:00 AM Last Post: Milfredo
	how can we record a video file from our program output (moving object)	Zhaleh	0	2,423	Aug-03-2020, 02:47 PM Last Post: Zhaleh
	Get the record number of a dBASE dbf file	DarkCoder2020	0	2,453	Jun-16-2020, 05:11 PM Last Post: DarkCoder2020
	Compare 5 variables in a record with an excel sheet!!	SEED	1	2,457	Apr-20-2020, 11:10 PM Last Post: michael1789
	How to insert record into MySQL using Phython	abhay_kala	1	2,901	Dec-06-2019, 04:34 PM Last Post: abhay_kala
	How to sharing object between multiple process from main process using Pipe	Subrata	1	4,562	Sep-03-2019, 09:49 PM Last Post: woooee
	fastest way to record values between quotes	paul18fr	5	4,435	Apr-15-2019, 01:51 PM Last Post: snippsat
	I'm dividing large numbers of about 25 million digits I need to retain decimal format	Pleiades	2	4,178	Apr-26-2018, 07:50 AM Last Post: Pleiades

Users browsing this thread: 1 Guest(s)

View a Printable Version

how to process 12 million record

User Panel Messages

Announcements