Python Forum
python multiprocessing to download sql table
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
python multiprocessing to download sql table
#1
Hi Team,

is it possible to use multiprocessing in below scenario.
downloading sql table into csv files.
to speed up downloading.


Import csv
Quary = "select * from UK_Table"
x = 0

With open("E:\\backup\\output.csv","w","newline="") as outfile
	writer = csv.writer(outfile,quoting = csv.QUOTE_NONNUMERIC)
	writer.writerows(col[0] for col in cursor.description)
	While True:
		rows = cursor.fetchmany(10000)
		if len(rows) ==0:
			print("no records found")
			break
		else:
	          x = x+len(rows)
               print(x)
               for row in rows:
                  writer.writerows(row)


conn.close()
print("success")
print('time taken' . time.time()-initial,"Seconds")



1)  Example of multiprocessing.

import multiprocessing
import os
def square(n):
    print("Worker process id for {0}:{1}".format(n,os.getpid()))
    return(n*n)

if __name__=="__main__":
    #input list
    arr = [1, 2, 3, 4, 5]

    #creating a pool object
    p = multiprocessing.Pool()

    #map list to target function
    result = p.map(square,arr)

    print("Square of each elements:")
    print(result)
Reply


Messages In This Thread
python multiprocessing to download sql table - by mg24 - Oct-31-2022, 03:28 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Is this a multiprocessing bug in Python - or am I doing something wrong? haimat 1 1,238 Oct-18-2023, 06:07 AM
Last Post: absalom1
  python multiprocessing help -- to extract 10 sql table into csv mg24 3 1,437 Nov-20-2022, 11:50 PM
Last Post: mg24
  PyRun_SimpleFile calling multiprocessing Python Class cause endless init loop Xeno 2 1,082 Sep-19-2022, 02:32 AM
Last Post: Xeno
  Python multiprocessing Pool apply async wait for process to complete sunny9495 6 6,569 Apr-02-2022, 06:31 AM
Last Post: sunny9495
  download with internet download manager coral_raha 0 3,005 Jul-18-2021, 03:11 PM
Last Post: coral_raha
  download pubmed PDFs using pubmed2pdf in python Wooki 8 5,599 Oct-19-2020, 03:06 PM
Last Post: jefsummers
  How can I download Python files from GitHub? bitcoin10mil 2 2,871 Aug-26-2020, 09:03 PM
Last Post: Axel_Erfurt
  python multiprocessing import Pool, cpu_count: causes forever loop | help to remove Hassibayub 0 1,894 Jun-18-2020, 05:27 PM
Last Post: Hassibayub
Big Grin python download manager with progressbar (not gui) ghostblade 1 1,946 Apr-23-2020, 11:05 AM
Last Post: snippsat
  Python Download GillietheSquid 2 2,059 Mar-27-2020, 09:15 PM
Last Post: GillietheSquid

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020