Python Forum
python multiprocessing to download sql table
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
python multiprocessing to download sql table
#5
Mulit-processing doesn't speed things up unless they are processor bound (doing lots of calculatons). Querying a database is not a processor bound application. You also need to come up with a way to make queries so each process can retrieve some well defined part of the database, and have a way to stitch the results together. Asking for every row in the one table in your database is not going to work.

That said, you should try it to see what really happens.
Reply


Messages In This Thread
RE: python multiprocessing to download sql table - by deanhystad - Oct-31-2022, 03:23 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Is this a multiprocessing bug in Python - or am I doing something wrong? haimat 1 1,293 Oct-18-2023, 06:07 AM
Last Post: absalom1
  python multiprocessing help -- to extract 10 sql table into csv mg24 3 1,487 Nov-20-2022, 11:50 PM
Last Post: mg24
  PyRun_SimpleFile calling multiprocessing Python Class cause endless init loop Xeno 2 1,115 Sep-19-2022, 02:32 AM
Last Post: Xeno
  Python multiprocessing Pool apply async wait for process to complete sunny9495 6 6,737 Apr-02-2022, 06:31 AM
Last Post: sunny9495
  download with internet download manager coral_raha 0 3,057 Jul-18-2021, 03:11 PM
Last Post: coral_raha
  download pubmed PDFs using pubmed2pdf in python Wooki 8 5,705 Oct-19-2020, 03:06 PM
Last Post: jefsummers
  How can I download Python files from GitHub? bitcoin10mil 2 2,896 Aug-26-2020, 09:03 PM
Last Post: Axel_Erfurt
  python multiprocessing import Pool, cpu_count: causes forever loop | help to remove Hassibayub 0 1,914 Jun-18-2020, 05:27 PM
Last Post: Hassibayub
Big Grin python download manager with progressbar (not gui) ghostblade 1 1,980 Apr-23-2020, 11:05 AM
Last Post: snippsat
  Python Download GillietheSquid 2 2,097 Mar-27-2020, 09:15 PM
Last Post: GillietheSquid

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020