Python Forum
python multiprocessing to download sql table
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
python multiprocessing to download sql table
#2
FYI: Possible alternative solution:

You might want to try pandas with sqlalchemy.
I have found this method to be very fast, and multiprocessing may not be necessary

The code is written as an example only, I don't expect that it will run without adding necessary schema, etc.
I have left notes where modifications are required.
The amount of code needed is very small.

It's easiest to create the ORM schema by first dumping your DBMS schema as a guide.
For example, In sqlite, this can be done from command line using sqlite3 as follows:
$ sqlite3
sqlite> .schema AD # your tablename here
with output similar to:
Output:
CREATE TABLE IF NOT EXISTS "AD" ( "RecordType" VARCHAR NOT NULL, "UniqueSystemIdentifier" VARCHAR NOT NULL, "ULSFileNumber" VARCHAR, "EBF_Number" VARCHAR, "ApplicationPurpose" VARCHAR, "ApplicationStatus" VARCHAR, "ApplicationFeeExempt" VARCHAR, "RegulatoryFeeExempt" VARCHAR,
Other DBMS will have similar utility.

Note: Code below will not run without modification
Example code:
from pathlib import Path
import pandas as pd
from sqlalchemy.ext.declarative import declarative_base
# you can add or remove datatypes from import as needed
from sqlalchemy import (Column, String, Integer, Date, create_engine,
    DateTime, ForeignKey)

# replace actual path to database, and DBMS type in engine statement below
engine = create_engine(f"sqlite:///{path_to_database}")
Base = declarative_base()

# You will need to replace schema below eith actual schema for UK_Table
# keep lines 14 and 16 
# I added the following as an example only
class UK_Table(Base):

    __tablename__ = 'UK_Table'

    # --------------------------------------------------------
    # create schema here -- replace example with your schema
    # Following is only an example schema
    # --------------------------------------------------------

    RecordType = Column(String, primary_key=True, nullable=False)
    UniqueSystemIdentifier = Column(String, primary_key=True, nullable=False)
    ULSFileNumber = Column(String, nullable=True)
    EBF_Number = Column(String, index=True, nullable=True)
    ApplicationPurpose = Column(String, index=True, nullable=True)
    ApplicationStatus = Column(String, index=True, nullable=True)
    ApplicationFeeExempt = Column(String, nullable=True)
    RegulatoryFeeExempt = Column(String, nullable=True)


class CsvFromSQL:
    def __init__(self):
        self.csvfile = Path("E:\\backup\\output.csv")
    
    def save_table_to_csv:
        df = read_sql(__tablename__, con=self.UK_Table.engine)
        # Note: see https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html
        # for csv write options.
        df.to_csv(self.cavfile)


def main():
    cfs = CsvFromSQL()
    cfs.save_table_to_csv()


if __name__ == '__main__':
    main()
Reply


Messages In This Thread
RE: python multiprocessing to download sql table - by Larz60+ - Oct-31-2022, 07:48 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Is this a multiprocessing bug in Python - or am I doing something wrong? haimat 1 1,261 Oct-18-2023, 06:07 AM
Last Post: absalom1
  python multiprocessing help -- to extract 10 sql table into csv mg24 3 1,454 Nov-20-2022, 11:50 PM
Last Post: mg24
  PyRun_SimpleFile calling multiprocessing Python Class cause endless init loop Xeno 2 1,092 Sep-19-2022, 02:32 AM
Last Post: Xeno
  Python multiprocessing Pool apply async wait for process to complete sunny9495 6 6,644 Apr-02-2022, 06:31 AM
Last Post: sunny9495
  download with internet download manager coral_raha 0 3,025 Jul-18-2021, 03:11 PM
Last Post: coral_raha
  download pubmed PDFs using pubmed2pdf in python Wooki 8 5,655 Oct-19-2020, 03:06 PM
Last Post: jefsummers
  How can I download Python files from GitHub? bitcoin10mil 2 2,877 Aug-26-2020, 09:03 PM
Last Post: Axel_Erfurt
  python multiprocessing import Pool, cpu_count: causes forever loop | help to remove Hassibayub 0 1,900 Jun-18-2020, 05:27 PM
Last Post: Hassibayub
Big Grin python download manager with progressbar (not gui) ghostblade 1 1,958 Apr-23-2020, 11:05 AM
Last Post: snippsat
  Python Download GillietheSquid 2 2,077 Mar-27-2020, 09:15 PM
Last Post: GillietheSquid

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020