FYI: Possible alternative solution:
You might want to try pandas with sqlalchemy.
I have found this method to be very fast, and multiprocessing may not be necessary
The code is written as an example only, I don't expect that it will run without adding necessary schema, etc.
I have left notes where modifications are required.
The amount of code needed is very small.
It's easiest to create the ORM schema by first dumping your DBMS schema as a guide.
For example, In sqlite, this can be done from command line using sqlite3 as follows:
Note: Code below will not run without modification
Example code:
You might want to try pandas with sqlalchemy.
I have found this method to be very fast, and multiprocessing may not be necessary
The code is written as an example only, I don't expect that it will run without adding necessary schema, etc.
I have left notes where modifications are required.
The amount of code needed is very small.
It's easiest to create the ORM schema by first dumping your DBMS schema as a guide.
For example, In sqlite, this can be done from command line using sqlite3 as follows:
$ sqlite3 sqlite> .schema AD # your tablename herewith output similar to:
Output:CREATE TABLE IF NOT EXISTS "AD" (
"RecordType" VARCHAR NOT NULL,
"UniqueSystemIdentifier" VARCHAR NOT NULL,
"ULSFileNumber" VARCHAR,
"EBF_Number" VARCHAR,
"ApplicationPurpose" VARCHAR,
"ApplicationStatus" VARCHAR,
"ApplicationFeeExempt" VARCHAR,
"RegulatoryFeeExempt" VARCHAR,
Other DBMS will have similar utility.Note: Code below will not run without modification
Example code:
from pathlib import Path import pandas as pd from sqlalchemy.ext.declarative import declarative_base # you can add or remove datatypes from import as needed from sqlalchemy import (Column, String, Integer, Date, create_engine, DateTime, ForeignKey) # replace actual path to database, and DBMS type in engine statement below engine = create_engine(f"sqlite:///{path_to_database}") Base = declarative_base() # You will need to replace schema below eith actual schema for UK_Table # keep lines 14 and 16 # I added the following as an example only class UK_Table(Base): __tablename__ = 'UK_Table' # -------------------------------------------------------- # create schema here -- replace example with your schema # Following is only an example schema # -------------------------------------------------------- RecordType = Column(String, primary_key=True, nullable=False) UniqueSystemIdentifier = Column(String, primary_key=True, nullable=False) ULSFileNumber = Column(String, nullable=True) EBF_Number = Column(String, index=True, nullable=True) ApplicationPurpose = Column(String, index=True, nullable=True) ApplicationStatus = Column(String, index=True, nullable=True) ApplicationFeeExempt = Column(String, nullable=True) RegulatoryFeeExempt = Column(String, nullable=True) class CsvFromSQL: def __init__(self): self.csvfile = Path("E:\\backup\\output.csv") def save_table_to_csv: df = read_sql(__tablename__, con=self.UK_Table.engine) # Note: see https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html # for csv write options. df.to_csv(self.cavfile) def main(): cfs = CsvFromSQL() cfs.save_table_to_csv() if __name__ == '__main__': main()