Mult-threading and locking file

[Larz60+]

I wrote a tutorial on automatically creating a database from csv files.
Note: Project uses sqlite database. Other DBMS can be used with a simple change to the model.

The project demonstrates dynamic model creation, and subsequent database load, achieving the following:

1. Given a path to a directory that contains all of the CSV files (one for each table) do:
   
   • Determine columns and sizes required for each.
     
   • Auto-create an SQLalchemy model class which represents the table that will be associated with the data from each CSV file.
     
   • Using pandas, populate the database.
     

to show the simplicity of the code, here's the database load module:

from FlightPaths import FlightPaths
import pandas as pd
import DbModel
import csv


class LoadDatabase:
    def __init__(self):
        self.fpath = FlightPaths()

        self.Model = DbModel

    def dispatch(self):
        self.load_all()

    def load_all(self):
        datafiles = self.fpath.ourairport_datafiles

        filelist = [fn for fn in datafiles.iterdir() if fn.is_file() and fn.suffix == '.csv']
        for fn in filelist:
            tablename = fn.stem.replace('-', '_')
            df = pd.read_csv(fn)
            print(f"engine: {self.Model.engine}")
            df.to_sql(tablename, con=self.Model.engine, if_exists='replace')


def main():
    lDb = LoadDatabase()
    lDb.load_all()


if __name__ == '__main__':
    main()

If you are interested, you will find the tutorial here: https://python-forum.io/thread-33843.html