Python modules for accessing the configuration of relevant paths

[Imago]

Below I have the files **main.py**, **train.py** and **dataLoader.py**. The purpose of path.py is to contain relevant paths inside my project. These paths however should not be hard-coded as multiple user on various machines may use it. The relative structure of the project stays the same on each system up to the base paths of the project. For example, the root folder for data, for evaluation, for the python etc. may differ though.

I want to use the following idea: The main gets an argument of its current user. This user is used too read in a config file setting the respective root paths of the project. Afterwards the interesting/relevant code of the project is executed, e.g. the dataloader.py. This code then uses paths.py and knows where to look up certain directories etc.

My code is working, but has certain caveats and in fact has to be adapted. I dislike my solution, because paths.py and interesting/relevant modules are imported inside the main function of main.py. I also feel uncomfortable that I have to set a new environment variable. Furthermore, the solution gets exceedingly more ugly, if I had to read in several paths from my config file as then each of them would again need their own environment variable.

Are there alternatives?

    ##main.py
    import configparser
    import os
    
    if __name__ == '__main__':
        # the section to read could be determined by args
        # rootDir= "rootDirOnTheServer" 
        rootDir= "rootDirAtHome"
    
        config = configparser.ConfigParser()
        config.read(filenames="paths")
        os.environ["MYVAL"] = rootDir
        import path
        from dataLoader import loadData
    
        train()
        print("path.getRoot() :", path.getRoot())
        print("path.subdirOfTheProject.base : ", path.subdirOfTheProject.base)
        print("Printing the path of dirOfTheProject:", path.dirOfTheProject.base)
    
    
    ###########################################
    ##paths.py
    import os
    
    
    def getRoot():
        val = os.environ["MYVAL"]
        return val
    
    
    class dirOfTheProject:
        base = getRoot() + os.sep + "inside the project dir"
    
    
    class subdirOfTheProject:
        base = dirOfTheProject.base + os.sep + "inside one of its subfoldrs"
    
    #######################################################
    ##dataLoader.py
    import path
    def loadData():
        src = path.subdirOfTheProject.base
        print("Here are all my data files: ", src)

[Larz60+]

Here's how I do it. This for an Oil Well project. All paths are relative, and are created if they don't already exist, when the script is first run. This allows simple access to all paths, and files if in path file. If you use sub directories, add a symbolic link and set depth=1 (the number of levels below main script path)

import os
from pathlib import Path


class OilWellPaths:
    def __init__(self, depth=0):
        os.chdir(os.path.abspath(os.path.dirname(__file__)))
        dir_depth = abs(depth)

        HomePath = Path(".")

        while dir_depth:
            HomePath = HomePath / ".."
            dir_depth -= 1

        rootpath = HomePath / ".."

        self.datapath = rootpath / "data"
        self.datapath.mkdir(exist_ok=True)

        self.csvpath = self.datapath / "csv"
        self.csvpath.mkdir(exist_ok=True)

        self.databasepath = self.datapath / "database"
        self.databasepath.mkdir(exist_ok=True)

        self.docpath = rootpath / "doc"
        self.docpath.mkdir(exist_ok=True)

        self.htmlpath = self.datapath / "html"
        self.htmlpath.mkdir(exist_ok=True)
        
        self.main_pagepath = self.htmlpath / 'MainPages'
        self.main_pagepath.mkdir(exist_ok=True)

        self.jsonpath = self.datapath / "json"
        self.jsonpath.mkdir(exist_ok=True)

        self.prettypath = self.datapath / "pretty"
        self.prettypath.mkdir(exist_ok=True)

        self.main_prettypath = self.prettypath / 'MainPages'
        self.main_prettypath.mkdir(exist_ok=True)

        self.logpath = self.datapath / "logs"
        self.logpath.mkdir(exist_ok=True)

        self.pdfpath = self.datapath / "PDF"
        self.pdfpath.mkdir(exist_ok=True)
        
        self.prettypath = self.datapath / "pretty"
        self.prettypath.mkdir(exist_ok=True)

        self.srcpath = rootpath / "src"
        self.srcpath.mkdir(exist_ok=True)

        self.textpath = self.datapath / "text"
        self.textpath.mkdir(exist_ok=True)

        self.tmppath = self.datapath / "tmp"
        self.tmppath.mkdir(exist_ok=True)

        # Database
        self.OilWellsDb = self.databasepath / "OilWells.db"

        # files
        self.state_info_file = self.tmppath / "StateInfoLis.html"
        self.statejson = self.jsonpath / "StateInfo.json"
        self.raw_state_list = self.textpath / "StateInfo.txt"
        self.state_well_info_json = self.jsonpath / "StateOilWellInfo.json"
        self.AlabamaProductionByCountyCsv = self.csvpath / "Alabama" / "AlabamaCountiesByProduction.csv"
        self.state_well_info_json_frozen = self.jsonpath / "StateOilWellInfoFrozen.json"


if __name__ == "__main__":
    OilWellPaths()

used like:

from OilWellPaths import OilWellPaths
import json


class GetCountyInfo:
    def __init__(self, backuphours=24):
        self.backuphours = backuphours
        self.owpath = OilWellPaths()

    def some_function(self):
        savefile = self.owpath.jsonpath / 'stateinfo.json'
        with savefile.open() as fp:
            stateinfo = json.load(fp)

def main():
    gci = GetCountyInfo()
    gci.some_function()

if __name__ == '__main__':
    main()