importing a module given a path to the file

[Skaperen]

i want to have my script take a path to a file that it gets (may be typed in, or read from some other file, or given as a command argument) and import it as a module without any copying and/or renaming of files. i vaguely remember that i saw, a few years ago, a function that maybe could do this, but i cannot recall what it is or where i saw it, now that i need it. any ideas?

[Gribouillis]

I have written such functions. Use them freely: I declare them to be under the MIT license

# SPDX-FileCopyrightText: 2023 Eric Ringeisen
# SPDX-License-Identifier: MIT

def import_from_source_file(module_name, file_path):
    """Import a source file as a module
    
    Arguments:
        @module_name: the qualified name of the module
        @file_path: the path to the file containing the module's code
        
    If a module with that name already exists in sys.modules,
    nothing is done and the existing module is returned.
    
    This function does not attempt to import the parent module
    nor to insert it in sys.modules.
    """
    from pathlib import Path
    import sys
    try:
        return sys.modules[module_name]
    except KeyError:
        pass
    # from a recipe in importlib.machinery's documentation
    import importlib.util
    if Path(file_path).is_dir():
        file_path = str(Path(file_path)/'__init__.py')
    spec = importlib.util.spec_from_file_location(module_name, file_path)
    if spec is None:
        raise ImportError(
            'Cannot get module spec from file location', file_path)
    module = importlib.util.module_from_spec(spec)
    sys.modules[module_name] = module
    spec.loader.exec_module(module)
    return module

def import_tree(root_name, tree):
    """Import a tree of files as submodules of a root module
    
    Arguments:
        @root_name: the qualified name of module
        @tree: a dictionary mapping keys to path to files or similar
            dictionaries
    
    Returns:
        the root module or None if root_name is ''
    
    Example:
        The call
        
        import_tree('spam', {
            'foo': 'bar.py',
            'baz': {'x': 'tmp/xxx.py', 'y': 'yy32.py'},
            'qux': 'qux/quux.py'})
            
        will create the modules
        
            spam
            spam.foo (from file bar.py)
            spam.baz
            spam.baz.x (from file tmp/xxx.py)
            spam.baz.y (from file yy32.py)
            spam.qux (from file qux/qyyx.py)
        
        These modules are inserted in sys.modules.
        Any already existing modules are not created.
    """
    from importlib import import_module
    import sys
    from types import ModuleType
    if root_name:
        try:
            root = import_module(root_name)
        except ImportError:
            root = sys.modules[root_name] = ModuleType(root_name)
    else:
        root = None
    def handle(mod, prefix, key, value):
        s = key if not prefix else (prefix + '.' + key)
        try:
            m = import_module(s)
        except ImportError:
            if isinstance(value, dict):
                m = sys.modules[s] = ModuleType(s)
            else:
                m = import_from_source_file(s, value)
            if mod:
                setattr(mod, key, m)
        if isinstance(value, dict):
            for k, v in value.items():
                handle(m, s, k, v)
    for key, value in tree.items():
        handle(root, root_name, key, value)
    return root

[Skaperen]

how much of that file is the important stuff plus what else it needs to work vs. how much of that file is sample code to show how to use it?

i figured out a way to accomplish my need without importing a module. my need is to read in a file that sets many variables with values that can be expressed in Python, including expressions that can involve previously set variables. what this way does is read in the code as a big string and call the builtin exec() function. what exec() returns is the dictionary i wanted to derive from that file of assignments.

initially, i thought importing the file as a module would be most direct though i would still need to get the assigned values out of it. then i though o using exec to import it by name by making the import statement code. while thinking over how to do that in various paths, it became clear that by just reading the file at that path, exec() does all i need.

while playing with an implementation, i found i could set values in the exec() global space and have assignments in the file get values from variable names so the files do not all need literals.

i am using this to create config files for a few scripts i am developing to manage various resources in the AWS cloud. the first reason i went with using Python code was to be able to do multi-line triple-quoted values, such as quoting a few lines to hold a list of resources or a script to run in a cloud instance".

[Gribouillis]

how much of that file is the important stuff plus what else it needs to work vs. how much of that file is sample code to show how to use it?

There is no sample code in that file. It contains only two functions. One of them allows one to import a python file as one would import a module, in a way that is compatible with Python's importlib mechanism. The other one allows one to import a bunch of python files as a tree of modules and submodules.

Of course, a simple exec may address your needs if you don't need to fill a module with this code.

i am using this to create config files for a few scripts i am developing to manage various resources in the AWS cloud.

I'm currently writing a library to manage configuration files. I'll make an announcement on this forum when it is ready to be used!

[Skaperen]

I'm currently writing a library to manage configuration files. I'll make an announcement on this forum when it is ready to be used!

will it allow a variety of different kinds of "config" files?

i have been using Python3 as my config file language by importing it as a module. i have been play with adding a number of tools to this. now it seem exec() might be a better way to go. i can pre-set variables for config files to use instead of literals. with import i have to add them to the config file and save it as a temporary. with exec() i can pass a dictionary in argument 2 and they don't get included in the return dictionary. exec() seems to be the clean way to go.

my latest project involves managing multiple projects and sub-projects where each might need config info for individual projects with variable inheritance from parent projects.

[Gribouillis]

will it allow a variety of different kinds of "config" files?

Yes it will be extensible with an arbitrary number of configuration 'protocols'. Currently I have 3 protocols, one is 'configparser' and another one is called 'methodic' and it is a pure python protocol. But the library will be easily extensible to other protocols such as 'json' or 'yaml' or your own custom protocols.

my latest project involves managing multiple projects and sub-projects where each might need config info for individual projects with variable inheritance from parent projects.

I'm working hard to finish my library. I'm sure it could help you. The library's main feature is that you are free to store configuration files anywhere in the file system, potentially even inside zip files or databases. I've been using a very preliminary version of this library for two years now and the concept is pleasantly flexible.

[Skaperen]

I'm currently writing a library to manage configuration files. I'll make an announcement on this forum when it is ready to be used!

will it have a Python API?

[Gribouillis]

will it have a Python API?

Yes it is pure Python code with an API and a command line interface.

[Skaperen]

i still like to write config files in Python. so, i'd like to have some kind of support for that. it needs to have a way to be controlled for security in cases where the config or data it uses comes from untrusted sources like a web query. advantages include:

1. config settings can have relations between each other such as a number setting that is 3 times plus 72 more than another number setting or a string that is the same as part of another.

2. the ease of a multi-line string that triple-quoting has.

3. accessing information from the file system or other files to establish settings.

4. the ability to do calculations that ordinary config files cannot do.