Custom importer and errors

[Fips]

Hi folks!

I'd like to split my package tree into several IDE projects and build a custom importer to import
'top.child1.child2'
from the directory
<python-path-entry>/top.child1.child2/__init__.py
so basically replacing the dots with slashes and having the package content lying directly in the project folder. I have come up with this:

# usercustomize.py

import sys
from importlib.machinery import ModuleSpec
from pathlib import Path

Loader = type(__spec__.loader)

class IdeHelper:
    @classmethod
    def find_spec(cls, name, path, target=None):
        for dirname in sys.path:
            dirobj = Path(dirname)
            if dirobj.name == name:
                break
        else:
            return None
        origin = str(dirobj.joinpath('__init__.py').absolute())
        ret = ModuleSpec(name, Loader(name, origin), origin=origin)
        return ret

sys.meta_path.append(IdeHelper)

which I'm on the right direction with. Unfortunately, I'm getting errors while importing a subpackage. With 'import top.child1' the error is

Error:
ModuleNotFoundError: No module named 'top.child1'; 'top' is not a package

whereas with 'from top import child1' the error changes to

Error:
ImportError: cannot import name 'child1' from 'top' (unknown location)

How can I make this work?

Best wishes,
Fabiano

[Pedroski55]

Try importing your modules "manually" first, see if that works??

import sys

# temporarily add the path to your modules
sys.path.append('/path/to/my/modules/')

Now, if you actually have a module called top.py in the folder /path/to/my/modules/:

import top

should work.

But, to me, it looks like you only have a directory called top.

ModuleNotFoundError: No module named 'top.child1'; 'top' is not a package

Where are your modules?

If you have a module called child1.py in the directory /top/ try this:

sys.path.append('/top/')
import child1

should work.

[Fips]

How do you mean "import manually"?

sys.path is not (generally) the problem, the directories are added as I expect:

>>> sys.path
['',
 '/home/fips/src/importest/top',
 '/home/fips/src/importest/top.child1',
 '/home/fips/src/importest/top.child1.child2',
 ...]

Also, importing the top package works, but it seems not to be regarded as a package? What does that even mean?

Thanks for your reply anyway.

[Pedroski55]

I believe Python "prepares" modules. In my folder for my own modules, /home/pedro/myPython/myModules/, there is a folder: __pycache__

__pycache__ contains all my home-made modules as .pyc files.

I believe they are already machine language files, which makes them faster.

Python will know if there are any new modules in /home/pedro/myPython/myModules/ and prepare the new module as a .pyc

Maybe that is your "package problem"? Maybe your modules are not "prepared"?

[Fips]

Good idea, but I've compiled all *.py files manually/explicitly without any change in the resulted behaviour.
Maybe the loader I reinstanciate needs to be told that 'top' is a package, not a module?

[Gribouillis]

I think all this is a terrible idea. In Python it is very important to distinguish the modules hierarchy from the OS folders hierarchy. These are really two different things and the idea of using folder names that contain dot characters '.' can only confuse the import mechanism.

On your local machine, a simpler thing that you could do is use the __path__ attributes of packages to locate subpackages in another directory. For example suppose I have 3 directories top, child1, child2 all in the same folder, I can write

# top/__init__.py

from pathlib import Path

__path__.append(str(Path(__file__).parent.parent))

def greet(module):
    print(f'Greetings from module {module.__name__}')

and also

# child1/__init__.py

from pathlib import Path

__path__.append(str(Path(__file__).parent.parent))

and

# child2/__init__.py

Now packages are properly imported

>>> from top import greet
>>> from top.child1 import child2
>>> greet(child2)
Greetings from module top.child1.child2
>>> 

Using this system, one can play with trees of directories to locate subpackages elsewhere in the file system.

That being said, it is not a good idea to make assumptions about the trees of directories when you want to package and distribute your modules. If you want to separate modules, write modules that can be installed independently, so you could write a top package, a top_child1 package and a top_child1_child2 package, each pip-intallable, and then you can write if you want

# top/__init__.py
import top_child1 as child1

and

# top_child1/__init__.py
import top_child1_child2 as child2

This is much more robust, but there are some differences, for example with this setup, top.child1 will not be considered by Python as a subpackage of top, so specific code like pkgutil that explore the subpackages will behave differently.

[Pedroski55]

Maybe the loader I reinstanciate needs to be told that 'top' is a package, not a module?

I thought you were saying top.py is a module in some path of yours.

If top is a package, maybe ending in .whl, it needs to be installed, presumably with pip, as far as I know!

importlib will import modules "on-the-fly" I don't know if importlib can install packages, them import them. Read the docs.