Custom importer and errors - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: General Coding Help (https://python-forum.io/forum-8.html) +--- Thread: Custom importer and errors (/thread-41954.html) |
Custom importer and errors - Fips - Apr-13-2024 Hi folks! I'd like to split my package tree into several IDE projects and build a custom importer to import 'top.child1.child2' from the directory <python-path-entry>/top.child1.child2/__init__.py so basically replacing the dots with slashes and having the package content lying directly in the project folder. I have come up with this: # usercustomize.py import sys from importlib.machinery import ModuleSpec from pathlib import Path Loader = type(__spec__.loader) class IdeHelper: @classmethod def find_spec(cls, name, path, target=None): for dirname in sys.path: dirobj = Path(dirname) if dirobj.name == name: break else: return None origin = str(dirobj.joinpath('__init__.py').absolute()) ret = ModuleSpec(name, Loader(name, origin), origin=origin) return ret sys.meta_path.append(IdeHelper)which I'm on the right direction with. Unfortunately, I'm getting errors while importing a subpackage. With 'import top.child1' the error is whereas with 'from top import child1' the error changes to How can I make this work?Best wishes, Fabiano RE: Custom importer and errors - Pedroski55 - Apr-14-2024 Try importing your modules "manually" first, see if that works?? import sys # temporarily add the path to your modules sys.path.append('/path/to/my/modules/')Now, if you actually have a module called top.py in the folder /path/to/my/modules/: import topshould work. But, to me, it looks like you only have a directory called top. Quote:ModuleNotFoundError: No module named 'top.child1'; 'top' is not a package Where are your modules? If you have a module called child1.py in the directory /top/ try this: sys.path.append('/top/') import child1should work. RE: Custom importer and errors - Fips - Apr-14-2024 How do you mean "import manually"? sys.path is not (generally) the problem, the directories are added as I expect: >>> sys.path ['', '/home/fips/src/importest/top', '/home/fips/src/importest/top.child1', '/home/fips/src/importest/top.child1.child2', ...]Also, importing the top package works, but it seems not to be regarded as a package? What does that even mean? Thanks for your reply anyway. RE: Custom importer and errors - Pedroski55 - Apr-14-2024 I believe Python "prepares" modules. In my folder for my own modules, /home/pedro/myPython/myModules/, there is a folder: __pycache__ __pycache__ contains all my home-made modules as .pyc files. I believe they are already machine language files, which makes them faster. Python will know if there are any new modules in /home/pedro/myPython/myModules/ and prepare the new module as a .pyc Maybe that is your "package problem"? Maybe your modules are not "prepared"? RE: Custom importer and errors - Fips - Apr-14-2024 Good idea, but I've compiled all *.py files manually/explicitly without any change in the resulted behaviour. Maybe the loader I reinstanciate needs to be told that 'top' is a package, not a module? RE: Custom importer and errors - Gribouillis - Apr-14-2024 I think all this is a terrible idea. In Python it is very important to distinguish the modules hierarchy from the OS folders hierarchy. These are really two different things and the idea of using folder names that contain dot characters '.' can only confuse the import mechanism.On your local machine, a simpler thing that you could do is use the __path__ attributes of packages to locate subpackages in another directory. For example suppose I have 3 directories top, child1, child2 all in the same folder, I can write# top/__init__.py from pathlib import Path __path__.append(str(Path(__file__).parent.parent)) def greet(module): print(f'Greetings from module {module.__name__}')and also # child1/__init__.py from pathlib import Path __path__.append(str(Path(__file__).parent.parent))and # child2/__init__.pyNow packages are properly imported >>> from top import greet >>> from top.child1 import child2 >>> greet(child2) Greetings from module top.child1.child2 >>>Using this system, one can play with trees of directories to locate subpackages elsewhere in the file system. That being said, it is not a good idea to make assumptions about the trees of directories when you want to package and distribute your modules. If you want to separate modules, write modules that can be installed independently, so you could write a top package, a top_child1 package and a top_child1_child2 package, each pip-intallable, and then you can write if you want# top/__init__.py import top_child1 as child1and # top_child1/__init__.py import top_child1_child2 as child2This is much more robust, but there are some differences, for example with this setup, top.child1 will not be considered by Python as a subpackage of top , so specific code like pkgutil that explore the subpackages will behave differently.
RE: Custom importer and errors - Pedroski55 - Apr-14-2024 Quote:Maybe the loader I reinstanciate needs to be told that 'top' is a package, not a module? I thought you were saying top.py is a module in some path of yours. If top is a package, maybe ending in .whl, it needs to be installed, presumably with pip, as far as I know! importlib will import modules "on-the-fly" I don't know if importlib can install packages, them import them. Read the docs. |