Oct-01-2019, 03:20 PM
Hello all!
I am awfully new to this site, but my lack of ideas has brought me here in hope that I might get help in here.
I am making an app, that is divided up into plugable modules that make up its functionality.
The general flow is like this: List a pre-defined directory, looking for any .py files and store those in a list.
Loop over the list and, using the importlib module (code snippet later), load the files into memory as python modules.
Using the inspect library, get all classes from the module object, returned in the last step, check if the class is a sub-class of one specifically pre-defined abstract class.
If all of those checks pass, instance the class and proceed with further functionality.
I have had all that working already, in the version of the app I made on Python 2, with the only difference being that instead of Importlib, I used the, now deprecated, imp library.
After upgrading to Python 3.7 and replacing imp with importlib, one key step broke - Inspect.getmembers now returns an empty list, instead of all the classes of the object. The code goes like this:
An example of a valid plugin file might be the following:
The entire code has been taken out of context, in snippets, so please, ignore the details like a lack of proper function / class definition. Each block of code resides in its own function, which in turn, is in a common class.
I already tried asking at StackOverflow, but received no responses, so I am now turning to this community. I am open to even bigger code modifications, if the idea stayed the same.
I am awfully new to this site, but my lack of ideas has brought me here in hope that I might get help in here.
I am making an app, that is divided up into plugable modules that make up its functionality.
The general flow is like this: List a pre-defined directory, looking for any .py files and store those in a list.
Loop over the list and, using the importlib module (code snippet later), load the files into memory as python modules.
Using the inspect library, get all classes from the module object, returned in the last step, check if the class is a sub-class of one specifically pre-defined abstract class.
If all of those checks pass, instance the class and proceed with further functionality.
I have had all that working already, in the version of the app I made on Python 2, with the only difference being that instead of Importlib, I used the, now deprecated, imp library.
After upgrading to Python 3.7 and replacing imp with importlib, one key step broke - Inspect.getmembers now returns an empty list, instead of all the classes of the object. The code goes like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 |
import importlib.utilimport loggingimport inspectimport errnoimport sysimport os module_source_code_paths = [] module_instances =[] module_inmemory_handles = [] self.logger = logging.getLogger(os.uname()[1]+"_addomain") # Create handlers that the logging library will use. Here, we create two handlers # c_handler logs to the console (stdout), while f_handler logs to the log file in /var/log/addomain.log c_handler = logging.StreamHandler(stream=sys.stdout) if self.arguments.debug: f_handler = logging.FileHandler("./addomain.log", "a") else: f_handler = logging.FileHandler("/var/log/addomain.log", "a") # Set desired logging levels if self.arguments.verbose: self.logger.setLevel(logging.DEBUG) c_handler.setLevel(logging.DEBUG) f_handler.setLevel(logging.DEBUG) else: self.logger.setLevel(logging.INFO) c_handler.setLevel(logging.INFO) f_handler.setLevel(logging.INFO) # Set the appropriate log format strings if self.arguments.debug: c_format = logging.Formatter("[%(filename)s:%(lineno)s] - %(levelname)s - %(message)s") else: c_format = logging.Formatter("%(message)s") f_format = logging.Formatter("[%(asctime)s] [%(filename)s:%(lineno)s] [%(levelname)s]: %(message)s") # Add the formatters to their handlers c_handler.setFormatter(c_format) f_handler.setFormatter(f_format) # Finally, add both handlers to the main logging object self.logger.addHandler(c_handler) self.logger.addHandler(f_handler)#First, we get all the python files in a pre-defined directory for (dirpath, dirnames, filenames) in os.walk(self.settings["MODULES_DIRECTORY"]): for filename in filenames: filename, file_extension = os.path.splitext(dirpath+filename) # And if it is a python file (or appears to be one...), then add it to the module_paths list # Used further to load them into memory and attempt to initialize their core class if file_extension == ".py": self.logger.debug("Found file "+filename+file_extension) self.module_source_code_paths.append(filename + file_extension)#Then we try to load those files into memory as modules, using the importlib.util.spec_from_file and importlib.util.module_from_spec functions, and save the returned object to another list for module in self.module_source_code_paths: dirpath, filename = os.path.split(module) modname = os.path.splitext(module)[0] spec = importlib.util.spec_from_file_location(modname, dirpath+filename) self.module_inmemory_handles.append(importlib.util.module_from_spec(spec))#Finally, we loop over the module objects and try to get a list of classes inside them, that we then check against several conditions, and save to another array for modobj in self.module_inmemory_handles: self.logger.debug(inspect.ismodule(modobj)) classes = inspect.getmembers(modobj, inspect.isclass) #This line is the issue, as this function used to return a list of classes from the module in Python 2, but in Python 3 returns an empty list |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
from addomain import AddomainModuleBaseimport errnoimport osimport reclass ConfFilter(AddomainModuleBase): #Thank you, stack overflow person, for this regex. Source: https://stackoverflow.com/a/20204811 __domain_regex = re.compile("(?=^.{4,253}$)(^((?!-)[a-zA-Z0-9-]{1,63}(?<!-)\.)+[a-zA-Z]{2,63}$)", re.IGNORECASE) def __init__(self, settings, arguments): if re.match(self.__domain_regex, arguments.domain) is None: raise NameError("Invalid domain name!") if "HOMEDIR" in settings: if not os.path.isdir(settings["HOMEDIR"]): raise OSError(errno.ENOENT, os.strerror(errno.ENOENT)) def run(self): pass def cleanup(self): pass |
I already tried asking at StackOverflow, but received no responses, so I am now turning to this community. I am open to even bigger code modifications, if the idea stayed the same.