Oct-01-2019, 03:20 PM
Hello all!
I am awfully new to this site, but my lack of ideas has brought me here in hope that I might get help in here.
I am making an app, that is divided up into plugable modules that make up its functionality.
The general flow is like this: List a pre-defined directory, looking for any .py files and store those in a list.
Loop over the list and, using the importlib module (code snippet later), load the files into memory as python modules.
Using the inspect library, get all classes from the module object, returned in the last step, check if the class is a sub-class of one specifically pre-defined abstract class.
If all of those checks pass, instance the class and proceed with further functionality.
I have had all that working already, in the version of the app I made on Python 2, with the only difference being that instead of Importlib, I used the, now deprecated, imp library.
After upgrading to Python 3.7 and replacing imp with importlib, one key step broke - Inspect.getmembers now returns an empty list, instead of all the classes of the object. The code goes like this:
I already tried asking at StackOverflow, but received no responses, so I am now turning to this community. I am open to even bigger code modifications, if the idea stayed the same.
I am awfully new to this site, but my lack of ideas has brought me here in hope that I might get help in here.
I am making an app, that is divided up into plugable modules that make up its functionality.
The general flow is like this: List a pre-defined directory, looking for any .py files and store those in a list.
Loop over the list and, using the importlib module (code snippet later), load the files into memory as python modules.
Using the inspect library, get all classes from the module object, returned in the last step, check if the class is a sub-class of one specifically pre-defined abstract class.
If all of those checks pass, instance the class and proceed with further functionality.
I have had all that working already, in the version of the app I made on Python 2, with the only difference being that instead of Importlib, I used the, now deprecated, imp library.
After upgrading to Python 3.7 and replacing imp with importlib, one key step broke - Inspect.getmembers now returns an empty list, instead of all the classes of the object. The code goes like this:
import importlib.util
import logging
import inspect
import errno
import sys
import os
module_source_code_paths = []
module_instances =[]
module_inmemory_handles = []
self.logger = logging.getLogger(os.uname()[1]+"_addomain")
# Create handlers that the logging library will use. Here, we create two handlers
# c_handler logs to the console (stdout), while f_handler logs to the log file in /var/log/addomain.log
c_handler = logging.StreamHandler(stream=sys.stdout)
if self.arguments.debug:
f_handler = logging.FileHandler("./addomain.log", "a")
else:
f_handler = logging.FileHandler("/var/log/addomain.log", "a")
# Set desired logging levels
if self.arguments.verbose:
self.logger.setLevel(logging.DEBUG)
c_handler.setLevel(logging.DEBUG)
f_handler.setLevel(logging.DEBUG)
else:
self.logger.setLevel(logging.INFO)
c_handler.setLevel(logging.INFO)
f_handler.setLevel(logging.INFO)
# Set the appropriate log format strings
if self.arguments.debug:
c_format = logging.Formatter("[%(filename)s:%(lineno)s] - %(levelname)s - %(message)s")
else:
c_format = logging.Formatter("%(message)s")
f_format = logging.Formatter("[%(asctime)s] [%(filename)s:%(lineno)s] [%(levelname)s]: %(message)s")
# Add the formatters to their handlers
c_handler.setFormatter(c_format)
f_handler.setFormatter(f_format)
# Finally, add both handlers to the main logging object
self.logger.addHandler(c_handler)
self.logger.addHandler(f_handler)
#First, we get all the python files in a pre-defined directory
for (dirpath, dirnames, filenames) in os.walk(self.settings["MODULES_DIRECTORY"]):
for filename in filenames:
filename, file_extension = os.path.splitext(dirpath+filename)
# And if it is a python file (or appears to be one...), then add it to the module_paths list
# Used further to load them into memory and attempt to initialize their core class
if file_extension == ".py":
self.logger.debug("Found file "+filename+file_extension)
self.module_source_code_paths.append(filename + file_extension)
#Then we try to load those files into memory as modules, using the importlib.util.spec_from_file and importlib.util.module_from_spec functions, and save the returned object to another list
for module in self.module_source_code_paths:
dirpath, filename = os.path.split(module)
modname = os.path.splitext(module)[0]
spec = importlib.util.spec_from_file_location(modname, dirpath+filename)
self.module_inmemory_handles.append(importlib.util.module_from_spec(spec))
#Finally, we loop over the module objects and try to get a list of classes inside them, that we then check against several conditions, and save to another array
for modobj in self.module_inmemory_handles:
self.logger.debug(inspect.ismodule(modobj))
classes = inspect.getmembers(modobj, inspect.isclass) #This line is the issue, as this function used to return a list of classes from the module in Python 2, but in Python 3 returns an empty listAn example of a valid plugin file might be the following:from addomain import AddomainModuleBase
import errno
import os
import re
class ConfFilter(AddomainModuleBase):
#Thank you, stack overflow person, for this regex. Source: https://stackoverflow.com/a/20204811
__domain_regex = re.compile("(?=^.{4,253}$)(^((?!-)[a-zA-Z0-9-]{1,63}(?<!-)\.)+[a-zA-Z]{2,63}$)", re.IGNORECASE)
def __init__(self, settings, arguments):
if re.match(self.__domain_regex, arguments.domain) is None:
raise NameError("Invalid domain name!")
if "HOMEDIR" in settings:
if not os.path.isdir(settings["HOMEDIR"]):
raise OSError(errno.ENOENT, os.strerror(errno.ENOENT))
def run(self):
pass
def cleanup(self):
passThe entire code has been taken out of context, in snippets, so please, ignore the details like a lack of proper function / class definition. Each block of code resides in its own function, which in turn, is in a common class.I already tried asking at StackOverflow, but received no responses, so I am now turning to this community. I am open to even bigger code modifications, if the idea stayed the same.