Python Forum

I have an environment (tinycore linux) where as the system boots, python modules get loaded. The program needs to start up quickly, and it takes a while to load some modules like scipy.

In python2, this was not a problem, I could:

try:
    import scipy
except:
    return

and try again later.

In python3, this does not work. I have to restart the process, it always fails even long after the module is available if it was not available when python3 started.

I tried
import importlib
importlib.invalidate_caches()

but this does not work either, it doesn't solve the problem.

I know it's possible to use multiprocessing and communicate over pipes to get around this issue but it's really making the code complicated and less efficient.

So, how do I get the interpreter to rescan the available modules after it's started since invalidate_caches doesn't do what it claims to?

What causes the initial failure? Are fileystems not mounted so you get a ModuleNotFoundError, or does something else happen?

I would suspect that importlib.reload(module) would be sufficient, but that depends on what the initial failure is.

ModuleNotFoundError: No module named 'scipy'

you cannot reload it because the module is not found so from importlib.reload(scipy)
NameError: name 'scipy' is not defined

Should I install a stub module that exists at boot then reload it later?

Are filesystems getting mounted? I don't see why a later import would fail here if the modules are getting introduced into the correct path.

>>> import os
>>> os.rename("tmdbsimple", "tmdbsimple.moved")
>>> import tmdbsimple
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'tmdbsimple'
>>> os.rename("tmdbsimple.moved", "tmdbsimple")
>>> import tmdbsimple
>>>

It turns out this only affects modules that use a .pth file to redirect where they are located.

so I have scipy.pth in /usr/local/lib/python3.6/site-packages
and in scipy.pth the path to the scipy

When I rearranged it so the module was located in the scipy directory and did not need this file, it now works as I needed it to.