Architecture for script executor

[marsokod]

I am working on a tool to execute python scripts for hardware test automation. I have a current workflow but that I feel that it is the wrong way of doing things. My current tasks involve a flask server (using the development server) to display a web GUI. When the user wants to run a scripts, it loads in a iframe the jupyter page for the script (in this case, I save the scripts as jupyter notebooks, but it does not have to be). The user then runs the script manually. For the communication with the hardware, everything is done with MQTT messages so the test scripts only need to know MQTT, and not all the different hardware interfaces. For that, the servers loads plugins which are handling the translation of the MQTT messages to the hardware specific protocol.

Now, I already see some problems:

• it is not full automation. I would like to give the user the possibility of clicking a button "Run this script" and give a webpage where they can monitor the currently running scripts
  
• linked to the previous point, one script cannot run another script. That could be useful
  
• to embed jupyter in an iframe and in my application, I had to decrease the security of it
  
• I would like the possibility to run the script step by step, like with a debugger. And have the possibility to stop the script when some events occur, like when an error has been discovered on the hardware. - the flask server is just a development server
  
• having a flask server, the jupyter kernel and my own application, I am starting to feel lost in the different threads/processes
  

Do you know any module/tool that does something like that? Or who could help me building it in a better way. Since I will do a lot of sqlite queries/file access in it, I want all the parts accessing them to be aware of a multiprocessing Lock variable and I am not fully confident about doing that with Flask.

I need to run this on Windows, though I would love to have a solution working on any OS. I can expand on why I chose MQTT if there is any interest, but that is a part I would like not to change.