Apr-09-2023, 03:26 PM
Hi,
I recently started learning Python, so I tried to write this project as an exercise. The idea of the concept is derived from the solution to the difficulties encountered when helping the Ministry of Finance to develop the system. Share it here.
![[Image: sg1lCzG.png]](https://i.imgur.com/sg1lCzG.png)
![[Image: V1Qri4h.gif]](https://i.imgur.com/V1Qri4h.gif)
Link
Motivation
JDR (Job Dependency Runner) is a set of small data governance tools developed by this project. In short, it is a set of "programs used to assist in the execution and management of programs".
At work, the action of "executing a program" is not particularly difficult in most cases. Usually, you edit the command first, then throw it into the shell, or an interface/platform, and then wait for the result to come out. Will use tools like crontab to pre-schedule.
With this method, if the scale is only one or two to a dozen programs, there may be no problem, but if there are hundreds or thousands of programs, it will be difficult to manage. The reason lies in the management issues derived from "quantity" and "dependency"
These management issues include: "What is the current state of the program?", "What is the sequence of program execution?", "If a certain program needs to be re-run, will it affect which downstream related programs?" When the number of programs is larger, it is less likely to be managed by the engineer's memory. Even if the records are assisted by files, maintenance and searching will take time and cost.
And because data analysis has become more and more important in recent years, the data governance issue of "whether the program is executed correctly and on time" has also been paid more and more attention. In order to solve these issues, I hope to implement a set of tools in this project, so that some management issues can be automated, dashboarded, and the results are presented in a visual way.
Maybe this project will overlap with some ETL tools (such as: SSIS, Trinity, DataStage, Automation) in function, because ETL tools also have the function of executing and managing programs, but because I haven't found a tool that can meet the needs , so that's another reason why I decided I wanted to develop my own.
I hope that users only need to maintain a work list (Excel format), and then after inputting the list into this tool, a graphical program dependency flow chart can be automatically generated. The graphical program dependency flowchart is a kind of DAG (Directed Acyclic Graph). After having a graph, many issues arise about how to operate it. I try to simplify these operations as much as possible, so that these operations and management behaviors can be easily performed only by making a setting on the graphical interface, pressing a button, and viewing a report.
Everyone is welcome to use this set of tools, but the design of the tools is based on my personal previous development experience and my own imagination, so if someone thinks that it is not easy to use, inconvenient, or not flexible enough, please feel free to feed these questions back to me, so that I can use them as a reference for improvement.
I recently started learning Python, so I tried to write this project as an exercise. The idea of the concept is derived from the solution to the difficulties encountered when helping the Ministry of Finance to develop the system. Share it here.
Figure 1. Appearance of JDR tool
Figure 2. Using JDR tools to execute and manage programs
Link
- Source code: https://github.com/Chen-Alfred/JDR
- Execution file: https://github.com/Chen-Alfred/JDR/tree/main/dist
- Documentation (English): https://hackmd.io/xsLDRVAMTF2gO0YHo3lxYw
Motivation
JDR (Job Dependency Runner) is a set of small data governance tools developed by this project. In short, it is a set of "programs used to assist in the execution and management of programs".
At work, the action of "executing a program" is not particularly difficult in most cases. Usually, you edit the command first, then throw it into the shell, or an interface/platform, and then wait for the result to come out. Will use tools like crontab to pre-schedule.
With this method, if the scale is only one or two to a dozen programs, there may be no problem, but if there are hundreds or thousands of programs, it will be difficult to manage. The reason lies in the management issues derived from "quantity" and "dependency"
These management issues include: "What is the current state of the program?", "What is the sequence of program execution?", "If a certain program needs to be re-run, will it affect which downstream related programs?" When the number of programs is larger, it is less likely to be managed by the engineer's memory. Even if the records are assisted by files, maintenance and searching will take time and cost.
And because data analysis has become more and more important in recent years, the data governance issue of "whether the program is executed correctly and on time" has also been paid more and more attention. In order to solve these issues, I hope to implement a set of tools in this project, so that some management issues can be automated, dashboarded, and the results are presented in a visual way.
Maybe this project will overlap with some ETL tools (such as: SSIS, Trinity, DataStage, Automation) in function, because ETL tools also have the function of executing and managing programs, but because I haven't found a tool that can meet the needs , so that's another reason why I decided I wanted to develop my own.
I hope that users only need to maintain a work list (Excel format), and then after inputting the list into this tool, a graphical program dependency flow chart can be automatically generated. The graphical program dependency flowchart is a kind of DAG (Directed Acyclic Graph). After having a graph, many issues arise about how to operate it. I try to simplify these operations as much as possible, so that these operations and management behaviors can be easily performed only by making a setting on the graphical interface, pressing a button, and viewing a report.
Everyone is welcome to use this set of tools, but the design of the tools is based on my personal previous development experience and my own imagination, so if someone thinks that it is not easy to use, inconvenient, or not flexible enough, please feel free to feed these questions back to me, so that I can use them as a reference for improvement.