Python Forum
free open-source automation task-runner project
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
free open-source automation task-runner project
#1
this is an open-source taskrunner cli & api automation tool. All lovingly made with Python!

https://github.com/pypyr/pypyr

use this if your bash or even python scripts start getting out of hand. some people use the api as a user-friendly way for end-users to create workflow/pipeline definition for tasks/jobs.

the core project has slowly been building and adding new features for a few years now, and it’s in use in a few companies I worked at.

I just made it a new documentation website for the automation task runner rather than just a big ol’ bloated README on github, so I’m less ashamed to share it around Dance So there you go, hope it’s useful!
Reply
#2
It looks like systemd on linux, isn't it ?
Reply
#3
(Sep-13-2020, 06:49 PM)jimy_byerley Wrote: It looks like systemd on linux, isn't it ?

hey, thanks for checking out the project, I much appreciate it!

haha, I can't quite tell if you mean "it looks like systemd" in the controversial sense because of all the hate systemd has gotten over the years. . . Big Grin

I personally wouldn't describe it as similar to systemd - systemd is just a way of creating & configuring run-time services on linux. . .

pypyr is a task-runner, so you can use yaml to define your own sequences of tasks with quite complex workflows. . . For examples, if boolean condition a then run this cmd, but don't run the other command, or jump to another sequences of steps. . . and with that you get automatic retries if you want. . .so you can model much more complex workflows or automation sequences than you would in systemd.

So for example, if you want to automate some tasks that reads values from a local .json value, do some manipulation of those values and then submit them to a web-server somewhere, you can use a task runner like pypyr to automate that without having to do all the repetitive/boring/boilerplate code you typically have to do for error handling and retry loops.
Daring_T likes this post
Reply
#4
Hi,

I am currently using subprocess module to automate my numerous simulation on TRNSYS. can pypyr simulate it automatically without much coding (I am a mechanical engineer, doing coding because there is no option).

My code in pyhthon right now has five for loops nested and its making me very difficult to store and manage output files according to number of simulation.

Can pypyr help?

(I checked out the webpage for it. seemed very difficult for me as I am used to the various commands or eevn terminology used on that page to explain the project).
Reply
#5
(Sep-19-2020, 12:21 PM)3SG14 Wrote: My code in pyhthon right now has five for loops nested and its making me very difficult to store and manage output files according to number of simulation.

Can pypyr help?

Hi @3SG14,

Yes, pypyr can certainly help. I saw your other post where you describe your issue in greater detail. I don't know exactly what you're trying to achieve, but the sample code you gave would look something like this in a pypyr pipeline:

# trnsys-simulate.yaml
steps:
  - name: pypyr.steps.default
    comment: set parameters to be evaluated
    in:
      defaults:
        label: 1
        # Volume of seasonal storage tank
        volume_iss: ['1', '2', '3']
        # Height of seasonal storage tank
        height_iss: ['4', '5', '6']
        # Volume of DH tank
        volume_dhw: ['6', '8']
        # Height of DH tank
        height_dhw: ['9']
        # Area of flat plate solar collector
        area_sc: ['10', '11', '12']
  - name: pypyr.steps.py
    comment: create the cartesian product of all input iterables
             save result to parameter_matrix, which the next step
             will loop through.
    in:
      pycode: |
        from itertools import product
        context['parameter_matrix'] = list(product(
                                                  context['volume_iss'],
                                                  context['height_iss'],
                                                  context['volume_dhw'],
                                                  context['height_dhw'],
                                                  context['area_sc']))
  - name: pypyr.steps.call
    comment: loop through the input combinations and run simulation for each
    foreach: '{parameter_matrix}'
    in:
      call: simulate

simulate:
  - name: pypyr.steps.contextsetf
    comment: create unique input file name for this iteration
    in:
      contextSetf:
        out_dir: "./out/{label:04d}"
        input_file: "{out_dir}/IRLPHv00.dck"
        label: !py label + 1
  - name: pypyr.steps.filereplace
    comment: create input file from template, by replacing tags w parameter values
             i is the iterator from the foreach
    in:
      fileReplace:
        in: z_PH_TEMPLATE.dck
        out: '{input_file}'
        replacePairs:
          PH_VolumeISS: '{i[0]}'
          PH_HeightISS: '{i[1]}'
          PH_VolumeDHW: '{i[2]}'
          PH_HeightDHW: '{i[3]}'
          PH_AreaSC: '{i[4]}'
  - name: pypyr.steps.cmd
    comment: run trnsys cmd with the input file created by previous step
    in:
      cmd: C:/TRNSYS18/Exe\TrnEXE64.exe "{input_file}" /h
  - name: pypyr.steps.cmd
    comment: move or copy outfiles to outdir using your preferred console cmd
    in:
      cmd: cp *.out {out_dir}/
If you save this file as trnsys-simulate.yaml, you will run it from the same directory like this:
Output:
pypyr trnsys-simulate
This will create in the out_dir directories like this:
out/0001/
out/0002/
out/0003/

It will also copy the input file (the dck file for that particular combination of parameters) so you can double-check everything as expected.

WARNING: I don't have trnsys on my system, nor do I know exactly what you're trying to achieve, and also I am not on a windows machine: I did test the above on linux/macOS, but I'm pretty sure you're going to have to fiddle with the out_dir, input_file and so on to make it appropriate for windows. So please don't expect it just to work first time after a copy/paste!
Daring_T likes this post
Reply
#6
(Sep-20-2020, 03:00 AM)yaythomas Wrote:
(Sep-19-2020, 12:21 PM)3SG14 Wrote: My code in pyhthon right now has five for loops nested and its making me very difficult to store and manage output files according to number of simulation.

Can pypyr help?

Hi @3SG14,

Yes, pypyr can certainly help. I saw your other post where you describe your issue in greater detail. I don't know exactly what you're trying to achieve, but the sample code you gave would look something like this in a pypyr pipeline:

# trnsys-simulate.yaml
steps:
  - name: pypyr.steps.default
    comment: set parameters to be evaluated
    in:
      defaults:
        label: 1
        # Volume of seasonal storage tank
        volume_iss: ['1', '2', '3']
        # Height of seasonal storage tank
        height_iss: ['4', '5', '6']
        # Volume of DH tank
        volume_dhw: ['6', '8']
        # Height of DH tank
        height_dhw: ['9']
        # Area of flat plate solar collector
        area_sc: ['10', '11', '12']
  - name: pypyr.steps.py
    comment: create the cartesian product of all input iterables
             save result to parameter_matrix, which the next step
             will loop through.
    in:
      pycode: |
        from itertools import product
        context['parameter_matrix'] = list(product(
                                                  context['volume_iss'],
                                                  context['height_iss'],
                                                  context['volume_dhw'],
                                                  context['height_dhw'],
                                                  context['area_sc']))
  - name: pypyr.steps.call
    comment: loop through the input combinations and run simulation for each
    foreach: '{parameter_matrix}'
    in:
      call: simulate

simulate:
  - name: pypyr.steps.contextsetf
    comment: create unique input file name for this iteration
    in:
      contextSetf:
        out_dir: "./out/{label:04d}"
        input_file: "{out_dir}/IRLPHv00.dck"
        label: !py label + 1
  - name: pypyr.steps.filereplace
    comment: create input file from template, by replacing tags w parameter values
             i is the iterator from the foreach
    in:
      fileReplace:
        in: z_PH_TEMPLATE.dck
        out: '{input_file}'
        replacePairs:
          PH_VolumeISS: '{i[0]}'
          PH_HeightISS: '{i[1]}'
          PH_VolumeDHW: '{i[2]}'
          PH_HeightDHW: '{i[3]}'
          PH_AreaSC: '{i[4]}'
  - name: pypyr.steps.cmd
    comment: run trnsys cmd with the input file created by previous step
    in:
      cmd: C:/TRNSYS18/Exe\TrnEXE64.exe "{input_file}" /h
  - name: pypyr.steps.cmd
    comment: move or copy outfiles to outdir using your preferred console cmd
    in:
      cmd: cp *.out {out_dir}/
If you save this file as trnsys-simulate.yaml, you will run it from the same directory like this:
Output:
pypyr trnsys-simulate
This will create in the out_dir directories like this:
out/0001/
out/0002/
out/0003/

It will also copy the input file (the dck file for that particular combination of parameters) so you can double-check everything as expected.

WARNING: I don't have trnsys on my system, nor do I know exactly what you're trying to achieve, and also I am not on a windows machine: I did test the above on linux/macOS, but I'm pretty sure you're going to have to fiddle with the out_dir, input_file and so on to make it appropriate for windows. So please don't expect it just to work first time after a copy/paste!

Angel Angel
Thank you so much for such a detailed solution... I am giving the last try to my approach now. If I am not able to find what I want I will try to copy your code and modify according to windows if required (will let you how it worked if I use it). Thanks for such a prompt reply. Appreciate it
Reply
#7
best wishes, hope you succeed!

this approach is using a low-code/no-code solution, so since coding isn't your "day job" per se, I thought it might be useful. I'd be very happy to step you through the low-code pypyr pipeline and explain some more if you'd like, let's see if we can get it working well for you Big Grin .
Reply
#8
Nice project, but I'd like a comparison with other python tools that I know of, such as the doit project. Did you find some inspiration in such projects?
Reply
#9
thanks Gribouillis, much appreciated!

The main inspiration for pypyr was. . . whenever I was writing automation scripts, typically it'd start off in bash, and then at some point of complexity you end up switching to python for the enhanced expressivity with control of flow and error handling. . . but then, you often need to jump back into the shell for some convenient cli tools that work better from the cli than the api. Also, for every automation script I ended up having to re-invent error handling, loops, input argument checks etc. So back then I coded a simple tool to let me do that easily without having to write boilerplate code - and that was the humble beginnings of pypyr. I open-sourced it from the beginning just in case someone else also found it useful, and I just added to it over the years as I used it for more and more things and other contributors have added some amazing work, so it's grown a lot!

The reason I wrote my own utility to do so is even though you can use something like make/gradle/ant to do a lot, or even all, of these things, it's not really what these tools were meant for, so you end up dealing with quite a lot of complexity when sometimes you really just want something a bit more lightweight and easy!

doit is a great project! I'd say that it addresses a different problem area than pypyr. So I don't want to use language like "better" or "worse" - nothing I'm about to say means anything is bad, it's just every different approach has different advantages and disadvantages, different tools for different jobs. The main areas of difference, for me at least, is this:
  • doit is based on writing code. pypyr is based on yaml pipelines, so it's more a no-code to low-code style solution.
  • doit is much closer to a typical build system in the spirit of make, ant, grunt or gulp. You think in terms of sources, build targets and actions. pypyr, by comparison, is more about sequencing tasks one after the after, in a pipeline.
  • doit is super extensible for anything you want, but you have to provide a lot of that functionality yourself by writing some code (this means you can get exactly what you want because you write it yourself). pypyr has ~30+ built-in no-code-necessary steps to do things like parse files, write json/yaml, work with $ENV vars for you - although you can still very easily add in your code custom code if you want to - but very often you can create a pypyr pipeline without having to write any code and just use the built-in steps.
  • A pypyr pipeline gives you more visually at-a-glance intuitive control of flow with conditionally executing certain steps, running loops, handling errors & retries automatically. . . But you don't have the advanced code-your-own flexibility and automatic dependency tracking that doit does very well.

In summary, if you're looking for a build tool to manage complex dependency chains, do incremental builds, that's more doit. If you're looking to sequence your devops operations with serial or parallel steps *after* that build sequence, that's more pypyr - interacting with external services, sequencing cmds to deploy your services, manipulate config files & templates, do status checks, retry operations if they fail, iterate over collections, send notifications.

Hope this helps!
Daring_T likes this post
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Open source RTS game Forward Only IlyaFaer 11 6,178 Aug-29-2022, 06:38 AM
Last Post: IlyaFaer
  Slack NLP : open source bhagvanarch 0 2,885 Dec-30-2018, 12:08 PM
Last Post: bhagvanarch
  Monte Carlo : Open source project bhagvanarch 0 2,578 Dec-29-2018, 11:28 AM
Last Post: bhagvanarch
  Camelot (Open source) - Extract tabular data from PDF! sidharthwadhwa 1 3,044 Oct-15-2018, 11:17 AM
Last Post: buran

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020