Workflow Software

Many researchers develop scalable and reproducible scientific workflows using a workflow tools. There are several workslow tools that you can use on the HPC.

An important factor when choosing a wokflow software for use in HPC systems is that it must have support for the job scheduling software being used. In our case that is PBS. For each “step” or “task” within a workflow you need to be able to launch a PBS job and to be able to specify the CPU and memory requirements for that step.

Common Workflow Language

The Common Workflow Language site can be found here: https://www.commonwl.org
This is a language specification so for implementations of this see https://www.commonwl.org/#Implementations.
Some users in c3 are using cwltool: https://github.com/common-workflow-language/cwltool.
See also https://github.com/common-workflow-language for CWL viewers and utilities.

Cylc

Cylc: https://cylc.github.io/
This looks quite good. It has support for PBS.

Nextflow

Nextflow: https://www.nextflow.io
Many users running genomics pipelines are using this. We have a short page on using Nextflow here: Using Nextflow

References

Wikipedia entry for the Common Workflow Language https://en.wikipedia.org/wiki/Common_Workflow_Language

Tutorial: How to run CWL with Nextflow https://www.biostars.org/p/422848/