Debian Science Project
Summary
Workflow
workflow management systems useful for scientific research

This task lists some packages providing workflow management systems useful for scientific research.

Description

For a better overview of the project's availability as a Debian package, each head row has a color code according to this scheme:

If you discover a project which looks like a good candidate for Debian Science to you, or if you have prepared an unofficial Debian package, please do not hesitate to send a description of that project to the Debian Science mailing list

Links to other tasks

Debian Science Workflow packages

Official Debian packages with high relevance

capsule-nextflow
packaging and deployment tool for Java applications
Versions of package capsule-nextflow
ReleaseVersionArchitectures
trixie1.1.1+dfsg-1all
bookworm1.1.1+dfsg-1all
sid1.1.1+dfsg-1all
Popcon: 2 users (3 upd.)*
Versions and Archs
License: DFSG free
Git

A capsule is a single executable JAR that contains everything an application needs to run either in the form of embedded files or as declarative metadata. It can contain JAR artifacts, dependencies and resources, native libraries, the required Java Runtime Environment version, the Java Virtual Machine flags required to run the application well, Java or native agents and more. In short, a capsule is a self-contained JAR that knows everything there is to know about how to run the application the way it is meant to run.

One way of thinking about a capsule is as a fat JAR on steroids (that also allows native libraries and never interferes with your dependencies) and a declarative startup script rolled into one; another, is to see it is as the deploy-time counterpart to your build tool. Just as a build tool manages your build, Capsule manages the launching of your application.

This package contains a fork of the original capsule project. This fork is suited as a dependency of nextflow.

coop-computing-tools
cooperative computing tools
Versions of package coop-computing-tools
ReleaseVersionArchitectures
stretch4.0-1.1amd64,arm64,armel,armhf,i386,mips,mips64el,mipsel,ppc64el,s390x
sid9.9-4.1amd64,arm64,armel,armhf,i386,mips64el,ppc64el,riscv64,s390x
trixie9.9-4.1amd64,arm64,armel,armhf,i386,mips64el,ppc64el,riscv64,s390x
bookworm9.9-2amd64,arm64,armel,armhf,i386,mips64el,mipsel,ppc64el,s390x
bullseye7.1.2-5amd64,arm64,armel,armhf,i386,mips64el,mipsel,ppc64el,s390x
buster7.0.9-2amd64,arm64,armhf,i386
Debtags of package coop-computing-tools:
uitoolkitncurses
Popcon: 1 users (1 upd.)*
Versions and Archs
License: DFSG free
Git

This is a collection of software that help users to share resources in a complex, heterogeneous, and unreliable computing environment. This includes:

  • Chirp: A personal filesystem and I/O protocol that allows unprivileged users to share space securely, efficiently, and conveniently. When combined with Parrot, Chirp allows users to create custom wide-area distributed filesystems.
  • Parrot: A transparent user-level virtual filesystem that allows any ordinary program to be attached to a remote storage device such as an FTP server or a Chirp server.
  • Makeflow: A workflow system for parallel and distributed computing that uses a language very similar to Make.
  • Work Queue: A system and API for building master-worker style programs that scale up to thousands of processors.
  • All Pairs: A computational abstraction for running very large Cartesian products.
  • Wavefront: A computational abstraction for running very large dynamic programming problems.
  • The Fault Tolerant Shell: A high-level programming language that allows users to combine the ease of shell scripting, the power of distributed programming, and the precision of compiled languages. Basically, parallel programming and exception handling for scripts.
cwltool
Common Workflow Language reference implementation
Versions of package cwltool
ReleaseVersionArchitectures
bullseye3.0.20210124104916-3+deb11u1all
bookworm3.1.20230209161050-1all
trixie3.1.20241024121129-1all
sid3.1.20241024121129-1all
stretch1.0.20170114120503-1all
buster1.0.20181217162649+dfsg-10all
upstream3.1.20241112140730
Popcon: 34 users (20 upd.)*
Newer upstream!
License: DFSG free
Git

This is the reference implementation of the Common Workflow Language standards.

The CWL open standards are for describing analysis workflows and tools in a way that makes them portable and scalable across a variety of software and hardware environments, from workstations to cluster, cloud, and high performance computing (HPC) environments. CWL is designed to meet the needs of data-intensive science, such as Bioinformatics, Medical Imaging, Astronomy, Physics, and Chemistry.

The CWL reference implementation (cwltool) is intended to be feature complete and to provide comprehensive validation of CWL files as well as provide other tools related to working with CWL descriptions.

Please cite: Michael R. Crusoe, Sanne Abeln, Alexandru Iosup, Peter Amstutz, John Chilton, Nebojša Tijanić, Hervé Ménager, Stian Soiland-Reye, Bogdan Gavrilović, Carole Goble and The CWL Community: Methods included: standardizing computational reuse and portability with the Common Workflow Language. Communications of the ACM 65(6):54-63 (2022)
Registry entries: SciCrunch  Bioconda 
gearman
분산 작업 대기열
Versions of package gearman
ReleaseVersionArchitectures
buster-backports1.1.19.1+ds-2~bpo10+1all
sid1.1.20+ds-1.2all
bookworm1.1.20+ds-1all
jessie1.0.6-5all
trixie1.1.20+ds-1.2all
stretch1.0.6-9all
buster1.1.18+ds-3all
bullseye1.1.19.1+ds-2all
upstream1.1.21
Debtags of package gearman:
rolemetapackage
Popcon: 0 users (0 upd.)*
Newer upstream!
License: DFSG free
Git

Gearman은 다른 컴퓨터로 작업을 내보내고, 작업에 더 적합한 컴퓨터로 함수 호출을 보내고, 병렬로 작업을 수행하고, 많은 함수 호출을 로드 밸런싱하거나, 또는 언어 사이에 함수를 호출하기 위한 시스템입니다.

이 패키지는 클라이언트와 서버 양쪽에 의존하는 빈 패키지입니다.

gearman-tools
Tools for the Gearman distributed job queue
Versions of package gearman-tools
ReleaseVersionArchitectures
stretch1.0.6-9amd64,arm64,armel,armhf,i386,mips,mips64el,mipsel,ppc64el,s390x
trixie1.1.20+ds-1.2amd64,arm64,armel,armhf,i386,mips64el,ppc64el,riscv64,s390x
sid1.1.20+ds-1.2amd64,arm64,armel,armhf,i386,mips64el,ppc64el,riscv64,s390x
bullseye1.1.19.1+ds-2amd64,arm64,armel,armhf,i386,mips64el,mipsel,ppc64el,s390x
buster-backports1.1.19.1+ds-2~bpo10+1amd64,arm64,armel,armhf,i386,mips,mips64el,mipsel,ppc64el,s390x
buster1.1.18+ds-3amd64,arm64,armhf,i386
jessie1.0.6-5amd64,armel,armhf,i386
bookworm1.1.20+ds-1amd64,arm64,armel,armhf,i386,mips64el,mipsel,ppc64el,s390x
upstream1.1.21
Popcon: 10 users (9 upd.)*
Newer upstream!
License: DFSG free
Git

Gearman is a system to farm out work to other machines, dispatching function calls to machines that are better suited to do work, to do work in parallel, to load balance lots of function calls, or to call functions between languages.

This package contains some command line tools for manipulating gearman jobs.

make
컴파일 지시를 위한 유틸리티
Versions of package make
ReleaseVersionArchitectures
experimental4.4.1-1~exp1amd64,arm64,armel,armhf,i386,mips64el,ppc64el,riscv64,s390x
stretch4.1-9.1amd64,arm64,armel,armhf,i386,mips,mips64el,mipsel,ppc64el,s390x
buster4.2.1-1.2amd64,arm64,armhf,i386
bullseye4.3-4.1amd64,arm64,armel,armhf,i386,mips64el,mipsel,ppc64el,s390x
bookworm4.3-4.1amd64,arm64,armel,armhf,i386,mips64el,mipsel,ppc64el,s390x
trixie4.3-4.1amd64,arm64,armel,armhf,i386,mips64el,ppc64el,riscv64,s390x
sid4.3-4.1amd64,arm64,armel,armhf,i386,mips64el,ppc64el,riscv64,s390x
jessie4.0-8.1amd64,armel,armhf,i386
upstream4.4.1
Debtags of package make:
develbuildtools
interfacecommandline
roleprogram
scopeutility
suitegnu
works-withsoftware:source
Popcon: 35026 users (7854 upd.)*
Newer upstream!
License: DFSG free
Git

GNU Make는 프로그램의 소스 파일에서 실행 및 다른 타켓 파일 생성을 제어하는 유틸리티입니다. Make는 생성 및 재생성이 필요한 거대한 프로그램의 일부를 자 동으로 결정하며, 그리고 생성 및 재생성에 필요한 명령어를 만듭니다. Make는 해당 입력이 최신이 될 때마다 입력 파일에 기반하여 타켓 (파일)이 자동으로 업 데이트 될 수 있도록 임의 태스크를 구성하는데 사용될 수 있습니다 -- Make는 컴퓨터 프로그램 구축에만 한정되지 않습니다. 실제로 Make는 범용의 의존성 해 결자 역활을 합니다.

The package is enhanced by the following packages: make-doc
Screenshots of package make
pegasus-wms
Scientific workflow management system for HTCondor
Versions of package pegasus-wms
ReleaseVersionArchitectures
stretch4.4.0+dfsg-6amd64,arm64,armel,armhf,i386,mips,mips64el,mipsel,ppc64el,s390x
buster4.4.0+dfsg-8amd64,arm64,armhf,i386
jessie4.4.0+dfsg-4amd64,armel,armhf,i386
Popcon: 0 users (0 upd.)*
Versions and Archs
License: DFSG free

The Pegasus project encompasses a set of technologies the help workflow-based applications execute in a number of different environments including desktops, campus clusters, grids, and now clouds. Scientific workflows allow users to easily express multi-step computations, for example retrieve data from a database, reformat the data, and run an analysis. Once an application is formalized as a workflow the Pegasus Workflow Management Service can map it onto available compute resources and execute the steps in appropriate order.

Screenshots of package pegasus-wms
python3-nipype
Neuroimaging data analysis pipelines in Python3
Versions of package python3-nipype
ReleaseVersionArchitectures
bookworm1.8.5-3all
bullseye1.6.0-2all
sid1.9.0-1all
upstream1.9.1
Popcon: 12 users (14 upd.)*
Newer upstream!
License: DFSG free
Git

Nipype interfaces Python to other neuroimaging packages and creates an API for specifying a full analysis pipeline in Python. Currently, it has interfaces for SPM, FSL, AFNI, Freesurfer, but could be extended for other packages (such as lipsia).

Please cite: SS Ghosh, C Burns, D Clark, K Gorgolewski, YO Halchenko, C Madison, R Tungaraza and KJ Millman: Nipype: Opensource platform for unified and replicable interaction with existing neuroimaging tools (eprint) 16th Annual Meeting of the Organization for Human Brain Mapping :106 (2010)
python3-wdlparse
Workflow Description Language (WDL) parser for Python
Versions of package python3-wdlparse
ReleaseVersionArchitectures
bookworm0.1.0-3all
bullseye0.1.0-2all
sid0.1.0-3all
trixie0.1.0-3all
Popcon: 1 users (2 upd.)*
Versions and Archs
License: DFSG free
Git

A Python package that provides the generated Hermes and Antlr4 WDL parsers for Python.

snakemake
pythonic workflow management system
Versions of package snakemake
ReleaseVersionArchitectures
stretch3.10.0-1all
trixie7.32.4-6all
sid7.32.4-6all
bullseye5.24.1-2all
buster5.4.0-1all
bookworm7.21.0-1all
upstream8.25.3
Popcon: 34 users (5 upd.)*
Newer upstream!
License: DFSG free
Git

Build systems like GNU Make are frequently used to create complicated workflows, e.g. in bioinformatics. This project aims to reduce the complexity of creating workflows by providing a clean and modern domain specific language (DSL) in Python style, together with a fast and comfortable execution environment.

Please cite: Johannes Köster and Sven Rahmann: Snakemake-a scalable bioinformatics workflow engine. Bioinformatics (2012)
Registry entries: Bio.tools  SciCrunch  Bioconda 
toil
cross-platform workflow engine
Versions of package toil
ReleaseVersionArchitectures
bullseye5.2.0-5all
sid6.1.0-4all
bookworm5.9.2-2+deb12u1all
buster3.18.0-2all
upstream7.0.0
Popcon: 2 users (2 upd.)*
Newer upstream!
License: DFSG free
Git

Toil is a scalable, efficient, cross-platform and easy-to-use workflow engine in pure Python. It works with several well established load balancers like Slurm or the Sun Grid Engine. Toil is also compatible with the Common Workflow Language (CWL) via the "toil-cwl-runner" interface, which this package make available via the Debian alternativess system under the alias "cwl-runner".

Please cite: John Vivian, Arjun Arkal Rao, Frank Austin Nothaft, Christopher Ketchum, Joel Armstrong, Adam Novak, Jacob Pfeil, Jake Narkizian Alden D. Deran, Audrey Musselman-Brown, Hannes Schmidt, Peter Amstutz, Brian Craft, Mary Goldman, Kate Rosenbloom, Melissa Cline, Brian O'Connor, Megan Hanna, Chet Birger, W. James Kent David A. Patterson, Anthony D. Joseph, Jingchun Zhu, Sasha Zaranek, Gad Getz, David Haussler and Benedict Paten: Toil enables reproducible, open source, big biomedical data analyses. Nature Biotechnology 35(4):314–316 (2017)
Registry entries: Bioconda 

Official Debian packages with lower relevance

cwlformat
code formatter for Common Workflow Language
Versions of package cwlformat
ReleaseVersionArchitectures
sid2022.02.18-3all
bookworm2022.02.18-2all
trixie2022.02.18-3all
bullseye2021.01.05-1all
Popcon: 0 users (0 upd.)*
Versions and Archs
License: DFSG free
Git

CWL Format is a specification and a reference implementation for a very opinionated CWL code formatter.

It outputs Common Workflow Language(CWL) in a standardized YAML format. It has no settings or options because you have better things to do with your time. And because CWL Format is always correct.

libbenchmark-tools
Microbenchmark support library, tools and documentation
Versions of package libbenchmark-tools
ReleaseVersionArchitectures
bullseye1.5.2-2all
stretch-backports1.3.0-1~bpo9+1all
buster1.4.1-3all
bookworm1.7.1-1all
trixie1.8.3-3all
sid1.8.3-3all
experimental1.8.4-1~exp1all
upstream1.9.0
Popcon: 0 users (0 upd.)*
Newer upstream!
License: DFSG free
Git

Library to support the benchmarking of functions, similar to unit-tests.

This package contains tools and documentation.

*Popularitycontest results: number of people who use this package regularly (number of people who upgraded this package recently) out of 248069