Scientific workflow

A scientific workflow is an automated, documented sequence of computational steps, data cleaning, transformation, analysis, modeling, that takes research data from raw input to final result, typically expressed as a directed acyclic graph of tasks with defined dependencies rather than a single monolithic script. Workflow management tools such as Snakemake, Nextflow and Galaxy schedule, parallelize and re-run only the steps affected by a change.

This differs from general-purpose workflow orchestration used in data engineering mainly by audience and emphasis: scientific workflow tools are built around the reproducibility and provenance needs of research, recording exactly which inputs, parameters and software versions produced each output, so a result can be traced and regenerated later, sometimes years afterward.

Scientific workflows are what let compute-heavy analyses like genomics analysis run reliably at scale on high-performance computing clusters, and they are increasingly what a funder or journal expects to see accompanying a publication's results, supporting reproducibility far better than a folder of manually run scripts. Common pitfalls are hidden dependencies on a specific machine's software environment that break the pipeline elsewhere, and insufficient research data management around the intermediate files a workflow produces, leaving no record of what a given output actually came from.