
Introduction to high-throughput computing with Snakemake
Luka Leskovec
University of Ljubljana, Faculty of Mathematics and Physics
A research calculation often consists of many smaller tasks: prepare inputs, run a program for several parameter choices, collect the results and produce a plot. Some tasks can run at the same time; others must wait for earlier results. This workshop teaches you to identify those dependencies and express them as a workflow that Snakemake can execute.
We begin with a Python program that generates random numbers, then work through a scientific analysis with several stages. Finally, we consider execution inside a cluster allocation and submission through a cluster scheduler.
Audience and prerequisites
This workshop is intended for beginners and intermediate users who want to automate computational work. You should be able to:
- navigate directories and run commands in a Linux or macOS terminal;
- read a short Python program and understand a loop;
- distinguish a program, its input files and its output files.
No prior Snakemake knowledge is required. The scientific example provides a concrete workflow; prior knowledge of lattice physics is not required to identify its tasks and dependencies.
Learning outcomes
By the end of the workshop, you should be able to:
- Split a calculation into tasks with explicit inputs and outputs.
- Identify which tasks can run concurrently and which must wait for data.
- Express a small workflow using Snakemake rules and wildcards.
- Check a planned run and inspect its results.
- Explain the difference between execution within one cluster allocation and submission of jobs through a scheduler.
Workshop sequence
| Part | Focus |
|---|---|
| Introduction and installation | Concepts, environment and installation check |
| First workflow | Serial Python, shell loop and independent jobs |
| Rules and wildcards | Targets, collection and reruns |
| Scientific workflow design | Data interfaces and dependencies |
| Scientific implementation | Full graph and one real fit; complete run as an extension |
| Cluster execution | One allocation and the Slurm executor |
| Summary | Transfer to your own work and questions |
Python 3, venv and Git should be installed beforehand. Prepare the scientific packages before the session when possible. The instructor should prepare the full scientific run before the session; the core lesson executes one fit and plans the complete graph, while the full run is an extension. Cluster submission depends on site access; local profile inspection and dry runs complete the core lesson.
Materials
We use two repositories:
- SnakeMake.intro: the small random-number example.
- SnakeMake.main: the scientific workflow. The
program/directory is the exercise starting point;program.solution/contains the reference workflow.
The installation instructions explain how to obtain both. Keep the exercise and solution directories separate so that you can develop your own workflow before comparing it with the reference.
Start with the introduction, then complete installation and the exercises.