Skip to content

EuroCC and SLING

Introduction to high-throughput computing with Snakemake

Luka Leskovec
University of Ljubljana, Faculty of Mathematics and Physics

A research calculation often consists of many smaller tasks: prepare inputs, run a program for several parameter choices, collect the results and produce a plot. Some tasks can run at the same time; others must wait for earlier results. This workshop teaches you to identify those dependencies and express them as a workflow that Snakemake can execute.

We begin with a Python program that generates random numbers, then work through a scientific analysis with several stages. Finally, we consider execution inside a cluster allocation and submission through a cluster scheduler.

Audience and prerequisites

This workshop is intended for beginners and intermediate users who want to automate computational work. You should be able to:

  • navigate directories and run commands in a Linux or macOS terminal;
  • read a short Python program and understand a loop;
  • distinguish a program, its input files and its output files.

No prior Snakemake knowledge is required. The scientific example provides a concrete workflow; prior knowledge of lattice physics is not required to identify its tasks and dependencies.

Learning outcomes

By the end of the workshop, you should be able to:

  1. Split a calculation into tasks with explicit inputs and outputs.
  2. Identify which tasks can run concurrently and which must wait for data.
  3. Express a small workflow using Snakemake rules and wildcards.
  4. Check a planned run and inspect its results.
  5. Explain the difference between execution within one cluster allocation and submission of jobs through a scheduler.

Workshop sequence

Part Focus
Introduction and installation Concepts, environment and installation check
First workflow Serial Python, shell loop and independent jobs
Rules and wildcards Targets, collection and reruns
Scientific workflow design Data interfaces and dependencies
Scientific implementation Full graph and one real fit; complete run as an extension
Cluster execution One allocation and the Slurm executor
Summary Transfer to your own work and questions

Python 3, venv and Git should be installed beforehand. Prepare the scientific packages before the session when possible. The instructor should prepare the full scientific run before the session; the core lesson executes one fit and plans the complete graph, while the full run is an extension. Cluster submission depends on site access; local profile inspection and dry runs complete the core lesson.

Materials

We use two repositories:

  • SnakeMake.intro: the small random-number example.
  • SnakeMake.main: the scientific workflow. The program/ directory is the exercise starting point; program.solution/ contains the reference workflow.

The installation instructions explain how to obtain both. Keep the exercise and solution directories separate so that you can develop your own workflow before comparing it with the reference.

Start with the introduction, then complete installation and the exercises.