Skip to content

Rules, targets and wildcards

We now connect the ten independent jobs to a second computation: collecting their results into a table. You will trace dependencies backwards from a target, explain expand and wildcards, and predict which jobs run after a change.

1. Start a separate workflow

Begin in snakemake-workshop with .venv active and the support archive already extracted. If you are in work-intro, first run cd ...

mkdir work-rules
cp SnakeMake.intro/app.py work-rules/
cp workshop-support/rules/Snakefile workshop-support/rules/collect_results.py work-rules/
cd work-rules

Our intended result is summary.tsv, a tab-separated table with a header and ten data rows. Its columns are file and value. It will list results/output_0.txt through results/output_9.txt in that explicit order, regardless of job completion order.

2. Read the dependency graph

The workflow has three rules:

app.py ──> results/output_0.txt ──┐
       ──> results/output_1.txt ──┤
       ...                       ├──> summary.tsv ──> all
       ──> results/output_9.txt ──┘        ↑
                                   collect_results.py

Arrows indicate file dependencies, not the order in which rules appear in the file. all names the requested final target; it does not produce a file named all. Each result can be created independently. The collector must wait for all ten.

The complete Snakefile is:

RESULTS = expand("results/output_{i}.txt", i=range(10))

rule all:
    input:
        "summary.tsv"

rule run_app:
    input:
        script="app.py"
    output:
        "results/output_{i}.txt"
    threads: 1
    resources:
        mem_mb=256,
        runtime=5
    shell:
        "python {input.script:q} > {output:q}"

rule collect:
    input:
        script="collect_results.py",
        values=RESULTS
    output:
        "summary.tsv"
    threads: 1
    resources:
        mem_mb=256,
        runtime=5
    shell:
        "python {input.script:q} {output:q} {input.values:q}"

input.values is the list of ten result files; input.script is the collector program. Keeping them separate avoids accidentally passing the script itself as a data file. Snakemake creates parent directories for declared outputs, so we do not need an extra rule to create results/. These two executable rules each request one thread, 256 MB and five minutes for the later cluster example; those are illustrative requests, not measured requirements.

3. Two uses of braces

Compare these expressions:

Expression Meaning in this workflow
expand("results/output_{i}.txt", i=range(10)) Python-side construction of ten concrete filenames before jobs are run
"results/output_{i}.txt" under output A rule pattern with a wildcard whose value is inferred from a requested filename
{input.script:q}, {output:q} in shell Substitution of a job's actual filenames into its shell command

For a request for results/output_7.txt, the output pattern gives i = "7". This is a filename-derived string, not a Python loop variable. expand does not execute jobs: it supplies the filenames that cause matching jobs to be required. A wildcard rule describes possible outputs beyond the default target list; requesting results/output_12.txt explicitly would also match this pattern.

A directed acyclic graph (DAG) describes the jobs and their dependencies. “Directed” means that outputs feed downstream inputs. “Acyclic” means no job depends, directly or indirectly, on its own output. We reason backwards to find what is needed, then execute upstream jobs before their consumers. The official basic tutorial demonstrates this connection between filenames and dependencies.

4. Inspect the collector and run

cat collect_results.py
snakemake --cores 2 --scheduler greedy --dry-run
snakemake --cores 2 --scheduler greedy
cat summary.tsv

Before the first run, expect ten run_app jobs, one collect job and all. The collector reads the ordered list passed by input.values. It requires exactly one result: N line per input, with an integer from 0 to 99, and raises an error for invalid content. It checks all rows before writing the table. This checks the format of this toy example; it does not establish statistical properties of the random numbers.

The output begins with file and value separated by a tab. There should be eleven lines total. Check both the format and row order:

python - <<'PYCODE'
import csv
from pathlib import Path
with Path("summary.tsv").open() as stream:
    table = csv.DictReader(stream, delimiter="\t")
    assert table.fieldnames == ["file", "value"]
    rows = list(table)
assert [row["file"] for row in rows] == [f"results/output_{i}.txt" for i in range(10)]
assert all(0 <= int(row["value"]) <= 99 for row in rows)
print("Ten ordered results: OK")
PYCODE

5. Predict reruns from dependencies

First repeat the default invocation:

snakemake --cores 2 --scheduler greedy

An unchanged run should have nothing to do. Now remove a generated upstream file:

rm results/output_4.txt
snakemake --cores 2 --scheduler greedy --dry-run

The default request can still report nothing to do: summary.tsv exists, and the missing intermediate is not itself a requested target. Snakemake does not automatically restore every historical intermediate. To retain both that intermediate and the table, request both explicitly:

snakemake --cores 2 --scheduler greedy --dry-run results/output_4.txt summary.tsv
snakemake --cores 2 --scheduler greedy results/output_4.txt summary.tsv

Now one generator job restores the missing file. The downstream collector also runs because its requested output depends on an input being regenerated. The other nine generator outputs are reused. Inspect summary.tsv again; its row count and order stay the same even if the new value happens to equal the old value.

What if only collect_results.py changes? It is an input to collect, but not to run_app. A relevant change therefore requires recollection, not ten new random numbers. Conversely, changing app.py affects every generator job and then the collector. Snakemake's default rerun checks also consider recorded code, parameters and other provenance; dependency reasoning assumes that no other tracked changes have occurred. See rules and dependencies.

Proceed to the exercises, then design a scientific workflow.