Skip to content

First workflow: ten independent jobs

By the end of this part you will run the same small calculation in a Python loop, a shell loop and a Snakemake workflow. You will distinguish one rule (a recipe) from one job (that recipe applied to a particular output), inspect the execution plan and recover a missing result.

Prepare your workspace

Complete installation first. Use the same terminal with .venv activated. Begin in the snakemake-workshop directory, containing .venv, SnakeMake.intro and SnakeMake.main. If you finished the installation in installation-check, use cd .. first.

Download workshop-support.zip, save it in snakemake-workshop, then extract it there:

python -m zipfile -e workshop-support.zip .
ls workshop-support
mkdir work-intro
cp SnakeMake.intro/app.py SnakeMake.intro/app_v0.py work-intro/
cp workshop-support/first-workflow/Snakefile work-intro/
cd work-intro

The extracted directory contains first-workflow, rules, science and cluster. We work in work-intro; the source repositories remain available for comparison. If this directory already exists from an earlier attempt, use a new directory name consistently for this part.

1. A serial Python program

Inspect app_v0.py:

cat app_v0.py
python app_v0.py

Its loop waits three seconds and prints an integer from 0 to 99, ten times. Expect ten lines with this format, with values chosen by the program:

result: 42

Run the serial program once. It requests ten sequential three-second sleeps, so you wait about 30 seconds plus overhead. The random generator is not seeded: the numbers are not a reference answer and may repeat. The delay makes scheduling visible; it does not model CPU-intensive work.

2. Move repetition into the shell

app.py contains one iteration of that calculation. The following shell loop starts it ten times sequentially and puts each output in a separate file:

mkdir shell-results
for i in {0..9}; do
    python app.py > "shell-results/output_${i}.txt"
done
ls shell-results
cat shell-results/output_0.txt

The shell waits for each python process before continuing. Separate output files alone do not make the execution parallel. Each file should contain one result: N line. We use indices 0–9, matching Python's range(10) and the original Snakefile. The original script/script.sh uses 1–10; this is an explicit index alignment in our teaching copy.

3. Describe the desired files

Open Snakefile. The complete teaching version is:

RESULTS = expand("output_{i}.txt", i=range(10))

rule all:
    input:
        RESULTS

rule run_app:
    input:
        script="app.py"
    output:
        "output_{i}.txt"
    threads: 1
    resources:
        mem_mb=256,
        runtime=5
    shell:
        "python {input.script:q} > {output:q}"

RESULTS is the concrete list output_0.txt, …, output_9.txt. The first rule, all, requests these files; it has no command to run. The name all is a convention. Its first position makes it the default target in this workflow.

run_app is the recipe for one output. For output_3.txt, the wildcard {i} takes the value 3. Ten requested filenames create ten independent jobs from this one rule. The app uses no input data file, but its source code is still a dependency: we declare app.py under input. The named input script makes the command readable. :q quotes file paths when needed by the shell.

Each job declares one thread. The memory and runtime declarations are teaching requests for the later cluster lesson: 256 MB and five minutes per job. They are not measured requirements, and they do not impose a local time limit. See the official rules reference for rule fields, wildcard matching and resource semantics.

4. Inspect, then execute

Still in work-intro:

snakemake --cores 2 --scheduler greedy --dry-run
snakemake --cores 2 --scheduler greedy
ls output_*.txt
cat output_0.txt

Before the first execution, the dry run should list ten run_app jobs plus the target rule all. It does not run Python or create the result files. The real run permits up to two one-thread jobs at a time. The shell-loop files live in shell-results/, so they do not satisfy the new targets in this directory.

Success means that output_0.txt through output_9.txt exist, each containing one integer in the range 0–99 after result:. Job completion order and values can differ between runs. Do not infer a measured speedup or CPU scaling from these sleeps.

5. Reuse and repair results

Run the same command again:

snakemake --cores 2 --scheduler greedy

With unchanged inputs, commands and outputs, Snakemake should report that nothing needs doing. It does not generate fresh random numbers simply because you invoke it again.

Now remove one generated file in work-intro and inspect the plan:

rm output_3.txt
snakemake --cores 2 --scheduler greedy --dry-run
snakemake --cores 2 --scheduler greedy
cat output_3.txt

The missing result requires one run_app job. The other nine files are reused. The regenerated random value may differ; it may also coincide by chance. For a later return to this lesson, enter snakemake-workshop, activate .venv, then cd work-intro.

Continue with the exercises, then rules and wildcards.