Skip to content

From a laptop to Slurm

The objective is to choose an execution mode, connect rule resources to scheduler requests, and prepare a site-specific configuration. Actual cluster execution requires access and instructions from your site; the local workshop can finish with inspection and dry runs.

Use the small work-rules workflow from rules and targets: ten run_app jobs create results/output_0.txt through results/output_9.txt, then collect creates summary.tsv. Both executable rules request threads: 1, mem_mb=256, and runtime=5. These are teaching requests, not measured requirements for the scientific fits.

Two execution modes

One allocation One cluster job per workflow job
Who submits? You submit one batch script Snakemake's Slurm executor submits jobs
Where does Snakemake run? Inside that allocation On a site-approved controller host
What limits concurrency? Allocated CPUs and local resource budgets --jobs, dependencies and cluster policy
Main tradeoff All work must fit the allocation Each job incurs scheduling overhead

For our tiny delayed tasks, one allocation is usually a more sensible demonstration. Per-job execution is useful when individual steps justify scheduler overhead or need different resources. Neither mode provides unlimited capacity: account quotas, partition limits, queue load and walltime still apply.

Before using a real cluster

Keep snakemake-workshop/, the inputs and the environment on storage visible at the same paths to the controller and compute nodes. Recreate .venv on the cluster using the Python environment described in the installation instructions; do not copy a macOS virtual environment to Linux. Reuse the installation steps and your prepared workflow files. Obtain the account, partition, permitted controller location, Python/module setup and any required reservation or QoS from your site's instructions.

Start in snakemake-workshop on the cluster. If you are in work-science/program, return with cd ../... The support archive must already be extracted here as workshop-support/.

Prepare separate working directories for modes A and B. Copy only the programs and workflow, so each mode starts without results or execution metadata from an earlier run. If these directories already exist from a previous attempt, choose new names and use them consistently below.

source .venv/bin/activate
mkdir work-cluster-a work-cluster-b
for directory in work-cluster-a work-cluster-b; do
    cp SnakeMake.intro/app.py "$directory/"
    cp workshop-support/rules/Snakefile workshop-support/rules/collect_results.py "$directory/"
done
cd work-cluster-a
python --version
snakemake --version
snakemake --cores 1 --scheduler greedy --dry-run

In this fresh directory, expect ten run_app jobs, one collect job and the all target: 12 jobs in the plan. The dry run creates no results. You are now in work-cluster-a; work-cluster-b remains untouched for mode B.

After completing a workflow, an unchanged dry run reports that nothing needs doing. Add --forceall when you want to inspect the full plan again:

snakemake --cores 1 --scheduler greedy --dry-run --forceall

A. Run locally inside one allocation

Read run-in-allocation.sh. The following command runs inside the batch script; do not run it separately in your terminal. Submit the script with sbatch below:

snakemake --executor local --scheduler greedy \
    --cores "$SLURM_CPUS_PER_TASK" --resources mem_mb=1024 --printshellcmds

Submit from the actual snakemake-workshop/work-cluster-a directory: SLURM_SUBMIT_DIR records where you invoked sbatch. Using sbatch --chdir alone does not change that variable.

The script changes to SLURM_SUBMIT_DIR, activates ../.venv, limits common numerical-library thread pools to one thread, and uses strict shell error handling. It receives the CPU count from Slurm, rather than using every CPU visible on the node. The local 1024 MB scheduling budget admits at most four jobs requesting 256 MB each; it is an estimate for scheduling, not a measurement or an operating-system memory limit. We request 2 GB from Slurm to leave space for Snakemake and other overhead.

Site-specific submission template

Replace REPLACE_WITH_ACCOUNT and REPLACE_WITH_PARTITION using your site's instructions. Confirm that the example CPU, memory and time request is permitted. This command submits a real job; only run it during an authorized cluster exercise. Do not run the script directly on a login node.

sbatch --account=REPLACE_WITH_ACCOUNT --partition=REPLACE_WITH_PARTITION \
    --nodes=1 --ntasks=1 --cpus-per-task=4 --mem=2G --time=00:10:00 \
    ../workshop-support/cluster/run-in-allocation.sh

The request reserves one task on one node with four CPUs. sbatch returning a job ID means submission succeeded; it does not mean the workflow finished. Its default combined output file is slurm-JOBID.out. Slurm does not transfer your input files automatically. See the official sbatch reference.

Follow the job using the ID printed by sbatch (replace JOBID):

squeue -u "$USER"
sacct -j JOBID --format=JobID,State,Elapsed,ExitCode
tail -f slurm-JOBID.out

squeue lists your pending (PD) and running (R) jobs; the job disappears from this list when it ends. sacct then reports its final state. In the main row (without a .batch or .extern suffix), COMPLETED with exit code 0:0 means that the batch script, and therefore Snakemake, finished successfully. FAILED, TIMEOUT or OUT_OF_MEMORY means you should read the log. Stop tail -f with Ctrl+C; this does not affect the job.

When sacct reports COMPLETED, inspect the log and summary.tsv (a header plus ten data rows). An unchanged local dry run should require no computation. Keep these results in work-cluster-a; mode B uses its own fresh directory.

B. Let the Slurm executor submit jobs

Return to snakemake-workshop (run cd .. if you are in work-cluster-a). Use the separate Slurm environment prepared in the optional installation step. If you have not created it yet, complete that step on the cluster; it also finishes in snakemake-workshop. Both the environment and Python must be available at the same paths on compute nodes.

Enter the fresh directory prepared for mode B, activate the Slurm environment and create the profile:

cd work-cluster-b
source ../.venv-slurm/bin/activate
export OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1
export MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1
mkdir -p profiles/slurm
cp ../workshop-support/cluster/profile/config.yaml profiles/slurm/config.yaml

Edit profiles/slurm/config.yaml, replacing both site placeholders:

executor: slurm
scheduler: greedy
jobs: 4
printshellcmds: true
slurm-keep-successful-logs: true
default-resources:
  slurm_account: REPLACE_WITH_ACCOUNT
  slurm_partition: REPLACE_WITH_PARTITION
  mem_mb: 256
  runtime: 5
set-resources:
  run_app:
    mem_mb: 1024
  collect:
    mem_mb: 1024

A profile is a YAML file containing Snakemake command-line settings. Keep site choices in the profile so the workflow remains portable. Here, runtime is walltime in minutes per job, mem_mb is memory per job, and each rule's threads: 1 supplies its CPU request. Explicit rule resources take precedence over defaults; set-resources overrides those rule requests. The profile therefore requests 1024 MB for each executable job in mode B, while mode A retains its 256 MB local scheduling estimates. Each separate cluster job also starts Snakemake and its job-step machinery, so resource planning must include that overhead. The 1024 MB value worked in our cluster test; it is a site choice, not a universal minimum. jobs: 4 limits outstanding workflow jobs; it does not allocate four CPUs or guarantee four simultaneously running jobs. Resource mappings are documented in the Slurm executor reference; profile syntax is in the Snakemake command-line documentation.

The exported thread limits keep common numerical libraries to one thread in the controller and its submitted jobs.

First inspect the plan, including resources. The fresh work-cluster-b directory should also plan ten run_app jobs, one collect job and all, regardless of whether mode A has finished:

snakemake --profile profiles/slurm --dry-run --forceall

This submits no jobs and does not prove that your account, partition, filesystem or compute-node environment works. After the instructor/site has checked those settings, the following command performs real submission from the permitted controller host:

snakemake --profile profiles/slurm

Run this controller outside an existing Slurm allocation, as in our successful test: the plugin warns about inherited Slurm settings when started inside a batch job. Keep the controller running until the workflow completes. Follow your site's policy for long-running controllers; do not assume they may run on a login node. The plugin records Slurm job logs beneath .snakemake/slurm_logs/; this profile retains successful logs too. Use the reported job IDs with site monitoring tools. A failed job requires reading both Snakemake's error and the Slurm job log before retrying.

For the scientific workflow or mode A, reactivate ../.venv/bin/activate. The toy workflow used here does not import NumPy; running the scientific fits with a new dependency set would require fresh validation.

The results for this mode are in work-cluster-b. Check summary.tsv for a header and ten data rows, then run snakemake --profile profiles/slurm --dry-run without --forceall: it should report nothing to do.

Continue with the exercises.