Skip to content

Solutions: cluster execution

Run the mode B commands from snakemake-workshop/work-cluster-b with .venv-slurm active.

C1. Count jobs and CPUs

At most four run_app jobs fit: 4 / 1 = 4 by CPU count and 1024 / 256 = 4 by the declared memory budget. Fewer can run when fewer are ready. collect depends on all ten outputs, so it starts only after they all exist and are current.

For per-job submission, jobs: 10 permits up to ten outstanding workflow jobs rather than four. Each still requests one CPU through threads: 1. Pending jobs count towards this limit; the cluster may run fewer jobs at a time. Changing the limit neither enlarges an allocation nor overrides site quotas. For a threaded application, its rule and the application command must agree on thread use; requesting more threads does not parallelize serial Python automatically.

C2. Override one rule's time limit

Extend the existing set-resources mapping in profiles/slurm/config.yaml, preserving both memory overrides. Do not add a second set-resources key. The complete mapping should now read:

set-resources:
  run_app:
    mem_mb: 1024
  collect:
    mem_mb: 1024
    runtime: 10

Then run:

snakemake --profile profiles/slurm --dry-run --forceall

The preview includes runtime=10 for collect; the ten run_app jobs keep runtime=5. set-resources overrides a rule's explicit request. Merely changing default-resources.runtime would leave both rules' explicit five-minute values in place. Both rules retain mem_mb=1024. To make the temporary command-line override self-contained, include both memory settings too:

snakemake --profile profiles/slurm --dry-run --forceall \
    --set-resources run_app:mem_mb=1024 collect:mem_mb=1024 collect:runtime=10

This is a planned scheduler request, not evidence that the job will finish within ten minutes.

C3. Diagnose a failure

jobs controls concurrency, not a job's walltime. touch summary.tsv creates or updates a filename without producing the ten-row summary, hiding a missing result instead of repairing it.

Read the failing job's Slurm log, Snakemake's error, and the site's accounting record for its job ID. Establish whether the limit belongs to the whole batch allocation (mode A) or an individual rule job (mode B). Check for an unexpectedly large input, slow storage, a program problem or an underestimated runtime before raising the appropriate limit within site policy.

After correcting the cause, first preview. In mode B:

snakemake --profile profiles/slurm --dry-run --rerun-incomplete

Only on the configured cluster, resume with:

snakemake --profile profiles/slurm --rerun-incomplete

Missing/unfinished work is rerun while current completed work can be reused; read the dry-run reasons before resuming. --rerun-incomplete authorizes rebuilding outputs recorded as incomplete; it does not make a failed result valid. For mode A, adjust the sbatch allocation request and resubmit the script, adding --rerun-incomplete to its Snakemake command if incomplete-output metadata requires it.

Finish with the workshop checklist.