Take the workflow with you
Check your understanding and choose a next step. We covered the following topics: independent delayed tasks, rules and targets, a scientific dependency graph, then two ways to run that graph with Slurm.
What you should be able to demonstrate
| Skill | Evidence from the workshop |
|---|---|
| Turn independent tasks into jobs | Ten results/output_*.txt files from one wildcard rule |
| Request a result and explain its dependencies | summary.tsv requires every random-number output |
| Distinguish planning from execution | Explain a dry run, run the workflow, then observe an unchanged rerun |
| Repair only the affected part | Remove one output in a disposable exercise copy and predict the resulting jobs |
| Design interfaces for an analysis | Trace raw HDF5 → candidate energies → selected energies → dispersion plot |
| Track actual results | Identify real output files rather than relying on a success marker |
| Choose an execution mode | Explain one allocation versus per-job Slurm execution |
| Monitor execution | Locate job IDs, logs and expected output files |
Use the first workflow, rules exercises, scientific design, implementation and cluster exercises to revisit any row you cannot yet explain.
Before calling your own workflow reproducible
- Name the final target and write down each program's inputs, outputs and working directory.
- Declare the scripts, configuration and data that actually affect results as dependencies. Ensure independent jobs do not write to the same paths.
- Start with a small dataset. Check a clean run, an unchanged rerun, rebuilding a missing output, and a deliberate program failure in a disposable copy.
- Record software versions, input provenance and parameters. For randomness, decide whether exact reproducibility is required and how seeds are recorded; this workshop's original random generator is unseeded.
- Estimate and then measure CPU, memory and walltime needs. Keep site-specific account/partition choices in a profile. Recheck the environment on compute nodes before scaling up.
- Write instructions for preparing the environment, running the workflow and checking the expected outputs.
Keep the original program/ and program.solution/ trees as source material and retain your changes in the separate working copy.
A useful next exercise
Choose one small analysis you currently run manually. Draw a graph with three to five steps. For every arrow, name the file that carries the data; for every box, write the command and working directory. Implement the final target and one rule first. Ask a colleague to reproduce it from your written setup instructions before adding the remaining steps.
For your next workshop iteration, record which step failed, the exact command, software versions and the relevant log. That makes a useful bug report and helps distinguish an installation issue, a workflow dependency problem and a failure in an individual program.
Further reading
- Snakemake command-line reference: dry runs, rerun controls, profiles and resources.
- Official Slurm executor documentation: current plugin settings; compare them with the pinned version before changing your environment.
- Slurm batch submission reference: job requests and batch-script behavior.
- NERSC's Snakemake guidance: an example of running workflows on the NERSC cluster. Its accounts, queues and environment instructions belong to NERSC, not automatically to your cluster.
Keep your source files, profiles and validation notes together so another person can reproduce your work.