Skip to content

Take the workflow with you

Check your understanding and choose a next step. We covered the following topics: independent delayed tasks, rules and targets, a scientific dependency graph, then two ways to run that graph with Slurm.

What you should be able to demonstrate

Skill Evidence from the workshop
Turn independent tasks into jobs Ten results/output_*.txt files from one wildcard rule
Request a result and explain its dependencies summary.tsv requires every random-number output
Distinguish planning from execution Explain a dry run, run the workflow, then observe an unchanged rerun
Repair only the affected part Remove one output in a disposable exercise copy and predict the resulting jobs
Design interfaces for an analysis Trace raw HDF5 → candidate energies → selected energies → dispersion plot
Track actual results Identify real output files rather than relying on a success marker
Choose an execution mode Explain one allocation versus per-job Slurm execution
Monitor execution Locate job IDs, logs and expected output files

Use the first workflow, rules exercises, scientific design, implementation and cluster exercises to revisit any row you cannot yet explain.

Before calling your own workflow reproducible

  1. Name the final target and write down each program's inputs, outputs and working directory.
  2. Declare the scripts, configuration and data that actually affect results as dependencies. Ensure independent jobs do not write to the same paths.
  3. Start with a small dataset. Check a clean run, an unchanged rerun, rebuilding a missing output, and a deliberate program failure in a disposable copy.
  4. Record software versions, input provenance and parameters. For randomness, decide whether exact reproducibility is required and how seeds are recorded; this workshop's original random generator is unseeded.
  5. Estimate and then measure CPU, memory and walltime needs. Keep site-specific account/partition choices in a profile. Recheck the environment on compute nodes before scaling up.
  6. Write instructions for preparing the environment, running the workflow and checking the expected outputs.

Keep the original program/ and program.solution/ trees as source material and retain your changes in the separate working copy.

A useful next exercise

Choose one small analysis you currently run manually. Draw a graph with three to five steps. For every arrow, name the file that carries the data; for every box, write the command and working directory. Implement the final target and one rule first. Ask a colleague to reproduce it from your written setup instructions before adding the remaining steps.

For your next workshop iteration, record which step failed, the exact command, software versions and the relevant log. That makes a useful bug report and helps distinguish an installation issue, a workflow dependency problem and a failure in an individual program.

Further reading

Keep your source files, profiles and validation notes together so another person can reproduce your work.