Design a scientific workflow
By the end, you should be able to turn an unfamiliar analysis into a dependency graph and name the files at each interface. We will follow design steps using SnakeMake.main.
We will use the supplied analysis programs as tools with file inputs and outputs, focusing on their dependencies and execution with Snakemake.
A few terms used in this example:
- INI: a text configuration file containing named settings, such as the model and fit interval.
- HDF5: a file format that stores named numerical datasets; here it contains the input correlation data.
- Fit interval: the range of time indices used when fitting the model to the data.
- Replica: one indexed member of the supplied resampled data.
1. Start with the final result
The supplied analysis fits correlation-function data, selects energy estimates and produces a dispersion plot. Our workflow target is dispersion/dispersion_pion.pdf, relative to the analysis's program directory.
The repository separates program/, containing the analysis programs, from program.solution/, containing the historical workflow solution and fit configurations. In the next part we will copy the programs into work-science/ and add a new workflow. Keep both original directories intact.
2. Inspect the interfaces
In your active workshop environment, start in snakemake-workshop. These commands only read files:
ls SnakeMake.main/program
cat SnakeMake.main/program.solution/correlators/inis/input_1.ini
cat SnakeMake.main/program/dispersion/file.list
The INI file specifies an HDF5 input, a model, a fit interval and an output directory. Its relative paths are interpreted from program/correlators. file.list names four energy files and associates each with a lattice extent and a squared integer momentum label. Its relative paths are interpreted from program/dispersion. Working directories are part of the program interface, even when they do not appear as command-line arguments.
| Stage | Reads | Writes |
|---|---|---|
| Fit correlators | pion_lambda.h5, one INI configuration, fit_correlator.py |
Four .jack energy files per distinct configuration |
| Select energies | All candidate energy files, combine_fits.py |
Four selected/transformed .jack files and energies_chosen.pdf |
| Fit dispersion | Four selected files, file.list, fit_dispersion.py, dispersion.py |
dispersion_pion.pdf |
Here .jack is the supplied text format for replica data, not a Snakemake feature.
3. Identify independent tasks
Each distinct correlator fit reads the same raw data but uses its own model and interval. Fits can run concurrently when their output names differ and resources permit. Selection must wait for all candidate fits: selecting from an incomplete candidate set changes the calculation. Dispersion fitting must wait for the selected energies.
raw HDF5 + configuration A → fit A → four energy files ─┐
raw HDF5 + configuration B → fit B → four energy files ─┼→ select energies
… ┘ │
four selected files
│
fit dispersion
│
dispersion_pion.pdf
This is the same fan-out and collection pattern as our random-number example. The different task durations do not change the dependencies.
4. Check the source before counting jobs
The reference snapshot has 18 INI files, but input_0.ini and input_6.ini specify the same fit: 1exp, tmin=10, tmax=22. Thus there are 17 distinct fits, each producing four files: 68 distinct candidate energy files. The apparent grid of two models and three choices for each endpoint is not complete: 1exp, tmin=8, tmax=22 is absent.
We retain those configurations and schedule each distinct output set once. Scheduling both duplicates concurrently would make two jobs write the same files.
5. Make completion observable
Use the actual .jack and PDF files as rule outputs. A marker such as fit.done only tells us that a command created a marker. It does not tell us whether the actual output files exist and are current.
We will check scheduling, failure propagation, output structure and unchanged reruns.
Work through the design exercises, then continue to implementation.