Implement and run the scientific workflow
We will connect the existing analysis programs with one Snakefile, inspect the full plan and run one fit. The complete analysis is an optional extension or instructor-prepared demonstration. Complete the design lesson first.
1. Prepare an isolated analysis
Start in snakemake-workshop with .venv active. You need both cloned repositories and the workshop-support directory obtained in the first-workflow lesson.
python -m pip install -r workshop-support/science/requirements-science.txt
python -m pip check
python workshop-support/science/prepare_science.py SnakeMake.main work-science
cd work-science/program
Installation should finish without errors and pip check should report no broken requirements. The helper creates a new work-science directory; it refuses an existing destination. It copies the original numerical programs, raw data and reference INIs, then installs the teaching Snakefile and plotting wrapper. It does not alter either repository.
work-science/
├── data/
│ ├── raw_data/pion_lambda.h5
│ ├── energy_fits/ (candidate energies)
│ └── dispersion_data/ (selected/transformed energies)
└── program/ ← run Snakemake here
├── Snakefile
├── correlators/inis/
├── energies/
└── dispersion/
The requirements pin the direct packages and Snakemake used for the workshop. The plotting wrapper produces the figures without an external LaTeX installation.
2. Read a rule as a file contract
The Snakemake rule reference explains input functions, expand, logs and thread declarations.
Open Snakefile in your editor. Its first rule, all, requests all candidate energies, all selected energies, the selection diagnostic and the final dispersion PDF. We explicitly retain the intermediate data as workshop targets so a missing candidate is rebuilt even when the PDFs already exist. Compare this choice with the missing-intermediate exercise in the previous workflow. The fit_correlator rule declares the raw data, configuration and program as inputs. Its output pattern describes all four momentum files from one fit.
The key configuration lookup is:
config=lambda wildcards: FIT_CONFIGS[wildcards.fit],
Here FIT_CONFIGS maps an output identity, such as tmin10-tmax22_model1exp, to an INI path. The small function looks up the configuration for the concrete job. It does not perform a fit while Snakemake reads the workflow. Reading the existing INIs during parsing determines the candidate set; duplicate output identities are scheduled once.
In the output expression:
expand("../data/energy_fits/d{momentum}_energy_{{fit}}.jack", momentum=MOMENTA)
expand resolves the four momentum labels immediately. The doubled braces retain {fit} as a wildcard for later job matching. Each job therefore has four outputs, rather than four jobs competing to run the same fit.
The command changes into correlators because the original program interprets INI paths from there. Snakemake's input/output paths remain relative to work-science/program; changing directory in a shell command does not redefine those declarations. Named logs keep the output from different fits separate. The adaptation also limits numerical-library thread pools to one thread per job, matching threads: 1.
Keep the candidate set controlled
The original selection script scans its input directory. Keep data/energy_fits limited to the generated candidates for this exercise. If you change or remove configurations, prepare a new working copy: stale files from an earlier candidate set could otherwise affect selection without appearing in the current dependency graph.
3. Follow the complete graph
combine_fits declares all candidate files as inputs. It writes the four selected energy files and its PDF. fit_dispersion declares those four files, file.list, both dispersion programs and the rendering wrapper. Snakemake therefore sees the whole dependency chain in one graph.
This teaching Snakefile replaces the historical nested Snakemake invocations and marker-file dependencies. It keeps the actual analysis programs and supplied candidate set. We do not execute the historical program.solution/Snakefile as part of this walkthrough.
4. Core exercise: plan everything, execute one fit
All commands in this section run from work-science/program:
snakemake --cores 2 --scheduler greedy --dry-run
For a fresh directory and the recorded repository snapshot, expect 17 fit_correlator jobs, one combine_fits, one fit_dispersion and one all: 20 jobs including the target rule. If counts differ, check the repository revision and INIs before executing.
Now run just one single-exponential fit by requesting one of its outputs:
snakemake --cores 2 --scheduler greedy ../data/energy_fits/d000_energy_tmin10-tmax22_model1exp.jack
head -n 3 ../data/energy_fits/d000_energy_tmin10-tmax22_model1exp.jack
snakemake --cores 2 --scheduler greedy ../data/energy_fits/d000_energy_tmin10-tmax22_model1exp.jack
One job creates all four momentum files for this configuration. The displayed header should begin with 400, the number of replica rows; the following lines contain an index field and an energy value. The second identical target request should do no work. You have now exercised the real fit program without waiting for every fit. Complete exercises 1 and 3, then continue to the cluster lesson or the extension below.
5. Extension: complete and inspect the analysis
The raw data contain 400 replicas in each of four channels. The instructor should prepare a full run before the session for discussion.
To complete your own analysis, stay in work-science/program and run:
snakemake --cores 2 --scheduler greedy
The fit branches can overlap, but selection waits for all of them. Runtime depends on your machine and installed numerical libraries; use the progress log rather than expecting a fixed classroom timing.
After successful completion, inspect the files and log:
python - <<'PY'
from pathlib import Path
for directory in ("../data/energy_fits", "../data/dispersion_data"):
print(directory, len(list(Path(directory).glob("*.jack"))))
for filename in ("energies/energies_chosen.pdf", "dispersion/dispersion_pion.pdf"):
print(filename, Path(filename).stat().st_size, "bytes")
PY
cat logs/dispersion.log
snakemake --cores 2 --scheduler greedy
Expect 68 candidate files, 4 selected files and two nonempty PDFs. Open both PDFs with your PDF viewer. Check labels and obvious missing content. The last command should report nothing to do when inputs and outputs are unchanged.
Continue with the exercises, then cluster execution.