Skip to content

Solutions: scientific workflow design

1. Graph

Raw HDF5 data and an INI file feed each correlator fit. The 17 distinct fits produce 68 candidate .jack files. All candidates feed selection, which produces four selected/transformed energy files and a diagnostic PDF. Those four energy files and file.list feed dispersion fitting and its final PDF.

The 17 distinct fits may overlap; selection and dispersion lie downstream. Selection must see the complete candidate set because its result depends on comparing candidates. Declaring only one candidate per momentum would permit premature selection.

2. Dependencies

Declare the INI, HDF5 data and program as inputs. The first two are data/configuration dependencies; the third makes changes to the analysis program visible as an input change. If only the INI is declared, replacing the raw data can leave apparently complete but stale results. Files read indirectly through an INI still need to be represented in the workflow.

Snakemake also tracks other reasons for rerunning, including rule-code and parameter changes. That does not automatically discover every file a Python program opens internally.

3. Missing inputs and stale outputs

  1. The HDF5 file is not declared as an input, so its change is absent from the dependency graph. If everything Snakemake tracks is unchanged, it can reuse stale outputs.
  2. Add the raw HDF5 file and fit_correlator.py, alongside the existing INI input. Newer shared raw data require all 17 distinct fits to rerun, followed by energy selection and dispersion. This propagates the input change to every dependent result.
  3. Explicitly requesting the missing output requires its job. One program invocation produces all four outputs of that fit; this missing file does not require the other 16 fits to rerun.

Two INIs describe the same model and interval and therefore the same four output paths. The teaching workflow schedules that fit once so two jobs do not write the same files.

Continue to implementing and running the workflow.