Skip to content

Add experiments stored in a single tar file - #154

Open
MBus123 wants to merge 1 commit into
aibasel:mainfrom
MBus123:experiment-archive
Open

MBus123 wants to merge 1 commit into
aibasel:mainfrom
MBus123:experiment-archive

Conversation

@MBus123

@MBus123 MBus123 commented Oct 31, 2026

Copy link
Copy Markdown

On cluster file systems with inode quotas (e.g. Lustre on Arrhenius), an experiment with tens of thousands of runs creates hundreds of thousands of files. This PR adds Experiment(archive=True) (also accepted by FastDownwardExperiment), which keeps the whole experiment directory in a single uncompressed tar file <path>.tar from the build step on. Everything else behaves as before.

How it works:

  • build writes all experiment and run files directly into the archive. Symlink resources store absolute targets, since runs execute elsewhere.
  • runs (local and Slurm) extract the experiment files once per worker into a temporary directory (respecting TMPDIR), then extract each run directory, execute it exactly like the existing job scripts (driver.log/driver.err, skip already-started runs), and append the files the run created or changed.
  • parse extracts each run directory into a temporary directory, runs the parsers, and appends the new properties file.
  • fetch reads the run properties and logs directly from the archive. The eval directory stays <path>-eval.

The archive is append-only: a later member overrides an earlier one with the same name. Writers hold an exclusive flock and readers a shared one. The archive always ends with exactly two zero blocks, so appending starts at a known offset and never scans the archive.

Two changes affect non-archived experiments:

  • FastDownwardRun now calls fast-downward.py by a path relative to the run directory instead of an absolute path, since the experiment directory doesn't exist for archived experiments.
  • Fetcher.fetch_dir() now delegates to a helper shared with the archive code path. Its behaviour is unchanged.

Limitations:

  • Custom steps that expect the directory <path> don't work with archived experiments.
  • A file that a run deletes from its run directory stays in the archive.
  • Extraction uses tarfile's filter argument, which needs Python ≥ 3.10.12, 3.11.4 or 3.12.

Testing:

  • tests/test_archive.py builds, runs, parses (twice) and fetches the same experiment in both modes and checks that the properties are identical. It also checks the end-of-archive invariant and that a failed append leaves the archive unchanged.
  • The vertex-cover example gives identical properties in both modes, apart from solve_time.
  • The archived Slurm job body was executed locally with SLURM_ARRAY_TASK_ID set, and four concurrent appenders produced an intact archive.
  • flock works on Arrhenius's Lustre mounts (mounted with flock).
  • I haven't run it on the cluster yet.

Experiment(archive=True) keeps the experiment directory in <path>.tar from
the build step on. Runs extract their files into a temporary directory and
append the files they create or change; a later member overrides an earlier
one. Writers hold an exclusive flock and readers a shared one, and the
archive always ends with two zero blocks, so appending never scans it.
Parse and fetch read the archive directly. Fast Downward runs call the
driver relative to the run directory, since the experiment directory does
not exist for archived experiments.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant