Repository navigation
Conversation
Experiment(archive=True) keeps the experiment directory in <path>.tar from the build step on. Runs extract their files into a temporary directory and append the files they create or change; a later member overrides an earlier one. Writers hold an exclusive flock and readers a shared one, and the archive always ends with two zero blocks, so appending never scans it. Parse and fetch read the archive directly. Fast Downward runs call the driver relative to the run directory, since the experiment directory does not exist for archived experiments.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On cluster file systems with inode quotas (e.g. Lustre on Arrhenius), an experiment with tens of thousands of runs creates hundreds of thousands of files. This PR adds
Experiment(archive=True)(also accepted byFastDownwardExperiment), which keeps the whole experiment directory in a single uncompressed tar file<path>.tarfrom the build step on. Everything else behaves as before.How it works:
TMPDIR), then extract each run directory, execute it exactly like the existing job scripts (driver.log/driver.err, skip already-started runs), and append the files the run created or changed.propertiesfile.<path>-eval.The archive is append-only: a later member overrides an earlier one with the same name. Writers hold an exclusive
flockand readers a shared one. The archive always ends with exactly two zero blocks, so appending starts at a known offset and never scans the archive.Two changes affect non-archived experiments:
FastDownwardRunnow callsfast-downward.pyby a path relative to the run directory instead of an absolute path, since the experiment directory doesn't exist for archived experiments.Fetcher.fetch_dir()now delegates to a helper shared with the archive code path. Its behaviour is unchanged.Limitations:
<path>don't work with archived experiments.tarfile'sfilterargument, which needs Python ≥ 3.10.12, 3.11.4 or 3.12.Testing:
tests/test_archive.pybuilds, runs, parses (twice) and fetches the same experiment in both modes and checks that the properties are identical. It also checks the end-of-archive invariant and that a failed append leaves the archive unchanged.solve_time.SLURM_ARRAY_TASK_IDset, and four concurrent appenders produced an intact archive.flockworks on Arrhenius's Lustre mounts (mounted withflock).