Skip to content

perf: cache /tmp/dash0_env_vars instead of re-reading it on every call - #97

Open
RaphaelManke wants to merge 1 commit into
masterfrom
perf/cache-env-vars-file
Open

RaphaelManke wants to merge 1 commit into
masterfrom
perf/cache-env-vars-file

Conversation

@RaphaelManke

Copy link
Copy Markdown
Member

Summary

  • try_read_env_from_file() did a blocking std::fs::read_to_string + a fresh serde_json parse on every call, with no caching.
  • get_resources_attributes() calls it from 4 independent sites (metrics_creation.rs, span_creation.rs, span_mutations.rs, exporter.rs), so a single invocation emitting traces, logs, and metrics could trigger this read+parse repeatedly.
  • std::fs is synchronous; calling it from async code blocks the tokio worker thread for the syscall's duration -- on every invocation, for the life of the warm container.
  • The file is written once by opt/shared.sh at cold start and never changes again for the life of the execution environment, so it only needs to be read once per process.

Fix

Cached behind Mutex<Option<Arc<Option<serde_json::Value>>>>, matching the existing shared-state pattern already used in state/global.rs. Added a #[cfg(test)] reset hook and called it at the top of the module's existing tests, since they intentionally rewrite the file mid-suite to exercise different fallback scenarios -- without the reset they'd silently see a previous test's cached content.

Measured

This crate has no [lib] target for a real criterion/divan bench to attach to, so measured in isolation by replicating the exact read+parse call in a throwaway benchmark: 9.4us/call uncached vs. 35ns/call cached -- a 265x difference.

Test plan

  • cargo build --release
  • cargo test --release (325/325 passing, including all 7 otlp::resources::tests::*)
  • Ran the full suite twice to confirm no test-order flakiness from the cache

@dash0-dev

dash0-dev Bot commented Oct 20, 2026

Copy link
Copy Markdown

Darkplane Automerge is enabled for this repository (mode: Dry run).

Trigger an evaluation for this pull request when it is ready:

Run evaluation

Evaluations on this repository start manually. To have Darkplane Automerge evaluate each pull request on its own, set Trigger to Automatic in the repository's configuration.

@dash0-dev

dash0-dev Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Darkplane Automerge would approve this pull request. Score 77.79 of 100, threshold 75. Customer-function safety 88 (floor 50), Telemetry & config contract 82 (floor 35), Regression coverage 38 (floor 30), Cold start & footprint 97 (floor 25).

Darkplane Automerge would approve this pull request. It scored 77.79 of 100, clearing the threshold of 75. Dry-run mode is on, so nothing was approved or merged.

  • 🟡 Regression coverage scored 38.
    The new ENV_VARS_FILE_CACHE in src/otlp/resources.rs has no test that pins caching: nothing checks that a second read skips the file, or what happens when the file is absent or unreadable on first call (the None is cached for the process lifetime). The existing src/otlp/resources.rs tests (~lines 158–410, run by cargo test, CI green) still cover value lookup, but the PR adds a reset call to each so they bypass the cache. A regression such as a stale or permanently missing resource attribute would pass CI.
The other 3 criteria
  • 🟢 Customer-function safety scored 88.
    The change in src/otlp/resources.rs memoizes the /tmp/dash0_env_vars read behind a parking_lot mutex. It adds no unwrap or panic, no new await, and no I/O on the proxied /invocation/next, /response or /error path (verified by reading). The one residual risk is a read that fails before write_env_vars runs in the wrapper (e.g. a read during INIT): that None would be cached for the process lifetime and drop wrapper-added resource attributes and the service.name fallback. That would only degrade telemetry, and the callers, exporter.rs and metrics_creation.rs among them, appear to run at invocation time, after the wrapper has written the file. I read the callers and did not trace every call path.
  • 🟢 Telemetry & config contract scored 82.
    The change in src/otlp/resources.rs only caches the /tmp/dash0_env_vars read, so precedence for OTEL_RESOURCE_ATTRIBUTES and OTEL_SERVICE_NAME and the emitted attribute keys stay the same, and no config or masking variable changes. Residual risk: a failed or missing read is cached as None for the whole process, so if anything built resource attributes before write_env_vars in opt/shared.sh wrote the file, faas.* and service.name would fall back to unknown_service. I couldn't confirm whether any such early call exists, so I scored it down a little.
  • 🟢 Cold start & footprint scored 97.
    src/otlp/resources.rs now caches /tmp/dash0_env_vars after the first read, which removes repeated file reads and JSON parses. The PR reports 9.4us per call uncached versus 35ns cached. once_cell and parking_lot are already in Cargo.toml, so there are no new dependencies. The one-time read still happens on first use, and the cached JSON stays in memory for the process lifetime, which is negligible.

An AI evaluator produced these scores from the diff, the repository history, and the pull request text. The pull request text is author-controlled, so treat the scores as signals rather than proof. Reviewed for commit 6882ae3.

Was this decision right? Give feedback

try_read_env_from_file() did a blocking std::fs::read_to_string plus a
fresh serde_json parse, with no caching, on every call. get_resources_attributes()
calls it from 4 independent sites (metrics_creation.rs, span_creation.rs,
span_mutations.rs, exporter.rs), so a single invocation emitting traces,
logs, and metrics could trigger this file read and parse repeatedly.

std::fs is synchronous; calling it from async code blocks the tokio
worker thread for the syscall's duration, on every invocation, for the
life of the warm container.

The file is written once by opt/shared.sh at cold start and never
changes again for the life of the execution environment, so it only
needs to be read once per process. Cached behind a Mutex<Option<Arc<...>>>
(matching the existing pattern in state/global.rs), with a #[cfg(test)]
reset hook since this module's own tests intentionally rewrite the file
mid-suite to exercise different scenarios.

Measured in isolation (throwaway benchmark replicating the exact read+
parse call, since this crate has no [lib] target for a real criterion/
divan bench to attach to): 9.4us/call uncached vs 35ns/call cached, a
265x difference.

This branch was successfully deployed

1 active deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant