[perfloop-agent]

Fixes #1853.

Description

The active-upload claims introduced in #1633 can leave healthy staged files ineligible for later uploads when a sibling fails. Repeated failures can prevent the cleanup sweep that would eventually release those claims.

Keep the selected batch paths until every upload task finishes, then release the whole batch on an I/O or join error while returning the first error. Also release validated files' claims when snapshot publication fails. Size-rejected files remain staged without contributing descriptors, and successful publication still precedes field statistics and local deletion.

LocalFS fault-injection checks reach recovery on later attempts for both file-role assignments after I/O, task-join and snapshot read/write failures. They also cover repeated validation rejection and retry after an existing manifest was updated. The collector regression test is in src/storage/object_storage.rs.

Cargo's upload-batch and staged-file cleanup tests passed (respectively 2 and 1 passed, with 0 failed or ignored). Rustfmt passed for the changed file. Clippy reports pre-existing diagnostics on both revisions and remains for CI.

Failed collections still do not partially publish. Error returns now wait for outstanding upload tasks, and retries may repeat completed remote uploads. Production fault frequency and affected batch sizes remain unknown.

---

This PR has:

• been tested to ensure log ingestion and log query works.
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added documentation for new or modified features or behaviors.

Case and evidence

---

Made by Perfloop. Human sponsor: Tomás Senart. To ask for a change, comment or review in plain words. How Perfloop handles pull requests.

Confidence Score: 4/5

Fix repeated storage and billing counts before merging.

Findings

1.  Retries inflate storage totals ▶

Fix with agent prompt

### Issue 1
src/storage/object_storage.rs:1258-1259
When one file uploads successfully but a sibling fails, `update_storage_metrics` has already counted the successful file. Releasing every claim here leaves that file staged and eligible for retry. The next attempt uploads and counts it again, even though it overwrites the same remote file. Releasing claims after snapshot failures causes the same problem.

Repeated failures inflate stored-byte totals and the parquet counts sent to `BillingMetricsCollector`. Keep retries, but make these increments happen once per staged file.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

This PR makes failed upload batches eligible for retry.

• Failed staging uploads leave files ready for another cycle.

Acknowledged behavior: perfloop-agent states that error returns wait for outstanding uploads and retries may repeat completed remote uploads.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Claim staged files] --> B[Upload and validate files]
  B --> C[Increase storage and billing metrics]
  C --> D{Any task failed?}
  D -->|Yes| E[Wait for all tasks and release batch claims]
  D -->|No| F{Publish snapshot}
  F -->|Failure| G[Release successful file claims]
  E --> H[Retry staged files]
  G --> H
  H --> B
  F -->|Success| I[Calculate field stats and delete staged files]

Loading

_{Reviews (1) · Last reviewed commit: "fix: release failed staging upload batch..." · Reviewed by Greptile}

[greptile-apps]

Retries inflate storage totals

When one file uploads successfully but a sibling fails, update_storage_metrics has already counted the successful file. Releasing every claim here leaves that file staged and eligible for retry. The next attempt uploads and counts it again, even though it overwrites the same remote file. Releasing claims after snapshot failures causes the same problem.

Repeated failures inflate stored-byte totals and the parquet counts sent to BillingMetricsCollector. Keep retries, but make these increments happen once per staged file.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/storage/object_storage.rs
Line: 1258-1259

Comment:
**Retries inflate storage totals**

When one file uploads successfully but a sibling fails, `update_storage_metrics` has already counted the successful file. Releasing every claim here leaves that file staged and eligible for retry. The next attempt uploads and counts it again, even though it overwrites the same remote file. Releasing claims after snapshot failures causes the same problem.

Repeated failures inflate stored-byte totals and the parquet counts sent to `BillingMetricsCollector`. Keep retries, but make these increments happen once per staged file.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.