perf: reduce duplicate push audit storage by dcoric · Pull Request #1795 · finos/git-proxy

dcoric · 2026-10-04T14:37:15Z

Description

Large pushes store the diff repeatedly in processor logs and content, and the final step is duplicated in lastStep. Store each diff once and replace an identical final step with an internal reference in the file, MongoDB, and PostgreSQL sinks. Detail reads restore the complete API response; existing records remain readable.

Push lists and profile activity use the same metadata allowlist across all three sinks. Unused top-level author, message, protocol, and errorMessage fields are excluded. Commit/tag metadata needed by the UI is preserved; full diffs and processor results remain available on the detail endpoint.

Related Issue

Storage portion of #1691 and PostgreSQL parity in #1794, following the merge of #1687. Repository activity summaries and the related UI refresh fixes are now in #1801. Both PRs are needed to complete the agreed scope. The Action.diff refactor in #1693 and #1701 remains separate.

Validation

1,705 unit tests passed; 91.62% line coverage.
67 MongoDB and 87 PostgreSQL integration tests passed, including the shared storage contract and legacy record compatibility.
Six real Git push/fetch scenarios and all 18 current Cypress tests passed against PostgreSQL in the combined verification build.
Manually verified metadata-only list responses, complete detail diffs, restored lastStep, and rendered push details.
Production build, TypeScript, formatting, and lint passed, with the existing tagPush Cypress warning.
Combined with perf: cache repository activity summaries #1801 locally: 729 unit/service/processor tests, 95 MongoDB integration tests, and 101 PostgreSQL integration tests passed. The sink adapter overlap has been resolved and verified locally for when the second PR is updated after the first merges.

A constructed push whose final step held a 2 MiB diff serialized to 8,389,308 bytes before compaction and 2,097,722 bytes afterward. Historical records are preserved rather than rewritten.

Checklist

CONTRIBUTING.md reviewed; Conventional Commits used
FINOS CLA on file
Tests and processor documentation updated
Unit/integration tests, lint, formatting, type checks, and build passed

No configuration schema changes.

Signed-off-by: Denis Ćorić <denis85@gmail.com>

netlify · 2026-10-04T14:37:21Z

✅ Deploy Preview for endearing-brigadeiros-63f9d0 ready!

Name	Link
🔨 Latest commit	`a1ee202`
🔍 Latest deploy log	https://app.netlify.com/projects/endearing-brigadeiros-63f9d0/deploys/6ad200a58c7dce6d6d808f79
😎 Deploy Preview	https://deploy-preview-1795.git-proxy.preview.finos.org
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

codecov · 2026-10-04T14:39:58Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.91%. Comparing base (7a30ae6) to head (a1ee202).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1795      +/-   ##
==========================================
+ Coverage   87.00%   87.91%   +0.91%     
==========================================
  Files         113      121       +8     
  Lines        6463     6809     +346     
  Branches     1253     1289      +36     
==========================================
+ Hits         5623     5986     +363     
+ Misses        536      514      -22     
- Partials      304      309       +5

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: Denis Ćorić <denis85@gmail.com>

jescalada

Looks good so far, just wondering if we could simplify the PR description/title a bit to make it easier to follow.

Since we're technically fixing the audit problem but also doing the push count recalculation stuff, it might even be better to have separate PRs for each 🤔

jescalada · 2026-10-05T01:53:38Z

+  commitFrom: 1,
+  commitTo: 1,
+  error: 1,
+  errorMessage: 1,


Are all the newly added fields here in pushListProjection actually being used? I'm wondering about the errorMessage for instance, since that might contain processor output that may or may not be relevant to the user.

I noticed there's also a few extra ones like protocol, message and author.

jescalada · 2026-10-05T02:36:22Z

+  return index.snapshot();
+};
+
+export const getRepoPushRollupsByCanonicalUrl =


It seems that when loading any of the following dashboard tabs...

Repositories (/dashboard/repo)

Activity (/dashboard/push)

Push Details (/dashboard/push/:id)

User Profile (/dashboard/user/:id)

...we're triggering getRepoPushRollupsByCanonicalUrl, which ends up doing the whole dirty entry updating.

I'm wondering if this isn't accidentally affecting performance, what would happen if multiple users were to access the same pages? If we recalculate the rollups for multiple repos at the same time, wouldn't conflicts arise on dirty detection too?

jescalada · 2026-10-05T02:40:13Z

+  return { pushes, summaries };
+};
+
+const backfillActivity = async (): Promise<void> => {


I noticed that backfillActivity is running on getRepos on the first proxy startup. Wouldn't a cleaner solution be to run this on a migration so that it doesn't block any operations while the proxy is already up and running?

jescalada · 2026-10-05T02:52:15Z

+  let initialization = initialized.get(database);
+  if (!initialization) {
+    initialization = Promise.all([
+      pushes.createIndex({ id: 1 }, { unique: true }),


Was the unique check on push IDs previously enforced? If not, would that cause any clashes on calling getRepos?

jescalada · 2026-10-05T02:55:17Z

+  throw new Error(`Push ${id} changed repeatedly during deletion; retry the request`);
+};
+
+export const rebuildRepoPushRollups = async (): Promise<RepoPushRollupsByCanonicalUrl> => {


I think this is only being used in tests, and we aren't exporting it via src/db/index.ts so I don't think it can be used:

jescalada · 2026-10-05T03:04:03Z


  const att = push.attestation;
-  const ts = DateTime.fromMillis(Number(att.timestamp));
+  const numericTimestamp = Number(att.timestamp);


We're doing the same conversion on RejectionInfo.tsx right below - might want to extract into a helper.

Signed-off-by: Denis Ćorić <denis85@gmail.com>

kriswest · 2026-10-19T14:23:00Z

+
+type StoredPush = Action & { _lastStepIndex?: number; _activity?: unknown };
+
+export const compactPush = (action: Action) => {


Adding a verbal description of what this is doing in a comment would be use full here and on restorePush

I see it in the architecture doc, but it'd be good to have the same description inline in the code

dcoric added 11 commits October 4, 2026 15:48

fix(proxy): avoid duplicating diffs in audit logs

cbf8554

Signed-off-by: Denis Ćorić <denis85@gmail.com>

perf(db): return consistent metadata-only push lists

cc6e851

Signed-off-by: Denis Ćorić <denis85@gmail.com>

test(e2e): verify compact push audits through approval

627577f

Signed-off-by: Denis Ćorić <denis85@gmail.com>

docs: explain diff storage and push list responses

c33df0f

Signed-off-by: Denis Ćorić <denis85@gmail.com>

perf(db): store repeated final audit steps once

4134790

perf(db): update file repository rollups on push changes

e341b9e

perf(db): refresh MongoDB activity for changed repositories

f7952a2

test(db): cover rollup recovery and report integration coverage

6f7c8c8

docs: describe push storage and activity rollup recovery

425d44a

fix(ui): refresh activity after review decisions

25a27e1

fix(ui): parse serialized review timestamps

c63f0e9

dcoric requested a review from a team as a code owner October 4, 2026 14:37

github-actions Bot added the automation label Oct 4, 2026

dcoric added 2 commits October 4, 2026 18:15

test(db): cover file activity initialization recovery

044a561

Signed-off-by: Denis Ćorić <denis85@gmail.com>

test(db): cover MongoDB activity recovery and deletion races

773cc10

Signed-off-by: Denis Ćorić <denis85@gmail.com>

jescalada requested changes Oct 5, 2026

View reviewed changes

kriswest mentioned this pull request Oct 5, 2026

5th October 2026 - GitProxy Meeting Minutes #1797

Closed

17 of 24 tasks

dcoric added 4 commits October 6, 2026 12:02

chore: merge main with the PostgreSQL sink

77de4d9

Signed-off-by: Denis Ćorić <denis85@gmail.com>

refactor: separate repository activity from audit storage

cf021fe

Signed-off-by: Denis Ćorić <denis85@gmail.com>

perf(db): compact PostgreSQL push audits and list responses

c1746a7

Signed-off-by: Denis Ćorić <denis85@gmail.com>

fix(db): preserve MongoDB unset literal types

d6f44ab

Signed-off-by: Denis Ćorić <denis85@gmail.com>

dcoric mentioned this pull request Oct 6, 2026

perf: cache repository activity summaries #1801

Open

4 tasks done

dcoric changed the title ~~perf: reduce push audit storage and repository rollup scans~~ perf: reduce duplicate push audit storage Oct 6, 2026

dcoric mentioned this pull request Oct 6, 2026

Complete PostgreSQL push storage and activity parity #1794

Open

fabiovincenzi mentioned this pull request Oct 7, 2026

refactor: store diff in action object (#1693) #1701

Merged

dcoric added 3 commits October 13, 2026 14:52

chore: merge main and retain PostgreSQL coverage

ea52907

Signed-off-by: Denis Ćorić <denis85@gmail.com>

fix(db): trim push summary metadata

a47440c

Signed-off-by: Denis Ćorić <denis85@gmail.com>

chore: merge main and reconcile diff storage

a1ee202

Signed-off-by: Denis Ćorić <denis85@gmail.com>

kriswest reviewed Oct 19, 2026

View reviewed changes

kriswest linked an issue Oct 19, 2026 that may be closed by this pull request

Reduce stored push document size and rollup scan cost across sink backends #1691

Open

kriswest modified the milestone: 2.4.0 Oct 19, 2026

Navigation Menu

Repository navigation

perf: reduce duplicate push audit storage - #1795

perf: reduce duplicate push audit storage#1795
dcoric wants to merge 20 commits into
finos:mainfrom
dcoric:perf/push-storage-rollups

dcoric commented Oct 4, 2026 •

edited

Loading

Uh oh!

netlify Bot commented Oct 4, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Oct 4, 2026 •

edited

Loading

Uh oh!

jescalada left a comment

Uh oh!

jescalada Oct 5, 2026

Uh oh!

jescalada Oct 5, 2026

Uh oh!

jescalada Oct 5, 2026

Uh oh!

jescalada Oct 5, 2026

Uh oh!

jescalada Oct 5, 2026

Uh oh!

jescalada Oct 5, 2026

Uh oh!

kriswest Oct 19, 2026

Uh oh!

kriswest Oct 19, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		type StoredPush = Action & { _lastStepIndex?: number; _activity?: unknown };

		export const compactPush = (action: Action) => {

Conversation

dcoric commented Oct 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue

Validation

Checklist

Uh oh!

netlify Bot commented Oct 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for endearing-brigadeiros-63f9d0 ready!

Uh oh!

codecov Bot commented Oct 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jescalada left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dcoric commented Oct 4, 2026 •

edited

Loading

netlify Bot commented Oct 4, 2026 •

edited

Loading

codecov Bot commented Oct 4, 2026 •

edited

Loading