Skip to content

Add remove_redundant_context_v1 A/B experiment to pr-sous-chef workflow - #56257

Merged
pelikhan merged 4 commits into
mainfrom
copilot/remove-redundant-context
Aug 27, 2026
Merged

pelikhan merged 4 commits into
mainfrom
copilot/remove-redundant-context

Conversation

Copilot AI commented Aug 27, 2026 •

Copy link
Copy Markdown
Contributor

pr-sous-chef is the highest-traffic tracked workflow, making it the best candidate to test whether removing a redundant "Required skip rules per PR" backup table from the main-agent prompt reduces execution duration without regressing eval pass rates — the same three skip conditions are already enforced deterministically by the prefilter step and re-verified by the pr-processor sub-agent.

Changes

  • Experiment: Added remove_redundant_context_v1 (control vs. candidate) to pr-sous-chef.md frontmatter, targeting grader:execution-duration as the primary metric with eval:comment-added / eval:pr-evaluated guardrails (≥0.90 pass rate), mann_whitney analysis, and a 15,000ms minimum effect threshold.
  • Graders: Declared graders: execution-duration: {} — since any explicitly listed built-in grader causes the rest of the built-in registry to populate with defaults, this reliably surfaces execution-duration even after merging with imported custom graders.
  • Shared graders import: Added shared/graders.md to imports:, pulling in the custom trajectory graders (recurrence-determinism, recurrence-laminarity, state-revisit-probability-rep).
  • Prompt gating: Wrapped the "Required skip rules per PR" table in a template conditional so it's only rendered for the control variant, keeping candidate behavior otherwise identical:
{{#if experiments.remove_redundant_context_v1 == 'control' }}
## Required skip rules per PR
...
{{#endif}}
  • Recompiled pr-sous-chef.lock.yml to reflect the frontmatter changes.

Copilot AI and others added 2 commits August 27, 2026 06:36
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Remove redundant context in pr-sous-chef workflow Add remove_redundant_context_v1 A/B experiment to pr-sous-chef workflow Aug 27, 2026
Copilot AI requested a review from pelikhan August 27, 2026 06:38
@pelikhan
pelikhan marked this pull request as ready for review August 27, 2026 06:39
Copilot AI balanced review requested due to automatic review settings August 27, 2026 06:39
@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot resolve the merge conflicts in this pull request

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an A/B experiment to measure whether removing redundant skip-rule context improves PR Sous Chef execution duration without reducing evaluation pass rates.

Changes:

  • Adds balanced control/candidate experiment configuration and grader instrumentation.
  • Gates the redundant skip-rules table to the control variant.
  • Regenerates the compiled workflow with experiment persistence and grader artifacts.
Show a summary per file
File Description
.github/workflows/pr-sous-chef.md Defines the experiment, graders, and prompt gating.
.github/workflows/pr-sous-chef.lock.yml Compiles experiment assignment, grading, and state persistence.

Review details

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

  • Files reviewed: 2/2 changed files
  • Comments generated: 0
  • Review effort level: Balanced

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

@copilot resolve the merge conflicts in this pull request

Merge conflicts resolved in 9c60c51 (merged main, recompiled pr-sous-chef.lock.yml).

@pelikhan
pelikhan merged commit 474c6ee into main Aug 27, 2026
1 check passed
@pelikhan
pelikhan deleted the copilot/remove-redundant-context branch August 27, 2026 06:55
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v0.87.8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[harness-experiment-proposal] pr-sous-chef — context assembly/remove_redundant_context A/B harness experiment

3 participants