Skip to content

docs: browser agents benchmark table shows the current release #39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
arjunpredev merged 1 commit into from
Oct 6, 2026
Merged

docs: browser agents benchmark table shows the current release #39

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions browser-agents/overview.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,9 +64,9 @@ Our runs on two public benchmarks of tasks on live websites, next to the numbers
| Benchmark | Browser Agents | TinyFish, published |
| --- | --- | --- |
| WebVoyager, 641 tasks | 93.9% | 91.1% |
| Online-Mind2Web, 300 tasks | 73.8% | 89.9% |
| Online-Mind2Web, 300 tasks | 78.3% | 89.9% |

The Browser Agents figures are the mean of two full runs of each benchmark: 94.2% and 93.6% on WebVoyager, 73.7% and 74.0% on Online-Mind2Web. Under a stricter WebVoyager rule that needs a screenshot showing each claim, the mean is 91.6%. TinyFish's numbers come from its own runs, with a different harness, patch set, and grader, and we did not re-run them. On Online-Mind2Web, Browser Agents trails TinyFish's published number.
The Browser Agents figures are the mean of two full runs of each benchmark: 93.1% and 94.7% on WebVoyager, 80.3% and 76.3% on Online-Mind2Web. Under a stricter WebVoyager rule that needs a screenshot showing each claim, the mean is 91.4%. TinyFish's numbers come from its own runs, with a different harness, patch set, and grader, and we did not re-run them. On Online-Mind2Web, Browser Agents trails TinyFish's published number.

The [benchmark report](https://pre.dev/browser-agents-benchmark.html) covers the protocol, results by site and level, the patched tasks and why, and a CSV of every task result from both runs.

Expand Down
Loading