🦀OC Bench

Same prompt, same harness, one shot per model. Hosted at model-bench.opencrabs.com

The task: "A realistic military armory with rifles, pistols, sniper rifles, grenades, bulletproof vests, ammo crates, weapon racks, animated monitors, and dynamic lighting. Build it in three.js." One-shot, auto-approve, no hand-holding. Harness: OpenCrabs v0.3.79 headless, fresh disposable profile per model, 30 minute hard cap, isolated build dir per model.

Leaderboard

#	model	status	wall time	tokens	cost	turns	tool calls	files*	size*	build
1	mimo-v2.5-pro xiaomi	completed	2m37s	184,752	$0.152	1	2	1	38.5 KB	view build
2	deepseek-v4-flash opencode-ds4flash	completed	13m01s	3,702,436	$0.595	1	30	8	3.9 MB	view build
3	Qwen-Ambassador/Qwen3.8-Max modelscope	timeout (30m cap)	30m00s	-	-	1	-	-	0 B	no build
4	opus claude_cli	timeout (30m cap)	30m00s	-	-	1	-	171	17.5 MB	view build
5	k3 moonshot	timeout (30m cap)	30m00s	-	-	1	5	1	1.6 KB	view build
6	opus-5 claude_cli	failed (rc=1)	0m00s	-	-	1	-	-	0 B	no build

* live on-disk counts excluding node_modules / vendored deps. Claude CLI runs report tokens/cost from opencrabs' view of the session; the heavy lifting happens inside the claude subprocess, so those cells understate real usage.

All samples

deepseek-v4-flash (opencode-ds4flash)

run	status	wall	tokens	cost	turns	tool calls	files	output
20260808-024352	failed (rc=1)	0m01s	-	-	1	-	-	0 B
20260808-025149	completed	13m01s	3,702,436	$0.595	1	30	8	3.9 MB

mimo-v2.5-pro (xiaomi)

run	status	wall	tokens	cost	turns	tool calls	files	output
20260808-024352	completed	4m27s	420,353	$0.279	1	5	1	31.8 KB
20260808-025149	completed	2m37s	184,752	$0.152	1	2	1	38.5 KB

Qwen-Ambassador/Qwen3.8-Max (modelscope)

run	status	wall	tokens	cost	turns	tool calls	files	output
20260808-024352	failed (rc=1)	0m00s	-	-	1	-	-	0 B
20260808-025149	timeout (30m cap)	30m00s	-	-	1	-	-	0 B

opus (claude_cli)

run	status	wall	tokens	cost	turns	tool calls	files	output
opus-run5	timeout (30m cap)	30m00s	-	-	1	-	3,474	66.7 MB

k3 (moonshot)

run	status	wall	tokens	cost	turns	tool calls	files	output
20260808-024352	timeout (30m cap)	30m00s	-	-	1	5	1	1.6 KB

opus-5 (claude_cli)

run	status	wall	tokens	cost	turns	tool calls	files	output
20260808-024352	failed (rc=1)	0m00s	-	-	1	-	-	0 B

Generated 2026-08-08 03:41 UTC by build_landing.py. Bench data: /srv/bench/runs/<run>/<model>/telemetry.json (harvested read-only from each profile's sqlite). Timeouts are a 30-min cap, not a crash.