AI Coding Benchmark Builder

• The direct competitors are not Claude, GLM, Codex, Copilot, or Semgrep. They are the benchmark and eval ecosystem:
• SWE-bench, Terminal-Bench, RealVuln, CAIBench, CyberSecEval, Aider benchmark, and vendor benchmark pages.
• LangSmith, Braintrust, Langfuse, and similar eval / observability platforms.
• Developers' own Google Sheets, Markdown tables, or one-off scripts.
• External signals:
• [Semgrep's GLM 5.2 cyber benchmark](https://semgrep.dev/blog/2026/we-have-mythos-at-home-glm-52-beats-claude-in-our-cyber-benchmarks/) compares the same IDOR dataset, prompt, models, and harnesses, and emphasizes that the harness affects outcomes.
• The [Codebase-Memory paper](https://arxiv.org/abs/2603.27277) shows that MCP-based knowledge graphs can reduce tokens and tool calls, which means coding-agent quality is not just a model choice.
• [RealVuln](https://arxiv.org/abs/2604.13764) brings real vulnerabilities, false-positive traps, and F1/F3 metrics into security scanner evaluation, proving that developers care about reproducible metrics.
• [CAIBench](https://arxiv.org/abs/2510.24317) argues that security ability cannot be measured by knowledge tasks alone, and that multi-step scenarios and scaffolding affect performance.
• Current gaps:
• Academic benchmarks are too heavy for indie developers to reuse quickly.
• Vendor charts are often hard to reproduce and cannot answer “will this work on my repo and task?”
• Eval platforms are useful after deployment, but less direct for choosing between Claude Code, Copilot, Codex, GLM, or an MCP memory harness today.
• Few tools combine prompt design, harness notes, context strategy, manual scoring, F1, cost, and report export in a lightweight flow.
• Open slot: a lightweight AI coding benchmark tool site for indie developers and small teams that want a practical, reproducible benchmark without building an enterprise eval platform.

8h MVP Checklist

1. 1.Build the benchmark case data model and 20-30 high-quality cases.
2. 2.Build Benchmark Builder and prompt / JSON template generation.
3. 3.Build Results Evaluator with finding checks, false-positive/missed-finding entry, and metrics.
4. 4.Build Markdown / JSON report export.
5. 5.Build Case Library filters and preview details.
6. 6.Add Home, About, FAQ, and SEO metadata.
7. 7.Add 2-3 static sample results for launch demos.