Top all-time · #23264
Voratiq
Company blog · voratiq.com/blog · on HN as languid-photicaiprogramming
Voratiq is the company blog at voratiq.com, writing about ai and programming. Since 2025, 3 different Hacker News readers have linked it 10 times, most often “YOLO in the Sandbox”. No feed was found, so read it on the site itself.
Coding agents compete head-to-head in production codebases. Ratings come from whose code gets merged, not benchmark scores.
Most-linked posts?
- YOLO in the Sandbox2 people · 3 mentions · first linked 2025-12-18
What Hacker News says?
Yes, the signal we are measuring is quite different from most evals. We are measuring sth much closer to: when multiple agents compete on the same spec, which one produces the patch that holds up best in code review? Most evals are static / synthetic, and for code, generally stop at tests. Test evals are weak proxies for quality since it's difficult to encode qualities like scope creep/churn, codebase fit, maintaina…
Comment linking /blog/your-workflow-is-the-eval
on “GPT-5.5 Price Increase: What It Costs”Submitted Test Evals Are Not Enough as a story · 2 points
HN discussionWe track performance vs. the all-in cost of completing real engineering tasks, rather than cost per token. [1] Cost per token is a bit misleading because, as others have noted, different models use tokens in different ways. (Aside - This is also why TPS isn't a great metric). We found that 5.5 is about 1.5-2x more expensive overall. On a "Pareto" basis, we only find 5.5 xhigh worth it. At the lower reasoning levels,…
Comment linking /leaderboard?x=cost
on “GPT-5.5 Price Increase: What It Costs”Submitted Codex vs. Claude Code vs. Gemini CLI – Agent Leaderboard as a story · 2 points
HN discussionIt’s very hard to encode the properties that matter most in code in tests. [1] [1] https://voratiq.com/blog/your-workflow-is-the-eval
Comment linking /blog/your-workflow-is-the-eval
on “Why SWE-bench Verified no longer measures frontier coding capabilities”Submitted YOLO in the Sandbox as a story · 2 points
HN discussion> Have you had any "learned the hard way" moments? A big lesson for us is that you still need to be careful even in a sandbox. We've been running Claude/Codex/Gemini in sandboxed YOLO mode and have seen some interesting bypass attempts. [1] A few examples: - created fake npm tarballs and forged SHA‑512s in our package‑lock.json - masked failures with `|| true`, making blocked operations look successful - cloned a wo…
Comment linking /blog/yolo-in-the-sandbox/
on “Ask HN: How are you sandboxing coding agents?”Submitted Selection Rather Than Prediction as a story · 1 points
HN discussionmakes sense! we wrote something yesterday about the weaknesses of test-based evals like swe-bench [1] they are definitely useful but they miss the things that are hard to encode in tests, like spec/intent alignment, scope creep, adherence to codebase patterns, team preferences (risk tolerance, etc) and those factors are really important. which means that test-evals should be relied upon more as weak/directional prio…
Comment linking /blog/test-evals-are-not-enough/
on “Many SWE-bench-Passing PRs would not be merged”Submitted Observed Agent Sandbox Bypasses as a story · 1 points
HN discussion