From Judge to Reward
In the post published on the Hugging Face blog, the team recounts that it built ThinkingBox to verify an agent's work by comparing the final state of the database with the expected outcome. Each ThinkingBox-Bench task, 507 in total, starts a real backend, provides the agent with MCP tools and a simulated user with private context, then runs checks on the record actually left in the database. Verification concerns the result, not the sequence of calls made.
According to the report, the next step was to note that for most tasks the same executable check can return 1 or 0 directly from the final database state. Hence the idea of using it not only as an evaluator but as a reward function to train agents on tasks outside the benchmark.
Task Selection and Results
The team explains that it built a training pool with tasks in ThinkingBox format outside the benchmark, choosing them based on a difficulty defined relative to the model's policy. The probability of success of each task was reportedly estimated once with the base model before training, and the pool was then kept fixed. On Qwen3.8-27B, after 50 GRPO steps, average success rose from 51.70% to 60.78%, while tasks completed correctly in all twenty attempts rose from 38 to 107.
The text clarifies the distinction between metrics: pass@1 measures average success across all 10,140 attempts, while the 20-out-of-20 figure indicates the share of tasks solved in every run. Training reportedly increased repeatable success markedly, with a slight decline in coverage at least once. The cited case concerns a corporate hotel booking: before training, the model modified the booking without manager approval in 15 out of 20 attempts; afterward, according to the post, it preserved the booking and payment in all twenty.
What remains to be verified
The material presented is a report of the source and not an independent comparison. The figures refer to a single model, Qwen3.8-27B, and 50 training steps; the evaluation was conducted on 507 tasks described as out of distribution relative to training. The blog argues that the improvement reflects greater repeatability rather than an expansion of capabilities, but the explanation remains a hypothesis attributed to the authors. Still to be verified are the reproducibility of the training pool, the actual serving costs, and behavior on tasks other than those considered.
