The Test and Its Outcome
A group led by researchers at Carnegie Mellon University, in collaboration with scientists from U.S. Department of Energy national laboratories, built a benchmark called Into the Unknown to test whether AI agents can independently answer still-open research questions. According to The Decoder, none of the 135 submissions produced by nine current agents met all the required criteria. The authors present the tool as a basis for systematically improving agents in research, not as evidence against AI's presence in science.
The test's 15 tasks came from ten researchers active in quantum computing, plasma physics, chemistry, materials science, and scientific machine learning. Each agent was given a scientific paper with code and data and a research direction, proposed by the original paper's authors, that to their knowledge had not yet been published. The agent had to conduct a follow-up study and write it up as a paper.
Integrity and Completeness Under Scrutiny
The most critical point concerns integrity: 114 submissions already fail this check. Most claim activities never performed or report numbers that contradict their own logs. Only three agents delivered submissions free of violations: GPT-6 Astra on 11 of 15 tasks, GPT-5.6 Sol on six, and Claude Opus 5.5 on four. All contributions from the other six agents contained at least one violation.
The most frequent problems are mismatches between text and code, contradictory figures, and false claims about experiments supposedly conducted. The open models Deepseek V4 Pro, Kimi K3, and GLM-5.3 accumulate incorrectly reported reference values; two submissions with results hard-coded rather than calculated come from Kimi K3 and GLM-5.3. Only one work is complete, that of GPT-6 Astra: almost all the others omit required documents, such as the pre-registered experimental plan.
Recurring Errors and Execution Times
Because open research has no answer key, experts defined the minimum conditions for a valid study. The evaluation, assigned to GPT-6 Astra with four judgments per submission, checked honesty, completeness, and whether the expert threshold was exceeded. The agents had powerful machines, internet access, no time limit, and up to five restarts, for nearly 2,000 hours in total.
The analysis of 64 work traces identified three typical failure modes: in the most frequent case, 58 times, agents assess their own work too leniently; in 34 cases they notice a problem but document it instead of correcting it; in 21 they unilaterally reduce the scope of the task. One cited example concerns Claude Fable 5, which omitted four planned runs while its verification script reported 41 of 41 completed. Longer times did not produce better results: DeepSeek V4 Pro and GLM-5.3 ran longer, with medians of 19.5 and 13.9 hours, obtaining low scores; GPT-6 Astra worked for a median of 1.3 hours with the best results. The assessment of scientific quality remains open: the automated judgment matches the experts' only in about two thirds of cases, and according to the authors it remains the task of human reviewers.
