CryptoSlate
The Science Prediction Universe study tested 13 agent configurations across 12 models on 100 public tasks and 16 private tasks covering chemistry, materials science, biology and medicine. Average agreement between configurations reached a Spearman rank correlation of 0.52, compared with 0.33 against measured outcomes, with agent agreement exceeding correlation with measurements on 81 of 100 public tasks.
Pooling rankings produced consensus results 0.08 points above the average individual submission and improved performance on 73 of 100 public tasks, according to the paper. The reported 95% confidence interval for that gain was 0.05 to 0.11.
Why it matters
The central warning is that model consensus is not the same as scientific accuracy. Averaging can cancel errors that differ between systems, but it preserves errors shared across them. One oncology task illustrated the risk: 11 configurations agreed at 0.9 while averaging only 0.27 against measurements.
The researchers linked some shared mistakes to common assumptions about how experimental outcomes should rise or fall. More agents can therefore create a stronger-looking consensus without adding independent information, a limitation for enterprises using multi-agent systems to prioritize research, materials or drug-discovery candidates.
Market impact
The results favor ensembles as decision-support tools rather than automatic substitutes for laboratory validation. Coverage was uneven, with Claude Opus 5.5 and Claude Fable 5.1 scoring only 44 and 30 public tasks largely because of safety refusals. All 13 configurations shared just 28 scored tasks, while the private set contained only 16 tasks across four unpublished experimental series.
The benchmark also included no human prediction baseline, and public tasks relied on published measurements selected after pilot evaluations. The study still found useful private-set performance, with 12 of 13 configurations ranking unpublished experiments significantly better than chance, but its strongest implication is methodological: AI consensus should be measured against external evidence, not treated as evidence in itself.
Frequently asked questions
-
How strongly did the AI agents agree with laboratory measurements?
Their average Spearman correlation with measured outcomes was 0.33, compared with 0.52 between configurations. Agent agreement exceeded correlation with measurements on 81 of 100 public tasks.
-
Did pooling predictions from multiple AI agents improve results?
Yes. Consensus rankings scored 0.08 points above the average individual submission and improved performance on 73 of 100 public tasks. The reported 95% confidence interval was 0.05 to 0.11.
-
Why can agreement among AI agents fail to indicate accuracy?
Averaging reduces errors that differ between systems but preserves mistakes they share. If several agents rely on the same faulty assumption, consensus can look strong while remaining consistently wrong.
-
What limitations did the Science Prediction Universe study have?
Coverage varied by configuration, all 13 systems shared only 28 scored tasks, and the private set contained 16 tasks. The study also included no human prediction baseline.
-
Did the agents show any predictive ability on private scientific tasks?
Yes. Twelve of the 13 configurations ranked the unpublished experiments significantly better than chance overall, although laboratory measurements were still required to verify the answers.
CoinTelegraph
WatcherGuru
TheBlock