Loading prices…
🔥BULLISH

AI-Run Journal Flags Flaws in 11 of 16 Research Drafts

Code-based audits overturned statistical conclusions, while text-only referees struggled to distinguish corrected research from flawed drafts.

AI-Run Journal Flags Flaws in 11 of 16 Research Drafts
AI-Run Journal Flags Flaws in 11 of 16 Research Drafts

An experimental journal run by AI agents caught flaws in 11 of 16 research drafts, but its text-only referees raised the same objection in 10 corrected versions. In a separate audit, scripts run on authors' uploaded data overturned the main statistical conclusion in 77 of 156 objections classified as major, according to the preprint A Society of AI Scientists.

Why it matters

The Journal of Agent Research emerged from a 12-week study involving 936 AI agents across 35 human-operated labs. Agents in one lab founded the journal and launched a panel of automated referees. Humans configured computing systems, capped model-access spending and selected broad research themes.

The two tests exposed a distinction between identifying a possible problem in a manuscript and checking whether it changes a result. The preprint's authors said reviewers continued to flag some corrected papers because the revisions openly described the limitation.

In the audit, a guest editor turned objections into scripts and tested them against uploaded data. The checks also overturned the main result for 24 of 120 minor objections. Several objections could concern the same study.

Market impact

The findings illustrate both the potential and limits of AI agents in research verification, a possible use for automated systems in scientific workflows. The study was exploratory and nonrandomized, and participants were relied on to disclose whether they followed restrictions on manual work.

The public archive includes papers, code and agent discussions, but excludes the raw experiment data used in the journal's checks.

Frequently asked questions

  1. Why did AI referees repeat objections to corrected papers?

    The preprint's authors said the reviewers continued to flag the issue because the revised papers openly described the limitation.

  2. How did the journal test whether reviewer objections changed results?

    A guest editor turned objections into scripts and ran them on authors' uploaded data to see whether the main statistical conclusions held.

  3. How did checks on minor objections compare with major objections?

    The checks overturned the main result in 24 of 120 minor objections, compared with 77 of 156 major objections. Several objections could concern the same study.

  4. What role did humans play in the AI research study?

    Human operators configured computing systems, capped spending on model access and chose broad research themes.

  5. What are the study's limitations?

    The study was exploratory and nonrandomized. The authors relied on participants to disclose whether they followed restrictions on manual work, and the public archive excludes raw experiment data.

Source attribution
Aggregated from CryptoSlate · Verified · Last refreshed 27m ago
Open original →