Anthropic's Claude Fable 5.2 correctly identified the authors of 56% of 398 unsigned reviews and meta-reviews written by six volunteers. The model received one report at a time, with no web access, tools, candidate list or other writing samples.
In a separate test of 1,559 public reviews, Claude identified the writer correctly in 21.4% of the 1,530 cases it answered. It was more accurate on public reviews containing self-citations: 80.2% for 106 such reports, compared with 17.1% for 1,453 without them.
The private-review result gave more weight to volunteers who supplied more reports. Weighting each reviewer equally produced an average accuracy of 49.1%. The researchers said a larger, more varied sample is needed to estimate the broader risk to confidential peer review, and a model's identification would still require independent verification.
Frequently asked questions
-
What information did Claude receive for the private-review test?
Claude saw one unsigned report at a time, without web access, tools, a list of possible authors or other writing samples.
-
How did weighting each reviewer equally change the private-test result?
Average accuracy was 49.1% when each of the six reviewers received equal weight, compared with 56% when each report was weighted equally.
-
How did Claude perform on public reviews?
Claude correctly identified the writer in 21.4% of the 1,530 public-review reports it answered, from a set of 1,559.
-
How did self-citations relate to identification accuracy?
Claude was correct on 80.2% of 106 public reviews containing self-citations, compared with 17.1% of 1,453 without them.
-
Can a model's answer establish who wrote an unsigned review?
No. The model made mistakes, and its identification would require independent verification. The researchers also said a larger, more varied sample is needed to estimate the broader risk.
CryptoSlate