Blog · 2026-10-15 · baryreview editors

Are AI detectors reliable? What the evidence says, and why it's the wrong question

If you publish AI-assisted content, someone has probably run it through a detector, and someone has probably been alarmed by the score. So the question “how good is AI detector accuracy, really?” is a fair one. The evidence is clearer than the marketing suggests, and it points in a direction most people do not expect: the detectors are not the problem, and beating them is not the goal.

This post lays out what the vendors claim, what independent researchers have measured, why the two disagree, and why a business publishing an article should care about a different question entirely.

What the vendors claim

Detection vendors publish benchmarks, and they are not shy about them. GPTZero’s accuracy page reports 99% accuracy distinguishing AI from human text with a false-positive rate of 1% or less, and 96.5% accuracy on mixed human-and-AI documents with a 0.9% false-positive rate. The brief we work from also notes that Turnitin targets a document-level false-positive rate under 1%.

To be fair to the vendors, the same GPTZero page says “no tool is likely 100% accurate,” notes that performance drops on mixed documents, and says the model is updated monthly, so published figures may not describe the current version. Those are honest caveats. The figures themselves come from the vendors’ own test sets, which is normal for any product benchmark, and it is also the reason independent numbers look different.

What independent evidence says about AI detector accuracy

Four sources, none of them selling a detector or a humanizer.

OpenAI retired its own classifier. In January 2023 OpenAI released a classifier for AI-written text and reported that it “correctly identifies 26% of AI-written text (true positives)” while “incorrectly labeling human-written text as AI-written 9% of the time (false positives).” On July 20, 2023, the company added a note that the classifier was “no longer available due to its low rate of accuracy.” The company with the most data about its own model’s output decided that the detection problem was not solved well enough to ship.

Detectors are biased against non-native English writers. Liang, Yuksekgonul, Mao, Wu and Zou, in a peer-reviewed paper in Patterns (2023), “evaluated the performance of seven widely used GPT detectors” on 91 TOEFL essays written by non-native speakers and on essays by US eighth-graders. The non-native essays were flagged as AI-written at an “average false-positive rate: 61.3%,” and “all detectors unanimously identified 19.8% of the human-written TOEFL essays as AI authored.” For the native-speaker essays, “the accuracy for native samples remained near perfect.” The authors’ explanation is that detectors key on low “perplexity,” meaning predictable word choice, and non-native writers tend to use a smaller vocabulary. Anyone who writes plainly, in a second language or by habit, looks more like a model to a detector.

Fourteen tools, none reliable. Weber-Wulff and colleagues tested “12 publicly available tools and two commercial systems (Turnitin and PlagiarismCheck)” in a 2023 study published in the International Journal for Educational Integrity. Their summary: “the available detection tools are neither accurate nor reliable,” the tools “have a main bias towards classifying the output as human-written rather than detecting AI-generated text,” and “content obfuscation techniques significantly worsen the performance of tools.”

Recent human writing gets flagged too. A 2026 preprint from Karr, Khvatskii, Hua and Chawla ran commercial detectors on published academic abstracts. Unmodified human-written abstracts from 2023 to 2025 were flagged 9% to 15% of the time at the default threshold, with non-STEM fields flagged more often. Abstracts that had received light AI edits, the sort of assistance most style guides permit, were flagged 38% to 80% of the time. And text run through a commercial humanizer was flagged less than 4% of the time. The authors conclude that “detector scores should not serve as standalone misconduct evidence.”

For a broader survey of the research, the Wikipedia article on AI content detection collects these and other studies with citations.

Why both sets of numbers can be true

The vendor benchmarks and the independent results are not lying to you. They are measuring different things.

Test sets differ. A vendor benchmark typically uses clean, long, native-English samples and fresh model output. Independent researchers use the text that exists in the world: short pieces, second-language writers, formal academic prose, text that was drafted by a model and then edited by a person. Detectors do worst on exactly the kinds of text that real publishing produces.

Thresholds move. Every detector reports a probability and applies a cutoff. Lower the cutoff and you catch more AI text and flag more humans. The Karr study’s 9% to 15% figure is at a 0.50 threshold; the vendor’s “under 1%” is at whatever threshold the vendor chose for the benchmark. Neither is wrong. They are different settings of the same dial.

Base rates punish you. Suppose a detector really did have a 1% false-positive rate. Run 10,000 human-written pages through it and about 100 are flagged. If AI-written pages are rare in that set, most of the flags you see are wrong. This is arithmetic, not a criticism of any product, and it is why the researchers above keep saying the score should not be the only evidence.

The target moves. Models change every few months, detectors retrain monthly, and humanizers are built specifically to defeat the current detector. The Karr finding that humanized text evades detection more than 96% of the time is the arms race in one number. A detector is a lock. Humanizers are built to pick it.

Why “will it pass a detector” is the wrong question

Here is the part that matters for a business publishing content.

Google does not run a detector on your page. Its guidance is about “rewarding high-quality content, however it is produced,” and its helpful-content documentation asks who authored the page, how it was made and why, not what a classifier scores it. The thing Google explicitly penalizes is “generative AI tools or other similar tools to generate many pages without adding value for users,” which is a judgment about value, not about provenance.

Readers do not run a detector either. They read the page. If it says something false, something vague, or something they already knew, they leave. If it says something specific, true and useful that they could not easily find elsewhere, they stay and maybe buy. A detector score of “0% AI” does not help with any of that. A page can score as fully human and still be padded, wrong and useless. A page can score as fully AI and be the best thing on the internet about its topic, because a person with knowledge checked every line.

Humanizers optimize for the wrong target, and they drift. A humanizer rewrites text to lower a detector’s probability score. The only thing it is measuring is the score. It is not measuring whether the sentence still means what you meant. A University of Hong Kong project studying these tools puts it plainly: “Humanization can introduce semantic drift, where the original intent of the text subtly shifts.” A community humanizer benchmark weights meaning preservation at only 32% of its score, which tells you how the category prioritizes. If your article said the warranty covers parts and labor for two years, a humanizer may hand you back a sentence about “comprehensive protection for an extended period,” which is both worse writing and a different claim.

So the loop looks like this: a model drafts the page, a detector flags it, a humanizer rewrites it to beat the detector, and nobody in that loop has checked whether the page is true. The two tools have been pointed at each other and the reader has been left out.

What to check instead

The question that actually determines whether an AI-assisted page works is “is it true, specific and useful?” That breaks into checks a person can run.

1. Every fact resolves. Each statistic, date, price, quote and name has a source you opened. If the source does not exist or does not say that, it goes. Models produce confident, specific, sourced-sounding claims that are invented; this is the single most common serious problem in AI drafts.
2. Claims about your own business are correct. The draft describes the product the model imagines. Compare it to the product you have.
3. The page keeps the title’s promise. “How to choose X” contains a decision. “Is X worth it” contains an answer.
4. There is something only you could have written. A number you measured, a mistake you made, a customer question you actually get. Without it the page is interchangeable.
5. The logic holds. No conclusion the body did not earn, no section that exists to reach a word count, no contradiction between paragraph two and paragraph nine.
6. The voice is a person’s. Hedges, tidy triads and empty transitions are worth removing because they waste the reader’s time, not because a classifier counts them.

None of these require a tool, and none of them are what a detector measures. They do require someone who knows the subject to read the page the way a skeptical customer would. That is the job an editor does, and it is the job we do at baryreview: a professional editor reads your published article, marks every problem, and emails you corrections you paste into the AI that wrote it. We do not run detectors and we do not humanize anything. The list of what the editor checks is the list above, in more detail.

A fair word about the detector vendors

None of this means detector companies are acting in bad faith. They publish benchmarks, they state caveats, they update their models, and the better ones tell educators not to treat a score as proof. The difficulty is the task. Detecting authorship from text alone, when the text may have been drafted, edited, translated or rewritten by any mix of people and models, is hard, and the independent evidence says it is not solved to the standard a consequential decision requires. That is a statement about the problem, not the people working on it.

It also means the detector is answering a question a publisher does not need answered. You know how your page was made. What you need to know is whether it is right.

Takeaway

Treat a detector score as one weak signal and never as a verdict, because independent research puts false-positive rates anywhere from 9% to over 60% depending on who wrote the text. Do not spend money making a page score “more human”; spend it making the page true, specific and useful, which is what readers and Google both actually measure. One informed reader checking every claim does more for a page than any rewrite tool.