Tips & Best Practices

Why an AI checker's score swings, and what to keep instead

We scored five texts with two open language models. The Declaration of Independence came out more predictable, the trait the first checkers read as machine-written, than a product description a model wrote, and eight small edits doubled the draft's perplexity on one of them. Here's how these checkers work, what the studies found, and what to keep as your record instead.

12 min read
Free Guide

The short answer

  • An AI checker's percentage is a model's estimate, not a measurement of who wrote the words. What the figure counts differs from checker to checker, and the verdict comes from a line someone chose. A few edits can move a text's score, and a score near the line can cross it.
  • We watched the underlying signal move. We measured perplexity, the signal the first checkers used, with two open language models. The 1776 Declaration of Independence came out far more "predictable" than a product description a model wrote for us. Changing eight phrases in that 188-word description raised its perplexity by more than half on one model and doubled it on the other.
  • The research agrees. OpenAI withdrew its own checker in 2023 for low accuracy. In a Stanford study, seven checkers on average flagged 61% of essays by non-native English writers as AI. In a 14-tool academic test, even the best tool was right only about four times in five.
  • Your drafts are the evidence. Version history, dated drafts and your notes show how a text was written. A percentage can't.

If you run a shop, this cuts both ways: your product copy might get flagged, or you might be judging a freelancer's work by a checker's percentage. Here's how the number is made, why it moves, and what to keep instead.

What people are asking

The same complaint keeps coming up in public reviews of AI checkers. People describe the same text scoring fully human one day and half AI the next. They say a percentage of "AI used" isn't a real metric, and they ask what is actually being measured. You can read them yourself in the Trustpilot reviews of one checker (October 2026), its Firefox add-on reviews (November 2026) and a Hacker News thread from 6 November 2026. In that thread, people point out that the percentage can be the share of text segments flagged, not a probability.

How a checker turns writing into a number

There are two broad kinds.

The first kind asks a language model how surprised it is. A language model predicts the next word. Feed it a text and you can measure how likely it found each word that actually came next. Average that over the text and you get perplexity: low means the words were what the model expected, high means they surprised it. Early checkers also looked at burstiness, how much that surprise varies from sentence to sentence, on the theory that people write unevenly and models write at one steady level.

GPTZero's own explainer from March 2023 describes exactly this, and gives a rule of thumb: "generally, a perplexity above 85 is more likely than not from a human source." The same page now carries a note: "As of autumn 2023, GPTZero no longer uses perplexity and burstiness for its AI detection because we migrated to a deep-learning based architecture."

The second kind is a trained classifier. A model is shown millions of examples labelled human or AI and learns to tell them apart. Pangram's technical report (version 3, July 2024) describes this approach and explains why it chose a classifier over perplexity: perplexity-based methods "fail on human-written documents that happen to be in the training set of LLMs, such as the Declaration of Independence."

Three different things get called "the score", and it helps to keep them apart:

  • The model's output for the text, or for each part of it.
  • The line someone chose, above which a part counts as AI.
  • The figure you're shown. Depending on the checker, that can be a probability for the whole text, or the share of the text whose parts crossed the line. Originality.ai's own explainer (25 October 2026), for example, says its score shows "the percentage of your text that is Likely AI, Likely Original, or Mixed writing".

The figure moves when the output moves, and with a line in play, a small change in output near it can flip a whole passage from one side to the other. None of the three is a record of who held the pen.

We measured it

We didn't run any commercial checker for this post, so nothing below is any checker's percentage or verdict. Instead we measured perplexity, the signal the first kind of checker was built on, on our own laptop with two open models, a small one and a larger one about seven times its size, so you can see how the signal behaves and how much the choice of model matters.

We scored five texts:

  1. 1The Declaration of Independence, its opening, from the National Archives transcript. Written in 1776, and a text that language models meet in training, as the Pangram report notes.
  2. 2Two paragraphs on coffee from Mrs Beeton's Book of Household Management, 1907 edition, chapter 48 on Wikisource. Human, and far less famous.
  3. 3The same Beeton paragraphs rewritten by us in short, common words. Most facts survive, though some detail is lost: "a smaller and rounder bean than any other" became "a small, round bean".
  4. 4A model-written product description. We asked the larger model for about 200 words on a 250 g bag of single-origin Colombian coffee beans.
  5. 5That description with eight phrases we changed, the kind of tidying an editor does. For example, "expertly sourced" became "bought", and "Rich in antioxidants and full of character, this" became "This".

For each text we report perplexity, and the spread of perplexity between sentences (the standard deviation) leaving out the opening sentence. Every text's first words arrive with no context, so its first sentence scores high whoever wrote it.

TextWordsPerplexity, small modelPerplexity, larger modelSpread between sentences, small / larger
Declaration of Independence, 17761801.361.38only two sentences
Beeton on coffee, 190725216.7010.587.76 / 2.91
Beeton in plain words (our rewrite)21816.3212.086.68 / 3.49
Model-written product description1889.414.201.33 / 0.45
The same, eight phrases changed16915.278.714.73 / 6.55

Measured on 17 November 2026. Lower perplexity means "more predictable", which the first kind of checker read as "more likely AI". These are perplexities, not checker percentages.

Four things stand out.

A text from 1776 came out the most predictable. The Declaration came out at 1.36 and 1.38, against 9.41 and 4.20 for the text a model actually wrote. Both models have very likely met it in training, so every word is the one they expect. That's the failure the Pangram report names, and any text that sounds like a lot of other text will score low too. Product descriptions and policy pages are written that way on purpose.

Small edits move the signal a long way. Eight changed phrases in a 188-word description took its perplexity from 9.41 to 15.27 on the small model, and from 4.20 to 8.71 on the larger one. The spread between sentences rose too, and nothing about who wrote the draft changed. We didn't measure any checker's percentage, but a signal that moves this much with light editing is what you'd expect to push a passage back and forth across a line.

Two models, two answers. The same draft scored 9.41 on one model and 4.20 on the other. The larger model is the one that wrote it, and it gave the text less than half the small model's perplexity. So a perplexity number depends on which model measured it, and a checker built on one model can disagree with a checker built on another.

No single line sorts these five. On the small model, any line high enough to call the model's draft (9.41) "AI" also calls the Declaration (1.36) "AI", and the edited draft (15.27) sits just below the 1907 text and our plain rewrite of it (16.70 and 16.32). Lines are set on one model's scale, so GPTZero's 2023 rule of thumb of 85 can't be laid over our numbers at all.

One result didn't go the way we expected. Rewriting Beeton in plainer words barely changed its perplexity on either model (16.70 to 16.32, and 10.58 to 12.08). That's one pair of paragraphs, though, against a study of 88 essays below, so trust the study over our single sample.

What the studies found

These are larger tests, each checked against its own text.

OpenAI withdrew its own checker. It launched an AI classifier on 31 January 2023 that, on OpenAI's own challenge set, caught 26% of AI-written text and wrongly labelled human text as AI 9% of the time. The page also said it was "very unreliable on short texts (below 1,000 characters)". It added: "Text that is very predictable cannot be reliably identified." The note added on 20 July 2023 reads: "the AI classifier is no longer available due to its low rate of accuracy."

Non-native writers are flagged far more often. Liang and colleagues at Stanford (Patterns, 2023) ran seven checkers over 91 essays written for the TOEFL English test. On average the checkers called 61.22% of them AI-generated. All seven agreed on 18 of the 91, and 89 of the 91 were flagged by at least one. The same checkers were near-perfect on 88 essays by US eighth-graders. When the researchers had a model enrich the TOEFL essays' word choices, the false positive rate fell to 11.77%. When they had it simplify the US essays' words, the rate rose from 5.19% to 56.65%. Plainer words, more "AI".

An independent test found the best tool right about four times in five. Weber-Wulff and seven co-authors (International Journal for Educational Integrity, 2023) tested 12 public tools and two commercial systems, Turnitin and PlagiarismCheck. They concluded the tools "are neither accurate nor reliable". The best of them, Turnitin, scored 76%, 79% or 81%, depending on which of their three ways of counting partly correct answers was used, and most tools did far worse. For AI text that had been run through a paraphrasing tool, the paper puts overall accuracy at 26%.

Paraphrasing undoes detection. Sadasivan and colleagues at the University of Maryland and Harvard (2023, revised January 2025) showed that repeatedly paraphrasing watermarked AI text of 300 tokens dropped the detection rate, at a 1% false positive rate, from 99.3% to 9.7%.

The newer classifiers claim much better. Pangram's report claims a 0% false positive rate on the same TOEFL essays and 0.09% on another set of essays by English learners. Those are the vendor's own figures on benchmarks it chose, and we haven't been able to test them. Even so, a classifier still gives you a score and a line, and the line is still someone's choice.

Why the figure moves

Put together, a checker's percentage for one text can change because:

What to keep instead

  • the text changed, even slightly (eight phrases moved ours a long way),
  • the length changed: OpenAI said its own checker was "very unreliable on short texts (below 1,000 characters)", which is roughly 150 to 200 words,
  • the checker's model changed, as GPTZero's did in 2023 when it moved to a trained classifier,
  • the text is predictable by nature: famous, formal, formulaic, or written by someone using a smaller vocabulary in a second language,
  • the threshold sits close to your text's score, so tiny differences flip the verdict.

If you need to show how something was written, keep the process, not the score.

Write where versions are saved. Google Docs keeps a version history of every document. Its help page shows how: open the document, click Last edit at the top right, and pick an earlier version to see who changed what and when. You can name a version (up to 40 per document) so it isn't merged with the ones around it. Only people with edit permission can browse the history, so walk a view-only reader through it on your screen.

If you code, use what you already have. Product copy kept in a Git repository has a dated, line-by-line history for free. A commit per draft is a stronger record than any checker's opinion.

Keep the scraps. The brief, your notes, the supplier's spec sheet, the photos you wrote from. A product description that mentions the actual roast date on the bag is easy to defend.

Ask for the reasons, not the number. If a client, marketplace or school shows you a percentage, ask which sentences it reacted to and why. A tool that can't say is giving you an opinion, not evidence.

Worked example: compare two drafts

Saved drafts help most when you can show what changed between them. Here's how we checked our own edit list for the measurement above, with the free text comparison tool.

  1. 1Open the tool and paste the earlier draft on the left. We used the model-written product description.
  2. 2Paste the later draft on the right. We used the version with eight phrases changed.
  3. 3Read the result under the two boxes. Removed text is struck through in red and added text is in green, in one running copy, with a count above it. Ours said 25 words added and 45 removed.
The Unwrite text comparison tool with the model-written coffee description in the left box, our edited version in the right box, and below them one copy with the eight changes marked: removed words struck through in red, added words in green, and a count of 25 words added and 45 removed

All eight edits are there, in place, and nothing else moved. Where an old and a new word share letters, the tool marks only the letters that differ, so "Experience" to "Here are" keeps one "e". That's the record: two dated drafts and an account of what changed between them, which says far more about who wrote it than any score.

On a phone the two boxes stack, and the marked copy follows them:

The same comparison on a phone-width screen, showing the count and the first lines of the marked copy with "expertly sourced" struck through and "bought" added

The comparison runs in your browser, so a client's unpublished copy isn't sent anywhere while you check it.

If you run a shop

Google doesn't rank by who wrote it. Google's guidance on AI-generated content (8 February 2023) is headed "Rewarding high-quality content, however it is produced" and says "Appropriate use of AI or automation is not against our guidelines." What it treats as spam is content made "with the primary purpose of manipulating ranking in search results". So a checker's score isn't what decides how your product pages rank. Being accurate, specific and useful to a buyer is.

Don't judge freelancers by a percentage. If you commission copy, ask for drafts in a shared document with version history, or a short note on sources. That's fairer to a writer who learnt English second, and it tells you more about the work.

What checking costs

If you still want a checker, here's what two of them charge, read from their own pages on 17 November 2026, in US dollars:

CheckerFreePaid, monthlyWords includedPay as you go
Pangram7-day trial, then $20 a monthIndividual $20; Professional $65300,000 a month (Individual)API at $0.05 per 100 words
Originality.ai10,000 words oncePro $14.95 ($12.95 a month billed yearly); Enterprise $179200,000 a month (Pro); 1.5 million (Enterprise)one-off credits, 100 words each, kept for two years

As a worked figure: checking 500 product descriptions of 300 words each is 150,000 words. That fits inside either cheaper plan for one month, or costs $75 at Pangram's API rate. We left out GPTZero and Copyleaks because their pricing pages didn't show our reader any prices, and we only quote what we could read.

Whichever you choose, the score is still a score near a line. Use it as a prompt to look closer, never as proof.

Related reading

For more on the patterns readers notice in AI drafts, see 14 signals that betray AI-written text. For tidying a model's draft before it goes on a page, see cleaning ChatGPT output for your website.

Sources

Every source was read on 17 November 2026, and every figure here was checked against the text around it.