On 14 August 2026 Anthropic said that future Claude models would generate watermarked text, using "a version of the SynthID-Text approach published by Google DeepMind in a Nature paper in 2024". "Nothing is added to the text and there are no hidden characters," the announcement says. The mark lives in which words the model chose, so the evidence for it is statistical. It builds with every word the model chose freely and erodes with every word someone replaces. A positive detection can show that a passage is very unlikely to have been produced without a particular key. It cannot show who wrote it, and a negative result is only weak evidence that the text was not produced with the key.

On 28 September 2026 we ran Google DeepMind's open SynthID Text implementation, as shipped in Hugging Face Transformers, on a laptop with Qwen2.5-0.5B-Instruct, a 0.5-billion-parameter open model, and keys we generated ourselves. We measured how length, scripted word swaps, paraphrase and a wrong key move the detection score. That is not Anthropic's watermark, key or detector, none of which is public, and it is not Gemini's; it shows how the published mechanism behaves.

Anthropic's kind of watermark changes where the randomness comes from

A language model does not pick its next token outright: it gives every candidate a probability and a random draw chooses one, and a text watermark works on that draw, either by biasing it or, in the kind Anthropic uses, by replacing its source of randomness. (A token is a word or part of a word; the examples here use whole words.)

Take "After lunch we walked down to the ___", with probabilities invented for illustration: park 0.30, beach 0.20, river 0.15, market 0.12, harbour 0.09, station 0.07, museum 0.04 and library 0.03. Without a watermark, a random number picks among them, so park comes out 30% of the time and library 3%. Temperature, top-k and top-p reshape these probabilities first, but the draw remains.

That kind of watermark, in Anthropic's words, "only changes the source of the randomness". Instead of an arbitrary random number, a hash of a secret key and the last few tokens seeds a pseudorandom function; the SynthID-Text paper uses the previous four (Dathathri et al., Nature, 2024). The function gives each candidate word a score of 0 or 1, its g-value. Someone without the key cannot compute the g-values, so to them the word choices look like ordinary sampling; to a reader, in Anthropic's words, a watermarked response "is indistinguishable from an unwatermarked one". With the key and the model's tokenizer, anyone can recompute the g-values from the text alone, without the model itself, which is what makes detection possible.

For our example and a demonstration key, computed by our own script, the first of several g-values is 1 for river, market, harbour and station and 0 for the rest. Schemes that change only the source of the randomness go back to a 2022 proposal by Scott Aaronson, which Anthropic names as the family's starting point; they leave the model's probabilities intact on average over keys. The best-known alternative is the green list of Kirchenbauer et al. (ICML 2023): treat the words scored 1 as green and raise their probabilities before an ordinary draw, so a watermarked text contains more green words than chance. That changes the output distribution itself, and the Nature paper classes the green list as distortionary, a design that can trade some quality for a stronger mark. SynthID-Text, the scheme Anthropic says it adopted, runs a tournament, and in its non-distortionary form belongs to Aaronson's family.

Tournament sampling, one word at a time

Tournament sampling, published by Google DeepMind as SynthID-Text in Nature in October 2024, replaces the usual single draw with a contest among several draws: it samples candidate words from the model's own probabilities and lets keyed scores knock them out in pairs until one word is left.

Bracket diagram for the next word after 'After lunch we walked down to the', with invented model probabilities: park 0.30, beach 0.20, river 0.15, market 0.12, harbour 0.09, station 0.07, museum 0.04, library 0.03. Eight words drawn from the model, beach, park, market, park, river, beach, park and river, each carry a keyed g-value of 0 or 1. In round one, beach beats park on a tie, market beats park, river beats beach, and river beats park. In round two, market beats beach, and river meets river in a tie. In round three, river, with g-value 1, beats market, with 0, and is the chosen word. Notes say the higher g-value wins each match and a tie is settled by a coin flip; that g-values come from a hash of the key, the preceding words and the candidate, with key demo-key-9 and SHA-256 standing in for SynthID's hash; that with this key river wins 49.75% of the time against a model probability of 15%; and that over all keys each word wins at its model probability.

A toy tournament computed by our own script. The probabilities are invented and SHA-256 stands in for SynthID's hash. The key was picked by a stated rule to show a winner other than the likeliest word, so it is not a typical case.

Three rounds need eight candidates. Drawn from the probabilities above, they are beach, park, market, park, river, beach, park and river. Repeats are normal: the paper calls the candidates "possibly non-unique", because likely words get drawn more than once.

Round one uses each word's first g-value. Beach (0) meets park (0), a tie, and a coin flip sends beach through. Market (1) beats park (0), river (1) beats beach (0), and river (1) beats park (0). Round two uses a second, independent set of g-values: market beats beach, and river meets river. In round three river (1) beats market (0), and the model writes "river".

The key only ever chooses between words the model itself drew, but it shifts the odds. For this key and context, our script computes that river wins 49.75% of brackets against its model probability of 15%, while park falls from 30% to 11.1%. That lean is the watermark.

Averaged over every possible key, the lean disappears. Enumerating every g-value assignment for one- and two-round tournaments returns the model's probabilities exactly, and simulating 100,000 keys for the three-round toy lands within 0.0012 of them. The paper calls this non-distortionary, and the guarantee is precise: per token, on average over keys. With one fixed key, responses to the same prompt become somewhat less varied; the paper says its configuration "preserves text quality and provides good detectability, while having some reduction to inter-response diversity". A distortionary mode, with more than two competitors per match, applies a stronger watermark at some cost to quality.

Real configurations run more rounds: the paper generally uses 30 layers, and so did we (in Transformers, each key adds one). Deeper layers add less; on the toy, the winner's expected g-value is 0.7044 in the first layer and, in our simulation, falls to 0.5281 by the 30th. The tournament comes last: the paper says SynthID-Text "is applied after any such modifications have been made" by temperature, top-k and top-p.

What a detector counts, and why length matters

A detector for this kind of watermark needs the key, the text and the model's tokenizer, but not the model itself: it recomputes the g-value of every token and asks whether their average sits higher than chance would allow.

For the green list this is a count: the detector expresses the excess of green words as a z-score, the number of standard deviations above chance, and Kirchenbauer et al. use z above 4 as their bar, a false-positive probability of about 3 in 100,000. The same green share gives a larger z in a longer text, because chance evens out over more words.

SynthID's simplest detector averages the g-values over every scored token and every layer. For text generated without the key, or scored with the wrong one, this mean score sits near 0.5; for text generated and scored with the same key it sits above 0.5, and we call the gap the drift. The mean is one of several scoring functions in the paper and in Google's reference code, which offers the mean and a weighted mean as detectors that need no training. The paper's default is a learned Bayesian scorer, and the one in Transformers must be trained: the Hugging Face blog recommends at least 10,000 examples. We used the mean because its statistics are easy to state. It is not necessarily the strongest: a preprint by Omidi, Dong and Wang argues that the mean score is "inherently vulnerable to increased tournament layers", while the Bayesian score is not.

Detection is a hypothesis test against "this text was not produced with this key". Under that null hypothesis the g-values behave like independent coin flips, as the paper's supplementary information shows, provided the detector skips any position whose preceding four tokens already appeared earlier in the text (repeated-context masking). Their average then has a known spread that shrinks with the square root of the number of g-values, and a threshold set a fixed distance above chance fixes the false-positive rate: the share of texts made without the key that still get flagged.

Because the spread shrinks with length, the text a detector needs depends on the drift. In an idealised calculation at a 1% false-positive rate with 30 layers, catching about 90% of watermarked texts takes about 43 tokens at a drift of 0.05, about 271 at 0.02 and about 1,085 at 0.01. Halve the drift and you need four times the text.

Entropy, how uncertain the model is about the next token, sets the drift. The toy is far too generous: with eight plausible words at every step, its first-layer winner has an expected g-value of 0.7044, matching the paper's Corollary 27, and its 30-layer average is about 0.60. Real text leans less, because many tokens are nearly forced, and when the distribution is "very low entropy", the paper notes, the tournament "cannot choose tokens that score more highly". Omidi et al. report an average mean score of 0.548 in their setups; our run's median over whole texts was 0.579, against a chance level of 0.4963 explained in the method section. Anthropic says the same in plain words: its watermark is "sparser on factual passages", and code "has generally less watermarking than some other forms of text".

Our run: the score holds steady while the bar drops

In our run, watermarked prose scored about as high at 25 tokens as at 400; what changed with length was how high a score had to be to count as evidence.

We gave the model 60 open-ended prose prompts (essays, stories and explanations of about 500 words), each generated once with the watermark and once without from the same random seed, plus eight low-entropy prompts such as lists, SQL and code. We scored the first 25, 50, 100, 200 and 400 tokens of each text, and the whole text. With one generation per prompt and condition, the spread below is across prompts, not reruns. None of it describes Claude's or Gemini's deployment; Transformers' own source says its hashing differs from the Gemini App's, so detectors trained on it "will not transfer".

Line chart of the median detection score in our run on Qwen2.5-0.5B-Instruct, for passage lengths of 25 to 400 tokens on a log scale. Watermarked text stays between 0.570 and 0.580 while the 1% false-positive line, drawn as a wide pale line, falls from 0.543 to 0.507. Text with 10% of words swapped stays near 0.55, above the line. Text with 25% swapped sits near 0.52, meets the line at 100 tokens and stays above it from 200. Paraphrased text sits near 0.51 and is just above the line only at 400 tokens. Text with 50% swapped, text scored with a wrong key, and unwatermarked text all stay near 0.50.

Line chart of the share of texts flagged at the 1% line in our run on Qwen2.5-0.5B-Instruct, for passage lengths of 25 to 400 tokens on a log scale. Watermarked text is flagged 95% of the time at 25 tokens and 100% from 50. With 10% of words swapped the share rises from 57% to 100%, with 25% swapped from 27% to 100%, and for paraphrased text from 13% to 53%. Text with 50% swapped, text scored with a wrong key, and unwatermarked text stay at 7% or below.

Measured in our run, 28 September 2026: SynthID Text in Hugging Face Transformers 5.17.0 on Qwen2.5-0.5B-Instruct, 30 layers, our own test keys, 60 prose prompts; median scores in the first chart, shares flagged in the second. "Swapped" means a script replaced that share of words at random; it is not human editing. The 400-token paraphrase point rests on 17 texts.

The key is what makes a text score. Clean watermarked prose had a median score of 0.570 at 25 tokens and 0.580 at 400. Unwatermarked texts scored with our key sat at 0.495 to 0.497, and the watermarked texts scored with a wrong key sat at 0.492 to 0.495. The words in that last case are identical to the first; only the key differs.

The bar is what moves. The 1% line, the score a text must beat for a 1% false-positive rate, falls from 0.5427 at 25 tokens to 0.5070 at 400. At that line, 57 of 60 watermarked texts were flagged at 25 tokens and every one from 50 on; at a stricter 0.1% line, 47 of 60 at 25 tokens and again all from 50. From 50 tokens, the lowest-scoring watermarked text beat the highest-scoring unwatermarked one: 0.5413 against 0.5174. Scoring the unwatermarked texts with 100 sets of unused keys flagged 0.89% to 1.10% of them at the 1% line, which confirms that line well but the 0.1% line only roughly, since the scores at each length come from just 47 to 60 texts.

Entropy drove the rest. Across all 68 watermarked texts, the whole-text score tracked the model's average uncertainty per step with a Spearman correlation of 0.92. The eight low-entropy texts, an illustration rather than a measurement at that size, had a median whole-text score of 0.514 against 0.579 for prose, and 6 of the 8 were flagged.

Our detection rates look far better than published ones, and should not be compared like for like. The Nature paper's Fig. 3a shows roughly half of watermarked texts detected at 100 tokens at a 1% false-positive rate (Gemma 7B-IT, temperature 0.7, human-written negatives); a preprint by Nemecek, Chaudhary and Ayday reports 39% at 200 tokens on Gemma-2-9B and Llama-3.1-8B with a mean g-value detector. Our untested inference is that entropy explains most of the gap: our small model at temperature 1.0 and top-k 40 averaged 1.61 nats per step on the median prose text, and larger models at lower temperatures lean less on each token.

How edits and paraphrase wear the watermark down

Because each SynthID g-value depends on a token and the four before it, replacing one token resets up to five scored positions to chance, so the evidence left after an edit tracks the share of five-token windows that survive.

If tokens are replaced at random, replacing 10% leaves about 0.9⁵ = 0.59 of five-token windows intact, and replacing 30% leaves 0.17. The green list, which looks back one token, keeps 0.81 of its windows at 10%. Google's SynthID Text documentation describes this trade for the window length, ngram_len: a larger value makes the watermark "more detectable", "at the cost of being more brittle to changes".

Our run shows the rule at work. For each edited text we took its excess over the chance level of 0.4963 as a share of its clean version's excess: the share of the signal it kept. The table adds one condition the charts omit, deleting 25% of words rather than replacing them; deletion shortens the texts, so none reaches 400 tokens.

Condition 5-token windows intact (median) Share of excess kept (median) Flagged at 100 tokens Flagged at 200 tokens Flagged, whole text
10% of words swapped 0.634 0.626 59/60 58/58 60/60
25% of words deleted 0.386 0.382 39/59 54/57 60/60
25% of words swapped 0.302 0.287 31/60 51/58 57/60
Paraphrased by the same model 0.173 0.133 19/57 21/52 28/60
50% of words swapped 0.057 0.056 2/60 1/58 6/60

At the median, the score keeps about the share of windows that survive. Text by text the link is looser: Spearman correlations of 0.45 to 0.56 for the 10% and 25% edits, and 0.14 for 50% swaps, where almost nothing survives. Window survival itself ran a little above the token rule: 0.634 at 10% of words swapped, against 0.9⁵ = 0.59. The likely reason, which we did not measure, is that a word is not always one token, so a five-token window often spans fewer than five words.

Length recovers diluted evidence: a smaller drift does not disappear, it just takes more text to show. With 25% of words swapped, 16 of 60 texts were flagged at 25 tokens, 31 of 60 at 100 and all 46 that reached 400. Deletion behaved the same way: at 25 tokens 12 of 60 texts were flagged, at 200 tokens 54 of 57, and over the whole text all 60. With 50% swapped, about 6% of windows survived and the median score fell to 0.500 to 0.503, close to chance. The detector found few of these texts, 1 to 3 per length and 6 of 60 over the whole text, against 1 of 60 unwatermarked texts: a small residue of signal, too little to rely on.

These are scripted edits, not human editing. A script replaced randomly chosen words with words from a fixed 100-word list, often leaving nonsense. People edit in clusters, which would leave more windows intact for the same share of words changed; that is our inference, untested. Anthropic's own wording fits the mechanism: "Light editing probably won't remove the watermark completely; a complete rewrite where every word is replaced will."

Paraphrase is the same mechanism applied unevenly. We asked the same 0.5B model, without the watermark and from one fixed prompt, to paraphrase each watermarked text once. Some outputs were not faithful; one begins "As an AI assistant, I don't have a physical form". Over the whole text 28 of 60 paraphrases were still flagged, mostly those that copied more: their median window survival was 0.289, against 0.101 for the rest, with a Spearman correlation of 0.84 between survival and score. The split is not clean, though: 15 of the 28 flagged paraphrases kept no more windows than the best-surviving unflagged one.

Larger studies agree that paraphrase weakens rather than simply erases the mark, given enough text. In the Nature paper's supplementary experiments on Gemma 2B-IT and 7B-IT, a Gemini Ultra paraphrase was "quite strong" as an attack, yet "the watermark is still detectable if the text is long enough", which on its Fig. C3 means thousands of tokens. For the green list, Kirchenbauer et al.'s reliability paper (ICLR 2024) found paraphrases by 14 human writers "clearly detected on average after 800 tokens" at a false-positive rate of 1 in 100,000.

How we ran the test, and how to rerun it

Every setting that affects our numbers is listed here, from a run on an Apple M2 Pro laptop with 16 GB of memory on 28 September 2026.

  • Software. Python 3.11.7, torch 2.14.0, transformers 5.17.0 and numpy 2.4.6, on Apple's MPS backend in float32.
  • Model. Qwen/Qwen2.5-0.5B-Instruct at revision 7ae557604adf67be50417f59c2c2f167def9a775, Apache 2.0, through its chat template.
  • Watermark. SynthIDTextWatermarkingConfig with ngram_len 5, 30 keys, sampling_table_size 65,536, sampling_table_seed 0 and context_history_size 1,024. Our keys relate to no production system and come from numpy.random.default_rng(20260927); the wrong-key control used seed 20260928.
  • Sampling. Temperature 1.0, top-k 40, top-p 1.0 and repetition penalty 1.0, following Hugging Face's SynthID example script and overriding Qwen's defaults of 0.7, 20, 0.8 and 1.1. Up to 420 new tokens, 600 for paraphrases.
  • Processor order. A hook on generate() showed top-k running before the watermark, the order the paper describes. Transformers 4.46 had it the other way round (issue #34630) until PR #34849, first released in 4.47.0. Google's documentation still says "v4.46.0+", so use 4.47.0 or later.
  • Detector. A mean g-value score re-implemented from Google's reference detector_mean.py (commit 9644e26), using Transformers' compute_g_values and compute_context_repetition_mask. Texts were decoded and re-tokenised before scoring, as a detector would see them; that changed the tokens of 12 of 136 texts and moved medians by at most 0.001.
  • Threshold. z = (score − p) / √(p(1 − p) / (30n)), with p = 0.4963 and n the number of unmasked scored positions (L − 4 for a prefix of L tokens). Flagged at 1% when z > 2.3263, at 0.1% when z > 3.0902.

The block below applies the same method and settings to one prompt: it rebuilds our keys, generates with and without the watermark from the same seed, and scores both texts. It runs as shown with the versions above. It will not reproduce our exact texts, which were generated ten prompts at a time.

import numpy as np
import torch
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    SynthIDTextWatermarkingConfig,
    SynthIDTextWatermarkLogitsProcessor,
)

MODEL = "Qwen/Qwen2.5-0.5B-Instruct"
REVISION = "7ae557604adf67be50417f59c2c2f167def9a775"
DEVICE = "mps" if torch.backends.mps.is_available() else "cpu"

# Our 30 test keys, one per tournament layer, from a fixed NumPy seed.
keys = [int(k) for k in np.random.default_rng(20260927).integers(0, 1_000_000, size=30)]
WM = dict(
    ngram_len=5,
    keys=keys,
    sampling_table_size=2**16,
    sampling_table_seed=0,
    context_history_size=1024,
    skip_first_ngram_calls=False,
)
SAMPLING = dict(do_sample=True, temperature=1.0, top_k=40, top_p=1.0, repetition_penalty=1.0)

tok = AutoTokenizer.from_pretrained(MODEL, revision=REVISION)
model = AutoModelForCausalLM.from_pretrained(MODEL, revision=REVISION, dtype=torch.float32)
model = model.to(DEVICE).eval()


def generate(prompt, watermark, seed=1000, max_new_tokens=420):
    chat = tok.apply_chat_template(
        [{"role": "user", "content": prompt}], tokenize=False, add_generation_prompt=True
    )
    enc = tok(chat, return_tensors="pt").to(DEVICE)
    extra = {"watermarking_config": SynthIDTextWatermarkingConfig(**WM)} if watermark else {}
    torch.manual_seed(seed)
    with torch.no_grad():
        out = model.generate(**enc, **SAMPLING, **extra, max_new_tokens=max_new_tokens)
    return tok.decode(out[0, enc.input_ids.shape[1]:], skip_special_tokens=True)


# The g-value table depends on the device that builds it: copy the one
# generate() used on DEVICE into a detector that runs on the CPU.
table = SynthIDTextWatermarkLogitsProcessor(**WM, device=torch.device(DEVICE)).sampling_table.cpu()
detector = SynthIDTextWatermarkLogitsProcessor(**WM, device=torch.device("cpu"))
detector.sampling_table = table
p = table.float().mean().item()  # chance level for one g-value: 0.4963 for the MPS table


def score(text):
    ids = torch.tensor([tok(text, add_special_tokens=False).input_ids])
    g = detector.compute_g_values(ids)[0].double()  # [positions, 30 layers]
    mask = detector.compute_context_repetition_mask(ids)[0].double()  # skip repeated contexts
    n = mask.sum().item()
    mean = (g * mask[:, None]).sum().item() / (30 * n)
    z = (mean - p) / (p * (1 - p) / (30 * n)) ** 0.5
    return mean, z


prompt = "Write an essay of about 500 words on why many cities are planting more street trees."
for watermark in (True, False):
    mean, z = score(generate(prompt, watermark))
    print(f"watermark={watermark}: mean g-value {mean:.3f}, z {z:.2f}, flagged at 1%: {z > 2.3263}")

One trap matters for anyone rerunning the method. In this version of Transformers, the same sampling_table_seed builds a different g-value table on Apple's MPS backend than on the CPU: only 50.34% of entries agree, and the share of ones is 0.4963 on MPS against 0.4996 on the CPU. That share is the chance level, which is why our null sits at 0.4963. Scored with the CPU-built table, our watermarked texts fell to chance, with medians of 0.498 to 0.502. The block above copies the table from the device that generated the text, and anyone scoring texts they saved from an earlier session must do the same. This is a property of this implementation, not of SynthID.

Two earlier attempts on 27 September stalled or were stopped, and none of their output was used. Two scoring bugs were fixed before the saved outputs: a 99-word replacement list, and a control standardised with the wrong chance level. No prompt was excluded, and we did not run the planned green-list comparison or train the Bayesian detector. The prompts, all 136 generations, the paraphrases, the scripted edits, every score and the g-value table are kept on file with our research record and are not published with this piece; the settings and code above are what a reader needs to rerun the method, not to inspect our outputs.

Who watermarks AI text, as checked on 27 September 2026

As of 27 September 2026, Anthropic and Google document keyed text watermarks in their assistants' output, OpenAI documents none, and neither Anthropic nor Google publishes error rates for its deployed detector. Vendor defaults have changed since then; our guide to EU AI Act marking lists what Anthropic, OpenAI and Google mark as of 21 October 2026.

Provider Text watermark, as documented Who can run detection Sources and dates
Anthropic "A version of the SynthID-Text approach". The Help Center model table lists text watermarks for Claude Fable 5.1, Mythos 5.1 and Opus 5.5, and for Claude Opus 5 on Anthropic's own surfaces. For Opus 5 on cloud partners, the page stated as of 27 September 2026 that watermarking would become available gradually from 14 September 2026 and fully within one week. Ten other listed models, including Fable 5, Sonnet 5 and Haiku 4.5, had no text watermark yet; models released before 2 August 2026 are to be covered by 2 December 2026. A detection API in private preview for "eligible organizations as required under EU law", such as regulators, law enforcement, media, fact-checkers, independent researchers, educational organisations and EU civil society groups, and for enterprises "similarly obligated" to verify watermarking for their own compliance. Announcement of 14 August 2026, updated 1 September 2026; Help Center page modified 22 September 2026.
Google SynthID-Text in Gemini. The Nature paper said it was "currently watermarking responses in Gemini and Gemini Advanced"; the SynthID page covers text "generated by the Gemini app and web experience". The current configuration is not published. The public verification surfaces on the SynthID page cover image, video and audio. Nature, 23 October 2024; SynthID page undated.
OpenAI None documented in the pages we checked. Its Help Center lists provenance signals for images and audio and a goal "to expand provenance signals to all modalities including text". In 2024 it wrote: "Our teams have developed a text watermarking method that we continue to consider as we research alternatives." Not applicable. Help Center capture of 22 September 2026; 2024 post updated 4 August 2024.

For Claude's watermark, Anthropic has published no number of layers, window length, distortionary setting, entropy cut-off, key management scheme, detector type, threshold or error rate. Nemecek, Chaudhary and Ayday's preprint (9 September 2026) lists similar gaps and calls for the "release of matched outputs, configuration disclosure, accredited audits, a shared evaluation protocol, and interoperable detection". Two of Anthropic's claims cannot be checked from outside: that the watermark "carries no identifying information", and that its method "does not have any practical impact on the quality or content" of Claude's outputs. The Nature paper's live test on about 20 million Gemini responses found no statistically significant difference in thumbs-up or thumbs-down rates, which supports the method in general, not Claude's configuration of it.

The legal driver is Article 50(2) of the EU AI Act, which requires providers of systems that generate text to mark outputs "in a machine-readable format" from 2 August 2026; systems already on the market have until 2 December 2026 under Regulation (EU) 2026/1744 of 8 July 2026. Anthropic applies its watermark worldwide because "we don't yet have a durable way to scope it by region". Who that duty falls on when an app is built on a vendor's model, and from when, is in our explainer on Article 50 marking.

What a detection result can and cannot show

A positive result from a keyed watermark detector supports one narrow statement, that the passage is very unlikely to have been produced without that key, and only when the threshold is calibrated for the passage's length and the key has stayed secret.

Anthropic's wording is similarly narrow: a watermark "can only determine that Claude was likely involved with the content at some point. It cannot distinguish 'Claude wrote this' from 'Claude heavily edited this.'" Its Help Center adds that a detected mark "provides a signal that content was processed by Claude, but is not fully conclusive". A positive says nothing about who pasted, edited or submitted the text, or why.

Scale changes what a flag means. In an illustrative calculation, screening 1,000,000 unwatermarked documents at a 1% false-positive rate produces 10,000 false flags. If 1 in 1,000 screened documents is watermarked and the detector catches 90% of those, only 8.26% of flags are true; at a false-positive rate of 1 in 100,000, 98.90% are. OpenAI made the same point about its own method in 2024: "applying it to large volumes of text would lead to a large number of total false positives".

A negative result is only weak evidence that the text was not produced with the key, because many ordinary things remove or thin the mark. Anthropic's Help Center lists the reasons a mark may not be found: an older model, text "heavily edited, paraphrased, translated, or mixed into other writing", or a very short passage. Factual text and code carry less mark to begin with. Google's documentation says detector confidence "can be greatly reduced when an AI-generated text is thoroughly rewritten, or translated to another language".

The premise can also break. If a key leaks, a positive stops being evidence about one provider: the Nature paper names "vulnerability to stealing, spoofing and scrubbing attacks", and the reliability paper notes that with a breached key, an attacker with white-box access to a strong paraphrasing model could break the watermark. A detector without repeated-context masking, or with one threshold for every length, can be miscalibrated. And while the published SynthID-Text scheme is zero-bit, marking only presence or absence, Nemecek et al. point out that designs able to carry an identifier exist, so "no identifying information" is a design choice outsiders cannot verify.

None of this describes style-based AI detection software such as Pangram, which looks for tells in phrasing and has no key; Anthropic contrasts its watermark with such tools explicitly.

Anyone handed a watermark result can test it against three points before acting on it.

  1. Whose key, and what false-positive rate at this length. If the provider has not published it, and as of 27 September 2026 none has, the result is the provider's assertion rather than a number anyone can check.
  2. How much freely chosen text there is. Long, open-ended prose carries the most evidence; short, factual, code-like or heavily reworked text carries little. In our run the evidence was graded: it fell as the share of surviving five-token windows fell, and rose with length. Over whole texts, scripted swaps that left a median 30% of windows intact were still flagged 57 times in 60, deletions that left 39% were flagged in all 60, and paraphrases that left 17% were flagged 28 times in 60. Only when very few windows survived, about 6% after swapping half the words, did detection fall close to the unwatermarked level: 6 of 60 flagged, against 1 of 60.
  3. What claim is being made. "This passage was very likely produced with this key at some point" can be supported. "This person used AI and did no work of their own" cannot, from a watermark alone, and a negative result supports neither side.

Whether outsiders can ever check these results depends on providers publishing the configurations, matched outputs and detector error rates that Nemecek et al. ask for, and none has yet.