Recent news story: Anthropic announces that very soon, all its models will embed a watermark in all output they generate to mark it as AI-generated. This applies to all categories of output, including text. And apparently, the watermark wouldn’t be just metadata in a file; it’s meant to stay with the text as it is copied and pasted elsewhere, so it must be in the text itself.
Question: How would you do that? The first idea that came to my mind would be an acrostic, where the first letters of each word spell out a message. This has a long literary tradition. No doubt Anthropic would come up with something much more sophisticated than that, but I suppose the general idea would be the same? That the structure and arrangements of characters in the text encodes a message hidden underneath the plain message spelt out in the text?
If that’s the general idea, then I suppose the watermark could easily be removed by editing the text? Or is there a way to introduce some kind of redundancy in this letter arrangement to ensure that it is preserved even if the text is moderately edited?
And apparently this watermark will be introduced in the output of all Anthropic models, including Claude Code. But this introduces a massive conflict of goals here - when you write code, you’re not free to arrange words and letters any way you like (and need to encode your watermark), because that will affect the functionality of the code you produce.
Quite possibly I’m oversimplifying this, and Anthropic has much more intricate and subtle techniques at its disposal than I am thinking of, but I can’t quite imagine how this is supposed to work.
I’m trying to find this in something official, but I see a lot of places saying that code, amongst a few other things, is exempt from this.
Marking of AI Content (Article 50(2)). Providers of AI systems (expressly including general purpose AI systems) that generate synthetic audio, image, video or text content must ensure that: (i) outputs are marked in a machine readable format that is detectable as AI-generated or manipulated; and (ii) their technical solutions are effective, interoperable, robust and reliable taking into account various factors, in each case, excluding AI systems that perform an assistive function for standard editing or do not substantially alter the input data, and AI systems used to detect, prevent, investigate or prosecute criminal offences. The Guidelines note that Article 50 applies to AI systems generally, not only general purpose AI (as noted in the AI Act itself), so narrow, single-purpose tools such as voice cloners are potentially caught. However, the Guidelines also confirm that certain AI systems and outputs fall outside Article 50(2), such as systems that merely reproduce or rearrange existing content (e.g. music playlists or recommender systems). The Guidelines also identify specific categories that fall outside of scope: (a) content that falls outside Article 50(2) altogether on the basis that there is no risk of deception, manipulation or to ensure integrity and trust in the information, and that is not perceptible to people or not intended to be interpreted, verified or acted upon by natural persons, including short outputs (e.g. single words, image captions and data labels), source code, machine-to-machine communications, and interim outputs in closed loop environments in industrial and product development workflows; and (b) what amounts to an assistive function for standard editing or does not substantially alter the input, such as grammar and spell-checking, format conversion and minor cropping.
My understanding is that it has to do with frequencies of certain letters and letter combinations. If the letter “z” is expected to appear every 500 characters on average but Claude output has it show up every 350, that’s a tell. Do that across several different markers and you have a pretty reliable watermark.
But they can’t actually tell users what those watermarks are or it does become trivial to hunt for them and smooth them away.
(Note: I’ll use the term “words” instead of “tokens” since it is easier to read and understand, but I’m happy to break it down into tokens if desired.)
When an LLM is generating output, it spits out one word at a time. Typically it doesn’t pick the most likely word each time. Instead it randomly chooses a word from a set of likely choices; perhaps from the top 40 choices. This can help make the text more natural.
Text watermarking can add an additional constraint to that random selection that is unique enough to detect later.
For example, it randomly chooses from the top 40 words, except:
for the first word, it’s never allowed to choose the 23rd choice,
for the second word, it’s never allowed to choose the 7th choice,
for the third word, it’s never allowed to choose the 38th choice
etc.
With enough text, you could analyze the text and determine the likelihood it was generated with that constraint.
You can use an algorithm to choose the next word to exclude. You can exclude multiple words. You can repeat the pattern so it’s possible to detect snippets from generated text. You can have rules to not apply the constraint – perhaps to whitespace.
I haven’t read any papers on this so I honestly don’t know what the actual techniques are, but you can see how it can be quite flexible.
One last thing: since an LLM is already randomly selecting a word from a list, adding a small constraint won’t alter the text that much. For any given word, you might have gotten the 40th “best” choice before anyway.
This is an interesting take on it, but doesn’t this mean that in order to check whether a text has an Anthropic watermark, you need to know the exact probability distributions for the model that produced the text? I would imagine that these distributions are a function of the weights in the model, which are proprietary and differ from model to model.
You don’t need to know how their marking scheme works. They’ll provide tools for detecting it.
Detecting Claude’s marks
We’re also working to enable users and other third parties to detect Claude’s embedded watermarks and provenance metadata. Detection checks whether a piece of text or a file carries a supported Claude mark. If a supported mark is found, it indicates that the content may have been processed by Claude.
We’ll share details on detection mechanisms in forthcoming technical documentation.
Yep, you need to know which model, what the distributions are, etc. As @Joey_P said, the vendors will provide tools. Same again for other types of output (images, video, etc.)
So it’s not exactly a watermark like on a printed page or even the secret code on printouts and photocopies.
It’s not adding hidden, non-printable characters though. It’s biasing the actual output content in a detectable way.
It’s not fool-proof. You can edit the text and make the detection tool think it’s only 75% likely to be generated. Vendors are pretty up-front about that.
However, it may be possible to make the editing onorus enough that the human editor may as well write it in the first place.
Like everything, it will be an arms-race, but it’s still helpful.
I’m extremely doubtful that anything is possible beyond detecting high-level patterns of AI writing, such as suggested in this Wikipedia article.
Nothing embedded at a lower level could possibly survive the right kind of filtering. Worst case, OCR or even text-to-speech and then speech-to-text. What “watermark” is going to survive that? It has to be based on a statistical analysis of writing style.
One other thing that I’m wondering about: If the watermark relies on characteristic probability distributions in the LLMs, doesn’t this mean that verifying whether a text contains the watermark (or not) requires you to pass it through the LLM that generated it, to see if the new output matches the patterns detected in the text you’re testing? Essentially, every time you verify a text you’d consume as much in terms of computing resources as the initial generation did. Wouldn’t this increase the strain on server farms many times over?
Yeah, when the checksum (or CRC) are added to the data. It’s basically adding an extra bit of signal to data. Also maybe similar to a parity bit.
I was thinking of it as similar to song identification where the detector just needs a snippet of the song and is resilient to mono/stereo, bit rates, etc.
In those cases the song wasn’t created with the extra signal so the IDs have to be looked up in a database.
However, if you can include the signal you could make it generate the same ID for every song.
This is the same idea as your checksum / CRC analogy where every chunk of data ends up with the same magic value.
Interestingly I just read the SynthID (Google’s image watermark) paper for work. They train a network to inject a series of bits. Images have a lot more to work with so in some sense it’s an easier problem, but there are a lot more augmentations (resizing, crops, stretching, instagram filters, screenshots, etc) that they want to be robust to while text is basically just overwriting or cutting portions. Google also does the watermarking after generation, not as a part of the generation.
I suspect Anthropic would do the opposite approach and jointly optimize watermark optimization and human preference during the RL finetuning portion of the model, but that’s merely a hunch since a post generation altering of the text is way more destructive than altering a handful of pixels in a 1024x1024 image.