If it’s meant to comply with transparency obligations, wouldn’t they have to publish the algorithm that’s used to imprint the watermark or to detect it? A proprietary algorithm that works only with Anthropic’s own tools would hardly be transparent.
But then it would be trivial to remove it, like telling a bank robber which strap of money has the ink pack in it. Plus, agree or disagree, people aren’t going to want a visible watermark (image or text) in the output.
In some circumstances, the transparency act does require a provider/deployer to disclose to the user, in ‘natural language’ that it’s AI. However, in general, for text/audio/images/video, the act only requires that it’s marked in a ‘machine readable’ format and detectable.
Providers of AI systems (including general-purpose AI systems) that generate synthetic audio, image, video, or text must ensure that outputs are both marked in a machine-readable format and detectable as AI-generated. This is a technical obligation aimed at provenance: enabling detection tools to verify whether content is AI-generated.
Except you made that part up.
They still appear to be working on the final wording of the transparency act, but from what I can tell, the tools should be free in most cases.
The Code, going beyond the express marking obligations of Article 50(2), requires providers to make available a mechanism to detect AI-generated or manipulated content. The mechanism must in principle be free of charge…
The California version of the rule includes a line stating the detection tools must be free of charge and accessible to the public.
This bill, the California AI Transparency Act, would, among other things, require a covered provider, as defined, to make available an artificial intelligence (AI) detection tool at no cost to the user that meets certain criteria, including that the AI detection tool is publicly accessible.
The result will probably do no more than comply with the words of that regulation.
Off the top of my head, I can think of the following ways to fool the detection:
- Short text snippets. Not enough words to embed enough “watermark signal”.
- Combine text from multiple AI generators. A tool to do so is trivial to create (thanks, Claude!)
- Randomly replace words in the text with others that have similar meanings.
- Use an older pre-watermakr or self-hosted AI to rewrite the entire text.
What if I do not pay money to Anthropic, but generate my own “AI-generated text” using my own custom software? I know people who have done that (for research purposes, not as part of some corporate scheme). Is that supposed to be illegal?
I also know plenty of sites, including this one, that have an explicit “no AI slop” rule, or, at least, do not allow you to pass automatically generated text or images off as your own. On the other hand, I do know of at least one or two experimental books that (quite openly) used procedural generation as part of the process, and that was the point.
I know at least that Google’s image watermark is supposed to be robust to tampering of the sort that is analogous to #3 in your list. It’s absolutely not perfectly robust, but Anthropic probably is attempting to make the method at least somewhat resistant to those attacks.
Trying to stay in FQ here…
If the main goal of the legislation is to capture the majority of brute-force LLM generated text, this is a good step; for example, some entity that want to blast a wall of fake, generated news stories.
- Short text snippets: Not critical to catch, a human could just write them
- Combine text from multiple AI generators: Depends on how long each snippet is and how many there are. The detector can still find it’s own signal if there is enough of it.
- Randomly replace words in the text with others that have similar meanings: This will obscure the signal and lower the detector’s score, but not eliminate it.
- Use an older pre-watermark or self-hosted AI to rewrite the entire text: Or create their own LLM. As I said, it’s an arms-race.
The perfect is the enemy of the good here. Someone can always create their own LLM to generate glurge. The point is to create some barrier of entry.
The point of the legislation is to require vendors to provide a free tool for their products. If you create your own and want to sell it, then you might have to comply. (I haven’t read the EU law; someone else can weigh in if they have.)
For research and hobbyist LLMs, there are the black-box detectors that @wolfpup discussed.
It’s an arms-race and will need multiple prongs of attack.
We’re all worried about the impact of AI glurge and deep fakes. This is one prong to address it.
Editor: Claude, write me a thesis on the subject of galaxy merging.
Editor: GPT04, take this thesis and rewrite it in a slightly different style.
Editor: Command-R+: take this thesis and rewrite it in a more technical style.
Etc etc.
Editor: thanks all. I’ll just do a little light editing myself, paraphrasing, perhaps including some spelling/grammar errors, to show I am human.
I’m fairly sure multiple pases through multiple different LLMs would eventually be undetectable, unless(in my example) Command-R+ has steganography of its own, AND the tool used to detect the LLM content is trained to use Command-R’s steganography “tells”.
… and later…
Editor: generic LLM: rewrite this thesis, ensuring that no steganographic “tells” from Claude. GPT04, or Command-R+ remain…
That just generates word salad (which I have seen people generate by hand and nevertheless be awarded Ph.Ds, which says something about the people who went ahead and signed it). OTOH, if you have a good thesis, why not use spelling/grammar-checking tools?
This is a separate, although important, problem. One person can create one document through a variety of tools that can defeat this type of detector. This would require a different detector. As an aside: I’m not sure if a student could defend a thesis created this way.
The watermarking is useful to detect when one person is posting 100K fake news stories, testimonials, reviews to Twitter, Reddit, or Amazon.
Sure, they can spend a lot of time and money to run each post through a pipeline of tools and if that obfuscates it enough to break the detector, then we’re back where we are today.
Likewise a state actor could create their own LLM; same as where we are today.
The point isn’t to use one method to catch 100% of the cases. It’s to require vendors to provide a tool that can be used to catch 95% of the cases
Why wouldn’t the watermarks from the last one you ran it through be detectable? It might erase the watermarks from the previous ones, but it would still leave it’s own.
But that would only work if it knows what those tells are. And, if it knows, wouldn’t you just be able to ask it what they are so you could edit them out yourself?
I ran across this video which does a good job explaining how LLM watermarking works. He explains three schemes, “hard watermarking” which is simple to understand and robust but degrades the quality of the output text, “soft watermarking” which is less robust (watermarked text might sometimes be detected as not watermarked and vice versa) but produces higher quality output, and Google’s Synth-ID (tournament sampling), which produces even higher quality output than soft watermarking. Anthropic says that they will be using Synth-ID for Claude watermarking.
One thing that occurs to me is that this scheme adds a lot of extra computation to each generated word in the LLM output. This will make LLM even more computationally expensive than it already is, and use more resources.
That’s a great video, thanks for sharing it.
The Synth-id technique doesn’t add as much overhead as it seems. The Supplementary information provides a vectorized version in Appendix E and compares the computational overhead in Appendix F.
From the main paper:
a latency increase of only 0.57%. In comparison, Gumbel sampling increases latency by 0.26% and Soft Red List by 0.28%. Furthermore, the computational complexity of all three watermarks remains constant even as the LLM grows
And from Appendix F:
Tournament sampling generally has higher computational complexity than Gumbel or Soft Red List sampling; however if |V | is large compared to Nm+1 then Tournament sampling (the tournament-based Methods Algorithm 2 implementation) may have lower complexity. Nonetheless, in the context of the computational complexity of generating text from a large LLM, these differences are in practice negligible (see Section 3 in main paper).
The paper is “Scalable watermarking for identifying large language model outputs” if anyone wants to read more.
Thanks for those snippets @CaveMike. Out of curiosity (no pressure if you don’t want to share) but what’s your background?
My degree is in Computer Engineering and I spent most of my career in software, but for the last 10-11 years, I worked on running Machine Leaning (AI) on mobile phones – voice recognition, camera, and most recently LLMs; I retired last month.
I was never a researcher – my math/stats skills are not strong enough. I would have to put in a lot of time to understand how the vectorizing is equivalent to the paired-tournaments in the synth-id paper.