If it’s meant to comply with transparency obligations, wouldn’t they have to publish the algorithm that’s used to imprint the watermark or to detect it? A proprietary algorithm that works only with Anthropic’s own tools would hardly be transparent.
But then it would be trivial to remove it, like telling a bank robber which strap of money has the ink pack in it. Plus, agree or disagree, people aren’t going to want a visible watermark (image or text) in the output.
In some circumstances, the transparency act does require a provider/deployer to disclose to the user, in ‘natural language’ that it’s AI. However, in general, for text/audio/images/video, the act only requires that it’s marked in a ‘machine readable’ format and detectable.
Providers of AI systems (including general-purpose AI systems) that generate synthetic audio, image, video, or text must ensure that outputs are both marked in a machine-readable format and detectable as AI-generated. This is a technical obligation aimed at provenance: enabling detection tools to verify whether content is AI-generated.
Except you made that part up.
They still appear to be working on the final wording of the transparency act, but from what I can tell, the tools should be free in most cases.
The Code, going beyond the express marking obligations of Article 50(2), requires providers to make available a mechanism to detect AI-generated or manipulated content. The mechanism must in principle be free of charge…
The California version of the rule includes a line stating the detection tools must be free of charge and accessible to the public.
This bill, the California AI Transparency Act, would, among other things, require a covered provider, as defined, to make available an artificial intelligence (AI) detection tool at no cost to the user that meets certain criteria, including that the AI detection tool is publicly accessible.
The result will probably do no more than comply with the words of that regulation.
Off the top of my head, I can think of the following ways to fool the detection:
- Short text snippets. Not enough words to embed enough “watermark signal”.
- Combine text from multiple AI generators. A tool to do so is trivial to create (thanks, Claude!)
- Randomly replace words in the text with others that have similar meanings.
- Use an older pre-watermakr or self-hosted AI to rewrite the entire text.
What if I do not pay money to Anthropic, but generate my own “AI-generated text” using my own custom software? I know people who have done that (for research purposes, not as part of some corporate scheme). Is that supposed to be illegal?
I also know plenty of sites, including this one, that have an explicit “no AI slop” rule, or, at least, do not allow you to pass automatically generated text or images off as your own. On the other hand, I do know of at least one or two experimental books that (quite openly) used procedural generation as part of the process, and that was the point.
I know at least that Google’s image watermark is supposed to be robust to tampering of the sort that is analogous to #3 in your list. It’s absolutely not perfectly robust, but Anthropic probably is attempting to make the method at least somewhat resistant to those attacks.
Trying to stay in FQ here…
If the main goal of the legislation is to capture the majority of brute-force LLM generated text, this is a good step; for example, some entity that want to blast a wall of fake, generated news stories.
- Short text snippets: Not critical to catch, a human could just write them
- Combine text from multiple AI generators: Depends on how long each snippet is and how many there are. The detector can still find it’s own signal if there is enough of it.
- Randomly replace words in the text with others that have similar meanings: This will obscure the signal and lower the detector’s score, but not eliminate it.
- Use an older pre-watermark or self-hosted AI to rewrite the entire text: Or create their own LLM. As I said, it’s an arms-race.
The perfect is the enemy of the good here. Someone can always create their own LLM to generate glurge. The point is to create some barrier of entry.
The point of the legislation is to require vendors to provide a free tool for their products. If you create your own and want to sell it, then you might have to comply. (I haven’t read the EU law; someone else can weigh in if they have.)
For research and hobbyist LLMs, there are the black-box detectors that @wolfpup discussed.
It’s an arms-race and will need multiple prongs of attack.
We’re all worried about the impact of AI glurge and deep fakes. This is one prong to address it.
Editor: Claude, write me a thesis on the subject of galaxy merging.
Editor: GPT04, take this thesis and rewrite it in a slightly different style.
Editor: Command-R+: take this thesis and rewrite it in a more technical style.
Etc etc.
Editor: thanks all. I’ll just do a little light editing myself, paraphrasing, perhaps including some spelling/grammar errors, to show I am human.
I’m fairly sure multiple pases through multiple different LLMs would eventually be undetectable, unless(in my example) Command-R+ has steganography of its own, AND the tool used to detect the LLM content is trained to use Command-R’s steganography “tells”.
… and later…
Editor: generic LLM: rewrite this thesis, ensuring that no steganographic “tells” from Claude. GPT04, or Command-R+ remain…
That just generates word salad (which I have seen people generate by hand and nevertheless be awarded Ph.Ds, which says something about the people who went ahead and signed it). OTOH, if you have a good thesis, why not use spelling/grammar-checking tools?
This is a separate, although important, problem. One person can create one document through a variety of tools that can defeat this type of detector. This would require a different detector. As an aside: I’m not sure if a student could defend a thesis created this way.
The watermarking is useful to detect when one person is posting 100K fake news stories, testimonials, reviews to Twitter, Reddit, or Amazon.
Sure, they can spend a lot of time and money to run each post through a pipeline of tools and if that obfuscates it enough to break the detector, then we’re back where we are today.
Likewise a state actor could create their own LLM; same as where we are today.
The point isn’t to use one method to catch 100% of the cases. It’s to require vendors to provide a tool that can be used to catch 95% of the cases
Why wouldn’t the watermarks from the last one you ran it through be detectable? It might erase the watermarks from the previous ones, but it would still leave it’s own.
But that would only work if it knows what those tells are. And, if it knows, wouldn’t you just be able to ask it what they are so you could edit them out yourself?
I ran across this video which does a good job explaining how LLM watermarking works. He explains three schemes, “hard watermarking” which is simple to understand and robust but degrades the quality of the output text, “soft watermarking” which is less robust (watermarked text might sometimes be detected as not watermarked and vice versa) but produces higher quality output, and Google’s Synth-ID (tournament sampling), which produces even higher quality output than soft watermarking. Anthropic says that they will be using Synth-ID for Claude watermarking.
One thing that occurs to me is that this scheme adds a lot of extra computation to each generated word in the LLM output. This will make LLM even more computationally expensive than it already is, and use more resources.
That’s a great video, thanks for sharing it.
The Synth-id technique doesn’t add as much overhead as it seems. The Supplementary information provides a vectorized version in Appendix E and compares the computational overhead in Appendix F.
From the main paper:
a latency increase of only 0.57%. In comparison, Gumbel sampling increases latency by 0.26% and Soft Red List by 0.28%. Furthermore, the computational complexity of all three watermarks remains constant even as the LLM grows
And from Appendix F:
Tournament sampling generally has higher computational complexity than Gumbel or Soft Red List sampling; however if |V | is large compared to Nm+1 then Tournament sampling (the tournament-based Methods Algorithm 2 implementation) may have lower complexity. Nonetheless, in the context of the computational complexity of generating text from a large LLM, these differences are in practice negligible (see Section 3 in main paper).
The paper is “Scalable watermarking for identifying large language model outputs” if anyone wants to read more.
Thanks for those snippets @CaveMike. Out of curiosity (no pressure if you don’t want to share) but what’s your background?
My degree is in Computer Engineering and I spent most of my career in software, but for the last 10-11 years, I worked on running Machine Leaning (AI) on mobile phones – voice recognition, camera, and most recently LLMs; I retired last month.
I was never a researcher – my math/stats skills are not strong enough. I would have to put in a lot of time to understand how the vectorizing is equivalent to the paired-tournaments in the synth-id paper.
Naming choices for variables and functions maybe, or whitespace and other stuff that is transparent to the compiler/interpreter. I suppose it could also be done by choosing one of many different possible methods to perform the same small action.
Another issue is - unless the watermark technique is common, then as implied in earlier posts, you would have to run any piece through a number of different detection tools to determine if it is indeed AI generated. …and hope the LLM used adheres to the requirement to watermark.
As for techniques that require the selection based on odds of predicted words or other similar characteristics - would that not change frequently as the LLM is continuously trained on more and more input? For example the odds that “Hormuz” follows “Strait” has probably increased dramatically for any LLM training on daily news feeds. Whereas “bab al-Mandab” was probably not in the top 40 straits until a month ago…
And if it is dependent on patterns, and can be altered with word substitutions and such - then you are back to the question “Is it AI?” then “Maybe, it seems so, but there’s no definitive answer”.
Thanks for sharing that video. I kinda sorta get it, but it’s way above my paygrade and there’s no way I could intentionally defeat, or even understand how to defeat, that kind of watermarking.
@CaveMike Thanks for your input. Good stuff.
Before I read this thread, I thought text “watermarking” was not going to be helpful for anything beyond say paragraphs of Claude produced AI text, not altered/lightly altered, and then Claude or Claude approved detector confirming the watermark. Is that still about right? Or is it more robust than that?
That’s correct. There are two classes of detectors (one made by an AI vendor and one not):
-
An AI vendor adds a watermark to their output and provides a detector that works with their AI. Short outputs and heavily modified outputs reduce the effectiveness of these detectors.
-
A third-party creates a detector that uses heuristics to detect AI output and does not use watermarking at all. These types of detectors can potentially identify multiple vendor’s AIs, but the overall effectiveness is lower than #1.
Would it be possible to have the AI just remember what it’s written? It might be prohibitive to remember it all verbatim, but maybe just some sort of hash?