Very pertinent questions and very interesting answer. Well done!
Let’s examine the release of Stable Diffusion, since it is on github. They explicitly remind you near the top that
The weights are research artifacts and should be treated as such.
and, in any case,
is precisely what they are doing (and claim they are doing). At the same time, they assert their published model is copyrighted, patented, and subject to a license, stating that they want to control how it is used. In particular,
commercial use is fine with them, but they warn you, among other things, that
You are accountable for the Output you generate and its subsequent uses. No use of the output can contravene any provision as stated in the License.
and that you are not allowed to use the models they contribute “[i]n any way that violates any applicable national, federal, state, local or international law or regulation.”
I am not aware (though that does not mean much) of anybody trying to sue their asses off for allowing their research scripts to surf the Internet for publicly available collections of images to look at, or for subsequently publishing the resulting models. Perhaps merely looking at a (published!) image does not violate any moral rights—if someone then used the model to generate something prejudicial based on that artist’s work it would be (at least according to the creators) the user who violated the artist’s rights, as well as the terms of the license.
There’s information stored about what images look like. Get enough of that information, and you’re storing an image. Is there enough of that information, for any individual image? The people who say “no” are also the people who say that they have no idea what the AI is actually storing.
I have no knowledge of how things are stored in the model but I do know, from experience, that getting the AI to extract a known artwork is very rarely successful and restricted to culturally hyperpenetrative examples like the Mona Lisa.
I entered An accurate reproduction of 1881 “Two Sisters (On the Terrace)” by Pierre-Auguste Renoir into Stable Diffusion v1.5 – which isn’t an obscure piece of art and certainly one I’d expect to be part of training a model on what Renoir’s works looked like and the results were…
…my career of French Impressionist art forgery might have to wait a while longer. Setting aside the Wish-dot-com quality of the brushwork, it didn’t even really know what the painting looked like aside from a very vague assumption (only one sister in all the examples even has a red hat). These aren’t shitty copies of Two Sisters (On The Terrace), this is “Vaguely Renoir-esque impressionism involving a couple of young women on a terrace”. I also tried to create a parody of Two Sisters using pop culture figures thinking maybe it would accurately pose and background the new characters but nope, not even close.
Sort of a side note to the above, some friends and I have been hitting the MidJourney bot hard, trying to ferret out which artists it’s trained in and recognizes as a style. The method is fairly rudimentary: ask the MJ bot for “Art by [name]” and see if it comes back with something besides the default MJ style. So far we have a list of 630 artists and probably tons more to go as there’s schools of art we haven’t really scraped yet but, interestingly, we probably have at best a 20% success rate. Most artists just don’t come up as a known style or modifier. And that’s using artists well enough known to get an entry on various art sites like WikiArt and self-selecting for ones with enough works listed to be good candidates. A friend tried using popular names off ArtStation and was getting below 5%. From what I understand, the MidJourney model is a good size larger than the Stable Diffusion model and I think that shows up when I try to use the same prompts in SD.
This isn’t to say that no one has a point about artists being in the model without permission or consent but I’ve also seen it framed as though these models just hoovered up every piece of art on the internet or through Google Image Search and that definitely isn’t the case. If you’re some random artist with a DeviantArt page selling D&D character art commissions, I’d guess that you’re almost certainly not in the model.
I think it’s clearly not just taking a couple of selected photos of broccoli and composting them into a new image with some filters applied. At least not Midjourney Ai. Each image is rendered from scratch, guided by prompts as far as style and composition. The data set trains the Ai on what properties generally makeup “broccoli” and properties of a particular style of art if prompted, but it is creating new images from scratch. I had Midjourney generate some images of broccoli, linked below:
Can’t help myself. Have to show off my first top 5% entry for lighthouse.
Nice. I’m back on a cold streak. I think I got a top 20% for my Surrealist entry. That’s the last one for me.
(actually I just checked - I got a top 5% for my stairs #nonhumblebrag)
I’m not sure how long it’s been there, but they’ve added multiple prompts, that you can weight, on Stable Diffusion at NightCafe … so no more, “too many words” issues. I did just find out that “intimate” is a banned word for some reason. I wanted an Intimate Portrait Composition. It took me forever to figure out that was the problem word.
I have fun with the “color portrait” preset on Night Cafe.
Here is one result for “decopunk cthulhu gothic horror nccp” (upper left) and some images evolved from it.
Here are some ID-type photos evolved from a base image and using gremlin, yokai, or yurei with the nccp preset.
So what is “NCCP”? Googling it just reveals a variety of nonprofit organizations.
Short for “Night Cafe color portrait”, the preset I referred to at the beginning of the post. The full preset is “Close-up portrait, color portrait, Linkedin profile picture, professional portrait photography by Martin Schoeller, by Mark Mann, by Steve McCurry, bokeh, studio lighting, canon lens, shot on dslr, 64 megapixels, sharp focus”.
Interesting how Stable 2.1 came up with so many distinct variations from one simple prompt.
Doing some comic characters with the Night Cafe color portrait preset. “Gwenpool” tends to give a woman wearing pink with shortish pink or blonde hair. “Wolverine Laura X-23” tends to give a woman with long black hair. I was impressed with how the system managed to understand those traits for characters that haven’t appeared in live action (well, technically X-23 was in Logan, but as a fairly young child) but then I realized–it must be drawing from lots of photographs of cosplayers.
(I also was trying to get pictures of a cyclops–as in the one-eyed monster-- a while back but instead was getting a guy in glasses or goggles. It took me a minute to realize that the system was trying to give me Scott Summers.)
“Scary Christmas” turned out kind of cool.
One of my colleagues at work was trying to create a Christmas card design for the school (our mascot is the Jaguars) by prompting for “Jaguar in a Christmas tree”. The car that was shining its headlights out of the tree was, in fact, recognizable as a Jaguar.
One of the prompt-only Gwenpool images (upper left) was pretty good so I duplicated it a bunch. It is interested how varied the background becomes. Also, the center one looks a heck of a lot like Scarlett Johansson.
This is an image generated with decopunk Chthulhu gothic horror NCcp on SD 2.1. I mirrored it to get a complete headshot and duplicated it a bit.
None of the duplicates quite live up to the sea cucumbery/penisy appendages on the original.
Somebody “tipped” one of my posts for the first time. Not only did I get the 1 credit, but also 10 bonus credits and a “tipee” badge.
I’m feeling pretty confident in my challenges entry for tomorrow, “Eyes.”
We’ll see. I’ve horribly misjudged what will get upvoted before.