I was re-watching The Empire Strikes Back and was thinking it would be cool to have a first-person game focusing on the Battle of Hoth. We could participate from the POV of an Alliance soldier in the snow trenches, or a snow speeder pilot, or even one of the AT-AT drivers.
While I doubt the answer is ‘no’ at the moment, might we reach the point where an AI could design and create such a video game for humans to play?
When I tried looking this up I got a lot of information about how AI is supposedly used to govern game characters’ behavior. Also, for all I know such a Star Wars game may already exist. But I’m interested in the AI potential to create something along these lines.
No, not “trivially”. Interactive game design—even for relatively simple games—is an interplay of objectives, challenges, interfaces, the player experience (i.e. the art, music, story, et cetera), and the mechanics which support all of this. Making an engaging game is very difficult and not something that can be just reduced algorithmically or easily done just by emulating art and interfaces of an existing game. Generative AI could certainly be used to create art and graphics (albeit it will need to be better than current systems to produce something that is thematically consistent) and could be used for extemporaneous NPC interactions or dynamically controlling the environment that the player is interacting with, but actually making a playable game that isn’t just a repetitive set of similar challenges is far beyond what current generative ‘AI’ can do, notwithstanding the problems that software generating AI has with building more than small elements of code without major problems.
That’s the easiest part. You don’t go to Dall-E and say “Give me video game assets” for something like this. You train a narrow model on appropriate Star Wars stuff and whatever artistic style you want.
Which will get you something that looks more or less like the style of Star Wars, but it won’t prevent having a TIE fighter with five panels sticking out in random locations, or an R2 droid with a second smaller R2 droid sticking out of its side, or a robot general wielding four lightsab…never mind, probably not worse than anything you see in the prequels.
I love AI, but it’s nowhere near that level yet. It can’t even produce a playable game of Tetris or Pac-Man for me. Like you can tell what game it was going for, but that’s about it, even after multiple iterations of trying to fix it.
You can control the generation of certain aberrant features through specific prompting (i.e. preventing the creation of stub fingers by specifying ‘beautiful hands’, or in this example, telling it to provide “A TIE fighter with parallel panels”) and there are post-generation image filters that will identify and alter or remove certain types of artifacts, but by default these systems still produce a lot of image errors. I have had to routinely look through AI generated images (and text) and can usually find artifacts of a generated image within seconds, especially if it is in natural light (generative AI still has real problems with naturalistic shadows and shading) or fine features that are not geometrically consistent or proportional, i.e. require perspective or differential focus. I don’t doubt that it will get better, and at some point you’ll be able to prompt for a very detailed image and get something close to what you were asking for and that only someone experienced in AI image forensics could readily distinguish from a photographic image, but it isn’t there yet.
Good thing we’re talking about video game assets then. “I knew it was an AI because I looked at the natural light pixels” is a long way away from “a TIE fighter with five panels sticking out in random locations”
Also, it’s not about prompting or post-production filters, it’s about models. I can train a specialized Lora in a half hour from my home PC if I want (in fact, I did just that today). I could potentially train a whole narrow model (using an existing model as a base) if I felt really dedicated. Someone trying to make an AI that can create consistent video game assets could definitely do so and it would be the obvious first step to trying to make a game-producing AI.
I took the OP as asking for speculation about future capabilities. And one day, it will be like a Holodeck or Matrix spike to the skull, trivial.
But is there any question that a current AI could invent a simple game both novel and playable?
Based on what I’ve seen from other prompts for music, images, chat ideas, etc, a talented commercial AI user could probably could set something up that makes use of known engagement algorithms and enticement features plus a library of existing games of varying simplicity: maze, poker, solitaire, chess, tic-tac-toe, crosswords, (memory) matching, number puzzles. Add fictional lore, characters, narratives, details and the game storyline could easily hold your attention for a few minutes while it generates a few images, soon animation, then some playable graphics.
Casino and gambling device devs were probably using AI for behavior study and pattern characterization since the earliest days.
By the way, there was a Star Wars Battle of Hoth chapter of a game called Shadows of the Empire for Nintendo64 that an old roommate of mine had. The snowspeeder parts were really cool, you had to use the harpoon and loop around a couple times to wrap the walker’s legs.
There is a very good book called Blood Sweat and Pixels which gives the behind-the-scenes dirt on the creation of a dozen or so well-known games, from Stardew Valley to Dragon Age Inquisition.
The book makes clear that making a video game is hard. Really, really hard. It’s not just about conceiving the premise, making some graphics, and coding the engine that allows the latter to embody the former. Sure, a lot of the challenges the designers and developers encounter in the book are about schedules, budgets, organizational politics, and other human concerns an AI could presumably disregard.
But — one of the repeated obstacles the game creators encounter is that the games they’ve spent so much time building are found to be not fun. The game loop is shallow or repetitive, or the mechanics are frustrating and nonintuitive, or the objectives aren’t clear, and on and on. This is why game development is so difficult and time-consuming: it’s very, very rare to get everything right on the first try. Everything must be tested and re-tested not just for bugs but for simple quality of gameplay. Everything is iterated, and re-iterated, and re-re-iterated, until the game is actually, in a meaningful sense, playable.
I would bet any amount of money in the world that an AI-generated game would be literally unplayable. You’d get a minute or so into it, tops, and know that it’s garbage.
You could, of course, use AI tools to make that iteration more efficient, and game developers are already doing this. But the feedback loop of trying the game and then deciding what doesn’t work and needs to be improved is fundamentally human and cannot be short-circuited by AI.
Again, I highly recommend the book. It’s extremely eye opening.
Simple ones, yes, maybe… barely. They can barely code a coherent web app right now (which are several orders of magnitude simpler than a video game). They can do simpler games with some help and persistence, but those are a far, far cry from the AAA productions like Battlefront 2 (or Shadows of the Empire, back in the day).
It’s one thing to have AI write something like Minesweeper or Snake. But modern big-budget video games are huge affairs requiring many years of work from hundreds of people and costing millions of dollar in labor and equipment. A video game’s production is a bigger affair than a Hollywood blockbuster. Imagine the complexity of the entire MS Office suite, combined with a full movie, then times that by a hundred. Maybe by a thousand, for something like Baldur’s Gate 3. It’s not something ChatGPT can trivially replace with just a well-worded prompt.
I have no doubt it’s an eventuality… but we’re just not quite there yet.
It’s advancing very quickly. They get better and better at generating 3D objects, scenes, and code every month. And certainly it’s already replacing people at least part of the time, piecemeal — that’s why there’s currently an ongoing SAG-AFTRA voice and motion capture actor strike, for example.
It might not even take a full decade to be able to replace all the roles and make an entire game by itself. It’s just not quite there today.
(And yes, in the meantime, there are many fantastic human-written Star Wars games that feature the Battle of Hoth, both first-person shooters and other genres like real-time strategy.)
I think what’s being discussed here is the notion of AI creating the code and graphical assets etc for a standalone game that would run as a conventional program on a computer; that’s one way to do it.
Another way to do it would be to train an algorithm to create the experience of the game, based on the inputs of the controls - so there’s no game program actually running, but rather, an AI is daydreaming what the game would look like if you were playing it, in real time.
Something along these lines has already been done already by training an algorithm on hours of gameplay data (from Doom), resulting in a playable AI - there is no copy of Doom running - it’s the same idea as AI-generated video, except the video is the thing that is happening next in the game and the prompt is the buttons you are pressing on the gamepad.
Of course that’s just a case of replicating the gameplay of an existing game, but I don’t see why it wouldn’t be possible in principle to have a model that is capable of generating something never before seen, based on initial prompts.
