With that said, it feels like we’re lab rats in an experiment AI companies are performing. They need us to help them figure out how to make their product work better. I think it shakes out personal use stays affordable (you’re the product kind of thing) and work use gets/stays expensive.
Also, one driver of what we pay is quite literally paying AI companies to delete/ not use my data. AI is currently very expensive to ensure your data/use remains private. It’s much much cheaper to use without that protection. I wouldn’t think most people using for personal reasons care, though. On the other hand, companies using it for work can be legally required to care.
Same as the dot-com bubble; by investment and venture capital providing the illusion of actual revenue.
That’s the catch. Cost has to be justified by benefit. What will be the real cost of using AI to enhance your business. How reliable is it to be left unsupervised?
If AI replaces your purchasing department, with autonomy, how long before it gives preferences in buying to companies that have paid the AI provider? (“Sponsored products”) How many other such examples will there be? Pay the AI provider to get your resume considered by its clients? And so on…
The tipping point arrives when AI companies try to pass on the true cost of running AI to its customers. The slide to the abyss happens when they need the customers to pay the R&D costs also. Meanwhile, the singularity moment it seems has been receding monthly.
The investment in fiber and servers in 2000 was worth it in the long run for the world as a whole, but in the short run it wasn’t worth it for lots of the companies making it.
The business plans of each of these companies are no doubt making optimistic assumptions, because in part because they fear being left behind if they cut back on investments. Totally understandable, but they can’t all be right.
AI is not going away, but there will be lots of corpses of losing companies on the way to it.
I completely agree. I don’t think its possible to monopolize profits off of AI because there is so much competition from other companies, combined with all the open source models emerging from China.
So the companies themselves will suffer because they can’t make a return on their investments (Nobody is going to pay $2000/month for a drop in agent if an open source model does the same thing for much cheaper), but society as a whole will advance from it.
Agreed. This comes up in every AI thread. I think I even gave some numbers in the *The Business of AI" thread. It’s only grown more true with agents.
Training and running inference on the model are similar in compute. It then becomes a question of how many training epochs/examples vs. aggregate number of tokens read/written.
From the 30k foot view, you’re not going to spend more effort building a tool than using it.
In addition to what others have said, Google caches AI results. The AI results are not necessarily created each time it does a search with AI.
But the question is, how often are they going to be building the tool? If people are going to ask about current events the tools will need to train on them nearly continuously, like Google has to traverse the web continuously. I don’t know how incremental the training can be. And when the AI starts writing new LLMs, you’ll need to train the new ones which will be coming out with increasing frequency.
I don’t have a concrete answer to who, but I’ll give some examples of how. If you buy a subscription to an AI service - for example, Claude Pro for $20/mo, and you use your maximum alotment of compute time available with that subscription, they give you $600 worth of compute (according to what they’d charge you if you did the same thing through the API). Now, the API price itself may or may not be profitable, at cost, or below cost - I don’t have that data, I’m not sure anyone does. But even if you assume the API costs are profitable with a 100% margin, you’re still talking about $300 worth of compute for $20. If you get the more professional oriented max 20x plan it costs $200 a month and gives you something on the order of $6000 in compute.
Famously, openAI was letting users create free videos for their Sora video model that cost them something like $12 per video and it was losing many millions per day (this is the only thing that comes remotely close to the aforementioned $10 per response mentioned above - I wonder if that’s what they were thinking of). They ran it for months losing over a billion dollars with no plans to make a profit at all.
Grok used to let users generate dozens to hundreds of videos a day off of a $30 subscription and even a bunch for few users, which could cost them thousands a month to a determined enough user. They’ve cracked down on it now exactly because they’re finally making a move towards stopping gushing massive amounts of capital for subsidized generation.
Copilot doesn’t even charge you anything for most of what it does - there’s a subscription but it’s really just for Office and Office integration. Microsoft is otherwise giving away pretty much everything for free.
I think the reason they do that is to compete for market share and mind share. They want to be the ones on top, the brand people use, so they’re willing to give away their product far below cost to get there.
Now - lots of business use is directly through the API interface rather than one of these subscriptions or web interfaces. I don’t know whether those are designed to make a profit or take a loss or not. But the companies themselves are losing massive amounts of money. No one is anywhere near profitability. Often times this can pay off in the long term if they win, but I do wonder if they can make a subsidized bet so large that they can never turn it profitable or have to massively downgrade their services and/or increase costs.
This is exactly where you want the economics to be, though, because it allows for consumption-based billing, which allows for better alignment of cost to production, as well as price, as well as more reliable demand modeling. And this shift is already underway. The consumer cost will more accurately reflects what it takes to provide the service, which reduces the risk of companies making big gambles on how to fund training.
The cost of training will nosedive anyway due to distillation. Distillation is going to happen (already happening in China) regardless of how the big frontier model vendors feel about it. If there’s any truth behind the fearmongering of AI datacenter overbuild, this is probably the likeliest cause.
Truth be told, nobody really has any idea what the economics of AI will look like in 5 years, and 99% of the general public has no idea what it really looks like right now.
What they’re doing is encouraging people to learn how to use it. People don’t adopt when rate limits are breathing down their neck about how to be more token-efficient with tech that they’re not really even sold on. It was really quite a brilliant stroke what we did to get developers hooked on coding tools, encouraging them to “tokenmax” and “just use it”, and then once they get locked into it, switch it to consumption-based billing. Of course they screamed about the unfairness of it all, but this is the cost of prioritizing shortcuts (AI) and hype (also AI) over craftsmanship (learning how to fucking program and use resources efficiently). These particular sheep deserve to be sheared, and they’ll be subsidizing prices for everyone else for some time to come.
Model distillation requires access to the larger teacher model. This could be done with llama4 because it was shared publicly, but the other frontier models are guarded behind an API wall.
You can use a frontier model API to generate datasets or evaluate results, but it is against TOS and not cheap. This isn’t distillation, but maybe people are using that term.
I expect frontier models will continue to grow in complexity as well as fork specializations, but the costs will continue to shrink compared to inference; eventually being negligible. Otherwise the tool wouldn’t be useful.
Bathed in his currents of liquid helium, self-contained, immobile, vastly well informed by every mechanical sense: Shalmaneser.
Every now and again there passes through his circuits a pulse which carries the cybernetic equivalent of the phrase, Christ, what an imagination I’ve got.
Distillation will happen as long as the economics favor it, ToS or no. The delivery model will need to evolve to adapt to that reality.
As distillation drives down the cost of training useful new models, then companies can afford to charge less for inference.
It’s possible that distillation will reduce the compute demand of training enough to reduce the need for datacenters currently being built. But I doubt that, and anyway, these are increasingly being funded by debt (bonds) rather than stock, which allows the risk to be spread and traded intelligently, making an actual bubble-type meltdown less likely.
Again, while there’s a lot of hype and optimism around this, under the covers this whole phenomenon is structurally a lot different from the first dotcom boom, both technically, financially, and economically. Some companies will go bust, sure. That’s going to happen in a goldrush. This is much different from supposing that the industry as a whole will collapse, or that the tech is a flash in the pan. The former is unlikely, the latter simply wrong.
My suspicion is that a very large number of data centers yet to be built and running are scams. AI is hot now. Sucker people to buy in. Getting the capital inflow may be similar to junk bonds. Or shell company stocks that sound good. But never actually intend to build. Just go bust, disappear and keep a big share of the investors money. On top of that is a likely gigantic pile of bets on these things. CDOs and such, shorting, whatever all the side bets and obligations to payout one way or another, if a data center does or does not get built. Or ever turns a profit.
Markets have turned into complex casinos. So many ways to siphon capital from useful purpose, to pure gambling and theft. The AI boom is a wonderful cash cow that will often pay off better to some people if a particular center never even lays a foundation.
Ultimately the plebes will pay, even as they lose jobs to what AI does actually function. But the plebes are fast going broke. I hope pension funds are being very careful about data center investment. If you are hoping to get a bump to your 401K by riding this roller coaster, be careful.
Apparently part of the bond market problems the last few days - treasury bond long term rates breaking 5%, and Bessent promising to buy back more bonds - was because AI bonds are taking away some of the money typically used to buy T-bills.
Anthropic used the term “distillation” in the complaint and that creates some overlap and confusion with the technique, “model distillation”.
Anthropic complains that competitors used their service to create their own models, but this doesn’t mean that the competitors used less compute to train. It means that the competitors used Anthropic (and likely others) to create augmented datasets (e.g. censored data) as well as evaluating training results. If anything, this uses more datacenter compute.
I’d dispute that. There is more to model production than just the compute cost of final training runs. At least 70% or the cost is in other stuff like discovery, performance tuning, speculative work that has to be thrown out. All of this is compute that the distiller doesn’t need to pay for. In Anthropic’s case, this is insult to injury because the distiller was also free riding on their compute to generate their augmented layer.
In any case, there are good reasons why the cost of AI inference should go down. It would be surprising if it didn’t, because this is the typical trajectory of most goods and services, it’s costly in the beginning until the practices and knowledge diffuse to the wild, where innovation is always working to reduce costs. Distillation of course isn’t the only factor but it’s a significant one.