I’ll have to challenge that one. Yes, they reported $80B in losses the first half of the year, but $72B of that was recognition of debt held that is (eventually) convertible to stock, not operating loss. Their operating loss was around $8.7B; not great, but not catastrophic.
Ironically, the revelation that their revenue was flattening at around $16B, and their large future capital commitments, seem to have shrunk their valuation; if that’s true, they’ll claw back some of that $72B.
I mainly use AI for image gen and do that locally. So if there was some big paywall, it would be annoying for a few days but I’d get over it.
Realistically, the vast majority of AI compute goes towards enterprise use – businesses, government and research; I’ve seen numbers suggesting that ‘creative’ use of AI combined (image, video & music) is less than 10% of the total and that includes corporate/professional use, not just hobbyists at home. I suspect that a major realignment in costs would be raising up the price for the big users and the free ChatGPT, Gemini, Claude, etc will remain free and both reduce how much you can use them at a time and stick with lower models. Burning through token on cutting edge models will be the stuff that’s suddenly costing 4x more.
One of the big Chinese companies, DeepSeek, just released some details on their inference system: https://finance.yahoo.com/technology/ai/articles/deepseek-open-sources-deepinfer-software-140650863.html
TLDR their prices are about 16x cheaper than comparable Western models, for slightly worse benchmark performance, while maintaining a similar profit margin.
a few random thoughts/facts:
I like the “california gold-rush” analogy, especially the part that says - the only ones that made good money there where those selling the shovels … I feel the same for AI … a lot lose leaves will shake out
cost per token today are 8% of what they were a year ago - that will continue to go down - esp. once we get deeper into recursive learning (AI generating AI models)
AI will go the way of Microsoft: their business is businesses … not joe doe and his laptop … so my guess is there will be expensive corp. tiers (once the value-chain of AI is clear, evident and reckoned by businesses - and the prices there will be high, oriented on the value they create (take the $300k SW-engeneer example upthread)) … and than there will be more limited household uses w/ a couple of interactions per day, where we all end up paying with our eyeballs ->Ads and all that junk!!!
the key Q is: how soon and how capable will be models that can be run on a home-computer? b/c that will be the dividing line in the future (no need for data-centers, etc…) … so the strategic seismic division will be AI run on prof. infrastructure by Anthropic et al … and the lighter models that run on our celulars.
AI will become a commodity pretty soon
yes, a few of my points contratict each other …
thoughts? challenges?
I’m excited about this part, but also kinda worried that the ecosystem would move in the wrong direction, becoming more locked down over time.
There’s already some on-device AI, especially on Google phones. Right now it’s just used for the simple and lightweight stuff, like Google Translate (remember when that used to be a hard problem?), scam detection, improving your selfies, and proofreading your emails.
But today’s phones (and most consumer computers) don’t really have the hardware horsepower to run the bigger, better models that do the complex reasoning and better image and video generation and such. It’s also quite cost-prohibitive, in the present day, to put such capabilities into consumer devices that might only need them a few minutes out of the day, as opposed to data centers that can share them between thousands of users all day. That said, though, the major hardware makers are starting to add more and more consumer AI chips into their home user devices, and maybe economies of scale will eventually make those affordable too, if RAM ever comes back down again.
The software is also kinda lacking and scattered, with Nvidia’s chips mostly working the best on their own software and Apple and Google each making their own frameworks. The phones also limit how other apps can interact with the onboard AI… I tried to build a SMS political spam filter for Anyone else? The political junk texts blitz? (which is a trivially easy thing to do now), but the Android operating system doesn’t let other apps receive RCS messages anymore (Google’s own proprietary version of iMessage). To their credit, at least you can swap out Gemini for Claude as the default assistant, for now. But generally speaking, the phones’ security models depend on not trusting their own user, whereas computers and LLMs typically assume the user has full permissions to modify whatever they want (which is what makes the agentic harnesses so powerful on computers). So unless they’re willing to relax some of those restrictions to give onboard AIs more access, they’re always gonna be a bit handicapped. Maybe some of the open-source Linux phones will start giving more direct access to AI chips.
I largely agree with you otherwise.
But of course, if we could fully predict the AI future, we’d both be trillionaires by now 
Was your issue limited to only being able to access RCS? Or also the NPU? I thought you’ve had luck using the Pixel TPU through Google’s AI Core API?
On Qualcomm and MTK devices, it should be possible to access their NPUs from non-system apps, but, cynically speaking, I think it won’t be for long. It’s easy to drain the battery or interfere with system AI if any old app can use the NPU. They don’t have robust govenors for system resources.
I think local AI will end up being a niche of a niche as open models dry up and frontier models widen the quality gap. For most use-cases it’s hard to make the argument for an answer that is good enough.
My assumption is it’ll be maybe ~5 years before AI is advanced enough that a highly competent chatbot can be run on a laptop’s hardware. However that is a small part of what AI does. Other areas like materials science, medical/biological research, math/physics research, etc will still require highly advanced data centers.
I’m not sure who will get rich and who won’t from all this. Like electricity made the economy as a whole get better, but I think the companies that made the hardware for electricity did well.
The companies that made hardware for AI will make a lot of money. But I have no idea which companies will emerge from this with AI programs that they can lease for hundreds of billion a year. If any.
Both. The AI Core API won’t let you access the TPU in the background (for the reasons you stated, probably).
But for something like filtering out political spam, you don’t really need a full-blown LLM, just a lightweight classifier. Claude’s suggestion was to use notifications & accessibility permissions to read the messages notifications and filter/hide them accordingly, then use either accessibility to manually mark them as spam, or masquerade as a RCS web client via libgm (like a paired Google Messages). Neither is a great choice.
I’m thinking maybe I just change my real phone number, port the old number to VOIP, and never use the real number for anything aside from 2FA texts from now on. That would altogether kill RCS, but seems like a small price to pay.
But really, I’d prefer Google just add this to Messages directly… it’d be like an afternoon of work for them. The political backlash from both parties would be swift and furious, though — probably the same reason CAN-SPAM doesn’t apply to political emails.
For many on-device use cases, though, a local model would be more than good enough! Things like better face & pet & sext detection in photos, automated daily briefings or dating profile pre-filtering, email triage and auto-replies, calendar automation, dealing with passwords and passkeys and 2FAs, bank account monitoring, restaurant searches and food ordering, navigating to the next place on today’s itinerary, product browsing & filtering, playlist recommendations, voice transcriptions, automatic captioning and translation… if I could, I’d just replace 90% of my phone apps with one giant AI button that could do all of that for me. We don’t really need an app store or even UIs anymore if the AI could just talk to a bunch of APIs & MCPs instead.
I’ve built a lot of that stuff on my computer already and they’re quite light to run. It’s just a pain getting them to work on phones because of the security model, and soon, the Android sideload & signing requirements. Doing it properly would probably require a complete fork of the Android Open Source Project and lying to apps about root and device integrity attestation, and also a hacked Google Play Services.
I don’t think Google would want that because it’s a threat to everything they do — search, ads, AI, phones, etc. They risk becoming a dumb pipe for someone else’s AI and ecosystem.
Apple, on the other hand, might actually be able to pull it off…
Ads are the traditional way that online services make use of “free” “customers”, but I don’t think that that’ll be the case for AI companies. We have something much more valuable to offer them: Training for their models.
There’s plenty of argument for that, if the “good enough” solution is enough cheaper. There are all sorts of applications where amateurs use a weaker version that’s “good enough”, even while the pros pay top dollar for a professional version.
Since we’re aiming for the optimal dystopia here… those don’t have to be mutually exclusive! “Which marketing techniques work the best inside a LLM” can be both ads AND training feedback loops. Yay!
Nah, I think that they’ll make it look ad-free to us, at least at that level. And then use what they learn about us to tailor-generate the ads that they present us with in other contexts.
Though I also wouldn’t be surprised if a chatbot told us what its favorite food was, or what brand of skin cream it personally uses.
At which level? (The link I edited in shows ChatGPT already showing ads in the chatbot)
You know… if these were actually useful ads, tailored to my specific interests and clearly labeled as sponsored… I might actually not mind them as much.
That was always the unfulfilled promise of personalized marketing, I guess. We just didn’t have the ability to actually execute it effectively before LLMs.
An honest, tailored recommendation chosen from amongst multiple options, some of which are sponsored… might actually be helpful, as long as it doesn’t become pay-to-rank like it did for Google’s ads (sigh).
Usually when people complain about targeted advertising, what they’re actually complaining about is poorly-targeted advertising. I, too, can appreciate properly-targeted ads.
But that’s not what I’m worried about here. I don’t mind being shown ads for things that are useful to math teachers (but not the general public), for instance, because I am a math teacher. What I’m worried about is advertisers learning our minds better than we know them ourselves, and advertising to us using our own subconscious biases that we don’t even know exist.
I’m not ever so sure this will remain true. Google seems to be busy pivoting their search to AI. By the time the bubble bursts, there might not be a plain Google search to go back to.
I don’t think it exists now.
I’ve only used ai directly once. My husband walked me through using his Claude account to update a mailman mailing list. It worked pretty well. I’ve also spent about half an hour playing with it a year ago, and that experience was terrible. It said a lot of dumb and useless things.
But every “Google” search i do these days is actually AI. It’s really annoying, because the quality of results is much lower than it was before AI. So much do that i bought Kagi, which may also use AI, but it’s pretty good and links to relevant sources.
Back in the day before LLMs were everywhere, we called what Google was doing then “AI”.
Nowadays, it’s usually giving an LLM-generated answer at the start of its search results, but I don’t think that the LLM is responsible for the list of hits after that. I think those are still generated with the same engine it used to use (with several years worth of incremental changes, of course).
I’ve spent the last few years advocating for LLMs on mobile phones. My team had success shipping 1B and 3B fine-tuned models for specific tasks, but let me argue the other side. When I look at how mobile apps have evolved, most of them are basically web clients. It’s so much easier to consolidate the complexity in the cloud; it’s HW and OS agnostic with instant SW updates.
Yes, you can fine-tune a 4B model to be good enough for one task in one language, but it takes data and training, is chipset-dependent, and event then it works on a subset of devices. Alternatively, you can send an HTTP request to a 4T or 4Q model and all it takes it prompt engineering.
I think AI apps will take the same route as regular apps – a web client to a server that can hide all of the orchestration complexity, load balance to the right sized model, and deploy improvements as fast as they are implemented.
(Apologies if this is too far off-topic.)
I think a “router” in front of the model(s) makes sense, but I’d prefer the first layer to be on-device.
Case in point: Google recently forced all Android phones to use their Gemini AI instead of the old Google Assistant. As a result, “Hey Google, set alarm for 9:45” became both far slower (like 6-7 seconds instead of 1-2) and a bit stupider (it will sometimes ask “Sure. Is that 9:45 am or pm?” where the old one will just choose the closest one). The old design kept some things local, including part of the speech transcription, I believe, and only “asked for help” from the cloud if needed. The new design (and what you’re proposing, I think?) inverses that, adding latency and removing reliability (like if you’re offline, even the local speech transcription and basic on-device functions stop working).
I’d argue — trying to segue back to this main thread, lol — part of the pricing & cost woes right now is because we don’t have this sort of “start simple, then ask an expert only if needed” architecture by default yet, and that most of the chatbots and harnesses will either stick with whatever model you set them to, or prefer to spawn weaker models below them.
The inverse, starting with weaker gateway/router models first and only routing to higher advisors when necessary, would’ve saved a lot of tokens for everyone! Claude supports this, but only as an option: Advisor tool - Claude Platform Docs
In fact, thinking back, I’m pretty sure we went through a similar phase with the first- and second- generation (pre-LLM) Google Assistants, Alexas, Siris, etc. It was all in the cloud at first until later optimizations brought some functionalities down onto devices, saving the companies some cost and also making them more predictable for the consumers.
I think their early research showed that most people just ended up using the voice assistants for super basic tasks (like setting alarms), not complex “help me find the perfect product” type things. Of course, that was far less monetizable for the big companies, so voice assistants stagnated for a decade until LLMs brought them back.
Now a new generation of management and developers are in charge, and I guess we have to re-learn those same lessons again!
(Sorry, too late to edit)
Back about the costs topic, just wanted to share this relevant snippet from my Claude Code usages:
This is currently on the $200/mo Claude plan (I upgraded for this month to finish a big project). If it were metered pay-as-you-go (instead of the subsidized subscription), it would’ve cost $1391.87 so far. The subscription is so heavily subsidized that some people juggle multiple subscriptions (and there are shady resellers who, as a service, will feed your prompt into their subs) — all just so you don’t have to end up paying the full API prices.
Of that, $1241.81 was me talking to the “primary” model, Opus. It “outsourced” some simpler work to the cheaper models, Haiku, which did most of the web searches and summarizations I needed (across hundreds of scientific papers): for a total of $3.28.
The research cost almost nothing. The main cost was Opus writing some 60,000 lines of code in a day and creating a bunch of tests to support and verify all that.
All that is to say — the costs are all highly fluid, and for what most people use AI for (lookups and summaries), it’s far far cheaper than what the frontier labs are offering. The expensive part is the “reasoning” and the training to support that sort of really hard work. And that’s mostly targeted towards professionals, researchers, hobbyists, defense, and cybersecurity, not the home user.