Somewhat ironically, ChatGPT is actually pretty good at explaining how it itself works: How LLMs work, explained in 3 Lord of the Rings haikus by ChatGPT
In that chat, I asked it to explain how LLMs work, and clarified with several follow-ups. Then I asked it to use a special mode, read a bunch of scholarly papers, and then analyze them vs its earlier explanations to summarize the major points with citations. It continually re-prompted itself in a “chain of thought” to read different sources and add them to its context. Then I asked about it hallucinations.
This conversation helped me understand how we go from “token prediction like your phone’s autocomplete” to “how is it able to actually understand a prompt, break down its question into tokens, and extract sufficient meaning in order to answer it”.
Of course, the sad thing is… I don’t know if that convo was entirely hallucinated. It seemed plausible enough, and the terms seem to check out, but we’re quickly veering towards an all-AI reality bubble where it’s impossible to verify anything anymore…
If you prefer human responses, we also recently had a similar thread about this, with a lot of discussion and additional info: chatGPT is a fucking liar
There were some explainer videos in post #38. Those visualizations are helpful for seeing a (simplified) representation of the token prediction process, especially at 3:03 in this “Transformers (how LLMs work) explained visually” video. That’s part of a whole series and several talks (by what seems to be a person). Who can tell anymore…