# ChatGTP et al, the creators don't know how it works? **URL:** \ **Category:** Factual Questions\ **Tags:** ai\ **Created:** [March 28, 2023, 6:28am UTC](https://boards.straightdope.com/t/chatgtp-et-al-the-creators-dont-know-how-it-works/981761 "2023-03-28T06:28:21Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![Crane](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/crane/32/3495_2.png) [@Crane](https://boards.straightdope.com/u/Crane)\ **Post date:** [March 29, 2023, 1:48pm UTC](https://boards.straightdope.com/t/chatgtp-et-al-the-creators-dont-know-how-it-works/981761/21 "2023-03-29T13:48:40Z")
I doubt the ‘we don’t really understand how it works’ pitch is the one he presents to the backers when he’s asking for another billion dollars. Its just good theater for a PR session. ---
**Author:** ![jjakucyk](https://avatars.discourse-cdn.com/v4/letter/j/3d9bf3/32.png) [@jjakucyk](https://boards.straightdope.com/u/jjakucyk)\ **Post date:** [March 29, 2023, 3:01pm UTC](https://boards.straightdope.com/t/chatgtp-et-al-the-creators-dont-know-how-it-works/981761/22 "2023-03-29T15:01:15Z")
> [@Crane](#): > > I doubt the ‘we don’t really understand how it works’ pitch is the one he presents to the backers when he’s asking for another billion dollars. Sounds a lot like “we need to pass this bill so we can know what’s in it”. “Give us teh moniez so we can figure out what this AI can do.” ---
**Author:** ![Crane](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/crane/32/3495_2.png) [@Crane](https://boards.straightdope.com/u/Crane)\ **Post date:** [March 29, 2023, 3:02pm UTC](https://boards.straightdope.com/t/chatgtp-et-al-the-creators-dont-know-how-it-works/981761/23 "2023-03-29T15:02:29Z")
‘Push the button and see what happens’ ---
**Author:** ![Babale](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/babale/32/15666_2.png) [@Babale](https://boards.straightdope.com/u/Babale)\ **Post date:** [March 29, 2023, 3:20pm UTC](https://boards.straightdope.com/t/chatgtp-et-al-the-creators-dont-know-how-it-works/981761/24 "2023-03-29T15:20:48Z")
> [@Crane](#): > > Its just good theater for a PR session. No, it’s a feature that’s pretty much inherent to neural networks. > [@jjakucyk](#): > > Sounds a lot like “we need to pass this bill so we can know what’s in it”. I think the CGP Gray video you linked above pretty clearly explains the difference. ---
**Author:** ![Crane](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/crane/32/3495_2.png) [@Crane](https://boards.straightdope.com/u/Crane)\ **Post date:** [March 29, 2023, 3:32pm UTC](https://boards.straightdope.com/t/chatgtp-et-al-the-creators-dont-know-how-it-works/981761/25 "2023-03-29T15:32:06Z")
> [@Babale](#): > > No, it’s a feature that’s pretty much inherent to neural networks. No it’s not. It would be difficult to size a Sigmoid or program back propagation if you didn’t know how it works. ---
**Author:** ![Babale](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/babale/32/15666_2.png) [@Babale](https://boards.straightdope.com/u/Babale)\ **Post date:** [March 29, 2023, 3:35pm UTC](https://boards.straightdope.com/t/chatgtp-et-al-the-creators-dont-know-how-it-works/981761/26 "2023-03-29T15:35:51Z")
> [@Crane](#): > > No it’s not. It would be difficult to size a Sigmoid or program back propagation if you didn’t know how it works I’ll refer you back to this post by @engineer_comp_geek that explains what you are missing: > [@engineer\_comp\_geek](#): > > If you simulate a handful of neurons, the overall simulation is simple enough that you can fairly easily figure out what everything is doing. But the simulation also won’t do anything useful or interesting. > > If you increase the number of neurons, the interconnections and interactions become so complex that the whole system is far beyond human understanding. You can easily focus on what an individual simulated neuron is doing, or what is happening along a particular simulated path, but the sheer number of interactions makes it impossible for a human to fully understand what the system is doing. > > And it is beyond that point that interesting things start happening. The neural network starts to behave like a real brain. The simulated signals along simulated paths start to look a lot like actual biological brain waves. The neural network can also be “trained”, by feeding it particular stimuli, after which it will respond to how it has been trained. But I suppose I was not specific enough: > [@Babale](#): > > No, it’s a feature that’s pretty much inherent to neural networks. It’s inherent to sufficiently large neural networks, not neural networks in principal. ---
**Author:** ![Crane](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/crane/32/3495_2.png) [@Crane](https://boards.straightdope.com/u/Crane)\ **Post date:** [March 29, 2023, 3:43pm UTC](https://boards.straightdope.com/t/chatgtp-et-al-the-creators-dont-know-how-it-works/981761/27 "2023-03-29T15:43:43Z")
> [@Babale](#): > > I’ll refer you back to this post by @engineer_comp_geek that explains what you are missing I’m not missing anything. The brilliant engineers who designed GPT controlled the parameters of every node in the system. They specified the activation window and the back propagation of every node in the net. It was not anything as ethereal as the simulation of neurons. They are not related. Its an engineering problem with a solution. The ‘gosh we don’t know how it works’ is just good theater. ---
**Author:** ![wolfpup](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/wolfpup/32/10618_2.png) [@wolfpup](https://boards.straightdope.com/u/wolfpup)\ **Post date:** [March 29, 2023, 3:51pm UTC](https://boards.straightdope.com/t/chatgtp-et-al-the-creators-dont-know-how-it-works/981761/28 "2023-03-29T15:51:57Z")
> [@Francis\_Vaughan](#): > > Depends what you mean by “know how it works”. > Of course the creators know how the code operates and how it was trained and how it operates. > What they don’t understand is how any specific input/response occurs. That lives in the emergent character of the trained system. This is the key. Of course we know how it was built since we built it. We know how it was trained since we trained it. But there is not remotely any practical way to know how or why ChatGPT produced any specific response. There is no way to predict ahead of time how well it will work. Perhaps most intringuingly, there is no way to predict the spontaneous emergence of new problem-solving skills on which it had never been explicitly trained, which can appear quite suddenly purely as a function of scale. ---
**Author:** ![Crane](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/crane/32/3495_2.png) [@Crane](https://boards.straightdope.com/u/Crane)\ **Post date:** [March 29, 2023, 3:56pm UTC](https://boards.straightdope.com/t/chatgtp-et-al-the-creators-dont-know-how-it-works/981761/29 "2023-03-29T15:56:31Z")
> [@wolfpup](#): > > Perhaps most intringuingly, there is no way to predict the spontaneous emergence of new problem-solving skills on which it had never been explicitly trained, which can appear quite suddenly purely as a function of scale. Then why did they invest a billion dollars to train it? Just to see what it would do? ---
**Author:** ![wolfpup](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/wolfpup/32/10618_2.png) [@wolfpup](https://boards.straightdope.com/u/wolfpup)\ **Post date:** [March 29, 2023, 4:17pm UTC](https://boards.straightdope.com/t/chatgtp-et-al-the-creators-dont-know-how-it-works/981761/30 "2023-03-29T16:17:59Z")
> [@Crane](#): > > Then why did they invest a billion dollars to train it? Just to see what it would do? In a nutshell, that’s pretty close to the truth. I mean, this is a research project, and that’s pretty much the nature of research. The justification for research funding is generally in the form of some evidence that it will be productive, but there are never guarantees. The Large Hadron Collider cost billions, and the justification was basically that cool things will likely happen at unprecedented levels of particle energy, and thus new discoveries will be made. The justification for the OpenAI research is that intelligent machines capable of natural language conversation and reasoning have many commercial applications. ---
**Author:** ![Crane](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/crane/32/3495_2.png) [@Crane](https://boards.straightdope.com/u/Crane)\ **Post date:** [March 29, 2023, 4:22pm UTC](https://boards.straightdope.com/t/chatgtp-et-al-the-creators-dont-know-how-it-works/981761/31 "2023-03-29T16:22:25Z")
> [@wolfpup](#): > > The justification for the OpenAI research is that intelligent machines capable of natural language conversation and reasoning have many commercial applications. Yes, and they had to convince somebody that they had a reasonable probability of success. Since this is a commercial product, I don’t believe it was funded by the American Altruists Association. ---
**Author:** ![Babale](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/babale/32/15666_2.png) [@Babale](https://boards.straightdope.com/u/Babale)\ **Post date:** [March 29, 2023, 4:29pm UTC](https://boards.straightdope.com/t/chatgtp-et-al-the-creators-dont-know-how-it-works/981761/32 "2023-03-29T16:29:13Z")
> [@Crane](#): > > Yes, and they had to convince somebody that they had a reasonable probability of success. Since this is a commercial product, I don’t believe it was funded by the American Altruists Association This is a strawman argument. You’re not responding at all to what @wolfpup is saying. To demolish your strawman, OpenAI had done work on previous models (ChatGPT 2, for example) and showed that increasing the size of the neural network and amount of training time it received resulted in corresponding increases to the performance of the language model. Graphing the performance vs resources available to the model, they were able to show that the largest models they could produce were still improving rapidly with size, with no sign of diminishing returns. They used this as evidence that increasing the size of the model further would result in even better results, and thus secured funding for ChatGPT 3. ---
**Author:** ![Crane](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/crane/32/3495_2.png) [@Crane](https://boards.straightdope.com/u/Crane)\ **Post date:** [March 29, 2023, 4:30pm UTC](https://boards.straightdope.com/t/chatgtp-et-al-the-creators-dont-know-how-it-works/981761/33 "2023-03-29T16:30:48Z")
> [@Babale](#): > > They used this as evidence that increasing the size of the model further would result in even better results, and thus secured funding for ChatGPT 3. Thanks for making my argument. They knew what they were doing. ---
**Author:** ![wolfpup](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/wolfpup/32/10618_2.png) [@wolfpup](https://boards.straightdope.com/u/wolfpup)\ **Post date:** [March 29, 2023, 4:34pm UTC](https://boards.straightdope.com/t/chatgtp-et-al-the-creators-dont-know-how-it-works/981761/34 "2023-03-29T16:34:35Z")
My understanding is that the probability of success was largely premised on smaller, simpler models that showed the language model concept to be promising. But if you’re trying to suggest that the researchers or funding providers knew exactly how ChatGPT would perform in a massively scaled-up version, that’s absolutely not the case. Even today the research continues and new surprises continue to appear. > [@Crane](#): > > They knew what they were doing. That’s not remotely the same thing as knowing how the assembled and trained system would perform, or even why it responds the way it does. For all anyone knew, it could well have been a total failure, proving only that scale alone doesn’t significantly improve cognitive skill. Instead it proved the opposite. ---
**Author:** ![Crane](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/crane/32/3495_2.png) [@Crane](https://boards.straightdope.com/u/Crane)\ **Post date:** [March 29, 2023, 4:40pm UTC](https://boards.straightdope.com/t/chatgtp-et-al-the-creators-dont-know-how-it-works/981761/35 "2023-03-29T16:40:57Z")
> [@wolfpup](#): > > My understanding is that the probability of success was largely premised on smaller, simpler models Well yes, that’s the nature of development. I’m not sure what the argument is here. You are familiar with R&D. I am familiar with R&D funding. We both know that there is a large degree of risk in development and total success is a pleasant surprise. We also know that there is an element of theater in presenting to the media. ---
**Author:** ![wolfpup](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/wolfpup/32/10618_2.png) [@wolfpup](https://boards.straightdope.com/u/wolfpup)\ **Post date:** [March 29, 2023, 5:09pm UTC](https://boards.straightdope.com/t/chatgtp-et-al-the-creators-dont-know-how-it-works/981761/36 "2023-03-29T17:09:14Z")
> [@Crane](#): > > We also know that there is an element of theater in presenting to the media. Sometimes that happens, but sensationalism is usually the creation of the media itself, not the researchers. I haven’t seen anything like that coming out of OpenAI. That there are mysteries and surprises in ChatGPT like the emergence of novel skills isn’t “theater”, it’s fact. ---
**Author:** ![Babale](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/babale/32/15666_2.png) [@Babale](https://boards.straightdope.com/u/Babale)\ **Post date:** [March 29, 2023, 5:25pm UTC](https://boards.straightdope.com/t/chatgtp-et-al-the-creators-dont-know-how-it-works/981761/37 "2023-03-29T17:25:29Z")
> [@Crane](#): > > Thanks for making my argument. They knew what they were doing. That… is not the point I was arguing against, whatsoever. It’s like the difference between understanding the mechanics of evolution, and understanding what individual genes do, versus being able to program with DNA the way we do with computer code. Or having an understanding of both neurons and psychology versus understanding the complete mechanics of the human brain. We understand the mechanics of neural networks, at an individual neuron level. We also have some ideas of how they evolve over time when trained, in a general sense. That doesn’t mean that we “understand how they work”, any more than we understand how conciousness arises in the brain or how to read DNA like a book. ---
**Author:** ![Mangetout](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mangetout/32/19_2.png) [@Mangetout](https://boards.straightdope.com/u/Mangetout)\ **Post date:** [March 29, 2023, 7:24pm UTC](https://boards.straightdope.com/t/chatgtp-et-al-the-creators-dont-know-how-it-works/981761/38 "2023-03-29T19:24:19Z")
> [@Babale](#): > > We understand the mechanics of neural networks, at an individual neuron level. We also have some ideas of how they evolve over time when trained, in a general sense. Absolutely right - the individual nodes are very easy to understand. The thing that’s difficult to understand is _what they’re actually doing together_ when ‘together’ means 175 billion of them are interacting with each other. ---
**Author:** ![Chronos](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/chronos/32/134_2.png) [@Chronos](https://boards.straightdope.com/u/Chronos)\ **Post date:** [March 29, 2023, 10:01pm UTC](https://boards.straightdope.com/t/chatgtp-et-al-the-creators-dont-know-how-it-works/981761/39 "2023-03-29T22:01:21Z")
The engineers at OpenAI had strong reason to believe that interesting new capabilities would emerge when they went from ChatGPT 2 to ChatGPT 3, and from 3 to 3.5, and from 3.5 to 4, because new capabilities had emerged every previous time when they had made the network or training dataset larger. But they didn’t even know what those new capabilities would be. And they certainly didn’t know how those capabilities would work. > [@Crane](#): > > I’m not missing anything. The brilliant engineers who designed GPT controlled the parameters of every node in the system. They specified the activation window and the back propagation of every node in the net. It was not anything as ethereal as the simulation of neurons. They are not related. This is quite simply factually incorrect, and I don’t know why you keep saying it. There are way too many parameters for humans to even be capable of having specified every one. The system set its own values for those parameters, in the process of digesting the training data set. ---
**Author:** ![Crane](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/crane/32/3495_2.png) [@Crane](https://boards.straightdope.com/u/Crane)\ **Post date:** [March 30, 2023, 12:09am UTC](https://boards.straightdope.com/t/chatgtp-et-al-the-creators-dont-know-how-it-works/981761/40 "2023-03-30T00:09:39Z")
> [@Chronos](#): > > This is quite simply factually incorrect, and I don’t know why you keep saying it. Read what you quoted. Are you not familiar with software programming techniques? [Previous page](https://boards.straightdope.com/t/chatgtp-et-al-the-creators-dont-know-how-it-works/981761.md?page=1) [Next page](https://boards.straightdope.com/t/chatgtp-et-al-the-creators-dont-know-how-it-works/981761.md?page=3)