The best modern OCR systems are very good at converting image text to text strings.
Very strange indeed, and not good news. I was under the impression that Altman was the intellectual center of OpenAI. It also doesn’t help that Musk has been stealing OpenAI employees (and also those from Google and Microsoft working on AI) to support his own stupid “Grok” project.
I know you’re just using common language here, but you can’t “steal” employees. The corporations would love for you to think about it that way, which we know because they’ve (Apple, Google, etc.) lost huge lawsuits in Silicon Valley where they colluded to stop “stealing” each other’s employees, suppressing salaries as a result.
Valuable employees move when they think they can get a better offer. Thinking of it as theft means taking the side of the megacorp, not the employees negotiating for their own interest.
Which is lower, whatever score they got with those questions, or the 0/0 they got on the version of the test with those questions excluded?
Yes, if by “all the time” you mean “one per year”. And that’s exactly what I’ve been saying they should do: Only take the test made after their training’s cut-off date.
At this point, you’re just claiming that they’re lying about their contamination checks. It’s possible. Hey, maybe that’s what got Altman fired. But it seems like a stretch to me.
You can search their GPT-4 whitepaper for “contamination” if you want details on the steps they took and the differences between contaminated and uncontaminated versions.
Even if their contamination checks were imperfect, you would still expect to see more of a difference than they saw if it was just memorization.
I’m not deep in the know or anything, but this is the opposite of my impression. I thought Ilya Sutskever/Andrej Karpathy/ Greg Brockman were/are the main brains there with Altman being management and PR. Brockman apparently just quit too. Odd.
Kara Swisher thinks we’ll see more departures:
It’s a little hard to reconcile this with the OpenAI statement that “he was not consistently candid,” which is just a euphemism for him lying to the board. It’s possible though that this is just cover for the ideological differences between Altman and the board.
It’s certainly true that Altman has stretched the mission statement of OpenAI beyond any reasonable level. They’re just a profit-maximizing company at this point, and not even close to being aligned with their original non-profit goals. If one takes a view sympathetic to the board, they probably did have to fire Altman to get back to the original goals (even approximately).
It seems like a lot less of a stretch than assuming that they somehow, without trying, managed to get a data set that included teaching calculus, while somehow omitting questions from the most-used version of the most-used test for teaching calculus.
It also, for that matter, seems a lot less of a stretch than assuming that an entity that still lacks the capabilities of a calculator, somehow managed to ace a test many of whose questions require the use of a calculator.
It didn’t ace AP Calculus. It scored 4/5. And a lot of questions that appear to need a calculator don’t. For instance, from an AP Calculus sample test:
Seems like it might need a calculator. Except that I can see immediately that the answer is A or B, because f’(0) is trivial to evaluate as -1. And I can plug in \pi \over 2 as well, with an answer of e-1, which is positive. So I know it went from negative to positive in the first quadrant and thus the answer must be A.
What GPT-4 is capable of in this respect I couldn’t say, but it certainly doesn’t need to evaluate \sin{x} or e^x to solve problems like this. Not to mention that approximate solutions are often good enough to pick out one answer out of 4.
Greg Brockman claiming this was basically a coup orchestrated by Ilya Sutskever.
So Altman was fired and Brockman was removed from the board, and then quit. This can’t be good for AI, regardless of the behind-the-scenes reasons. An industry that depends so much on massive computing resources and intensive research can’t afford to be fragmented like this.
.
It’s been crazy few days with all the drama–Altman agreeing to return if the board resigns, the board making motions in this direction but calling it off, a final backstab with them hiring a new CEO… but I think we’re at the end:
I did not predict this at all, but in a way it makes perfect sense, and is potentially a huge coup for Microsoft. They get a solid AI team and the team gets access to Microsoft’s significant resources. And Microsoft can potentially wind down their relationship with OpenAI eventually, once they develop their own internal systems.
OpenAI seems like the big loser here, but maybe they wanted this outcome. Their new CEO, Emmett Shear, wants to slow down development by 5-10x:
OpenAI staff threatening to quit en masse.
And now the next phase is here, with Google Gemini being released today. It looks like another step above even GPT-4, and it’s fully multi-modal:
It also does math natively, unlike GPT-4
They have three models. Pro, Ultra, and Nano. Amazingly, the ‘nano’ one runs locally on Pixel phones and will bring AI to a bunch of features. Pro is available today in Bard, and Ultra is coming later.
The performance is amazing. The thing can talk in real time and outputs images and sentences just about instantly.
If anyone’s curious about what LLMs “look like” on the inside, check out this visualizer:
Even trivial LLMs like nano-gpt (with 85k params, as compared to 175B params for GPT-4) are quite complicated. The visualizer also has a step-by-step tutorial on the dataflow, but if that’s too much info, the 3D visualization is still interesting enough.
I’m unclear whether this means that math emerges from the LLM, or if it means that it has math capabilities that do not emerge from the LLM, but instead are explicitly added via something like Wolfram Alpha.
GPT-4 connects to Wolfram Alpha to enhance its math abilities. And some abilities have emerged from training. GPT-4 has special circuits that enable it to do simple arithmetic, for example. Those evolved on their own after a certain data/parameter size.
We don’t know much about Gemini yet. My understanding is that it combines an LLM with some of the features of Google Deepmind, which I think is similar to Q learning, which is supposedly what Q* the next LLM from OpenAI is using.
Their video is extremely misleading. None of it is real-time or taken directly from the video. See Google’s explanation here:
It’s still impressive, but it’s not real-time language or video processing, nor do the responses happen as quickly as they show. It’s standard text prompts along with added photos.
Disturbing how Q gains sentience, invents time travel, and goes back to troll 4chan. Pretty novel way for AI to kill all humans, though.
I’m still waiting for the ability to solve British cryptic crossword clues. When it can do that, I’ll definitely be impressed.
We should write a story about the real use of AI: to hunt down and murder time travelers before they wreck our timeline.
Because if we ever do invent time travel, we’d be all over the past screwing it up unless something stops us. Since we don’t see future people running around, I have to assume the robot death squads are working as intended.