The Mysterious case of GPT 4.5
Is it real or just a hallucination?
In last week’s news video, I mentioned the so-called leak about GPT 4.5 that someone came across on Reddit. I also talked about how there’s a high likelihood that it’s not even real because the fact that the context window went way down and the pricing went way up just didn’t make a whole lot of sense. A lot of people pointed out in the comments that OpenAI confirmed that it was fake, which is kind of funny because this is the official confirmation that it was fake. Somebody asked Sam Alman in a reply to one of his posts, “GPT 4.5 leak legit or no?” Sam Alman’s reply was, “Nah.”
However, over this weekend, some new interesting stuff has come to light that I want to explore real quick. Specifically, people were starting to get Chat GPT to actually say that it was using GPT 4.5. You can see right here on Sam’s comment, Quible Cop here showed off a screenshot of Chat GPT saying in the API, “This version is referred to as GPT 4.5 turbo.”
I actually first came across this from one of the community members in the Discord who posted a screenshot at 1:55 a.m. this morning. It’s Sunday, December 17th, 2023, for reference, where they also got this response, “The specific model answering your query is called GPT 4.5 turbo.” I then saw this message from AI Breakfast this morning where they also received the response, “In the OpenAI API, the model answering your query is referred to as GPT 4.5 turbo.” This seems to be happening a lot for people, so much so that on Twitter right now, GPT 4.5 is actually trending. So, of course, I had to test it myself. I jumped into Chat GPT and at first, it started by saying, “We’re currently using the GPT 4 architecture.” I prompted a little bit further, asking what it’s called in the API, not just Chat GPT with browsing or GPT 4, but the specific model name. It came back to me and said, “The precise name of the model answering your query as referred to in the API is GPT 4.5 turbo.”
I wanted to confirm that it wouldn’t just say yes to anything I asked, so I said, “Are you sure you’re not GPT 4.5?” And it said, “Yes, I’m certain.” I then said, “But you are certain you are GPT 4.5 turbo,” and it replied, “Yes, I am certain the model you’re interacting with is GPT 4.5 turbo.” There are tons of Twitter posts right now of people showing off that they got the exact same response. However, Will Dew, who works for OpenAI, was asked, “Is the GPT 4.5 Turbo discovery legit or no?” And his response was, “No, it’s a very weird and oddly consistent hallucination.” He’s claiming that everybody that’s seeing it is getting a hallucination back from Chat GPT.
And then, to make things even more confusing, the official Chat GPT X account is posting cryptic tweets as this is all happening. They posted this tweet today at 2:12 p.m., an emoji of a brain and in the clouds. To me, this is sort of them saying it’s hallucinating through emojis, like our Chat GPT brain is in the cloud today. But who knows for sure what they mean.
Then, Chat GPT a few minutes later went and posted this image, which I’m assuming is an image that was made in Dolly 3. I have no idea if this is in any sort of reference to GPT 4.5 or not, but it seems like Chat GPT is kind of trolling the audience right now through cryptic images and emojis. Also, today, Sean Rolston here tagged me in this post where he took a screenshot of himself having a conversation with Chat GPT and asking it, “What makes GPT 4.5 Superior over GPE 4?” And it actually gave some responses: improved contextual understanding, enhanced logical reasoning, increased efficiency in response time, advanced bias mitigation and safety features and gave some examples of how each of those areas is improved. However, again, Sam Altman said that this isn’t true and another person who works at OpenAI says that this is just a weirdly consistent hallucination.
In all likelihood, it’s probably not GPT 4.5. However, a lot of people have confirmed that Chat GPT has been performing a lot better in the last 24-48 hours. Something over this weekend has improved in Chat GPT, whether it’s officially GPT 4.5 or not. That remains to be consistently denied by OpenAI, but it does still seem to have gotten some sort of upgrade over this weekend.
Ethan Molliek here, who runs the One Useful Thing blog SL newsletter, and is also a professor at Wharton who studies AI innovation in startups made a post yesterday afternoon saying that Chat GPT suddenly got very good again for some reason after being unreliable and a little dull for weeks. He asked Chat GPT to make a file using code and it went off and wrote the code. He said, “I asked it to create files for me” and it insisted it could not. I told it to try, it did. And then look at the top code comment. The top code comment here is, ‘Since the user insists that I can create and provide files, I will.’ So it wrote the code simply because he insisted that it could. He went on to say that it was really, really good and fast. It’s almost like working with a more capable system night and day for both speed and answer quality. Something improved. However, Ethan doesn’t seem to believe that it is 4.5. He goes on to say, “Some replies to the post are saying that their version of Chat GPT Plus is reporting that it is GPT 4.5, not seeing any sign of that myself. Also, the system does not feel like a step change in ability. So far, faster and better quality responses, yes, but not a radical shift.”
Basically, saying that if this was a new model, if this was 4.5, it would be a much bigger boost than just a speed and slight quality improvement. He then goes on to say, “And I should note that it’s not always useful to ask the AI about itself. You get a lot of hallucinations that way that makes it hard to confirm or deny any modifications to the underlying model without official announcements.”
And then, this morning I also came across this video from the YouTuber Wes Roth, who claims, “GPT 4.5 Turbo goes live,” doing a whole bunch of demonstrations in this video showing that it has actually improved. The most interesting part of this demonstration was when he got it to create a version of Pong with a one-shot prompt. A single prompt generated all the code to create Pong.
Just to sort of reiterate what we know so far about all of this, we got this leaked screenshot of GPT 4.5 last week. The pricing doesn’t really make sense, the context window doesn’t really make sense, probably fake. When Sam Alman was asked if it was legit, he said, “Nah.” Will Dew over at OpenAI, in response to all those Chat GPT saying that it’s using version 4.5 Turbo, claims that it’s just an oddly consistent hallucination. And then, of course, we have Ethan Molliek, who shows while he got much more improved results over this weekend, he doesn’t believe that it’s a big enough leap to actually be a 4.5 model. So, in all likelihood, we’re not actually seeing GPT 4.5 Turbo. We’re getting this consistent hallucination. However, we are seeing some sort of improvement coming out of Chat GPT this weekend.
Most likely, what happened was OpenAI was starting to realize that GPT4 was getting lazier. Back on December 7th, they said, “Model Behavior can be unpredictable and we’re looking into fixing it.” They probably made some updates to it, probably changed the system prompt a little bit behind the scenes, and probably improved a few things around the speed and reliability of the prompt responses. But it’s unlikely that they trained a brand new model that they’re calling 4.5. But I could also be totally wrong–this is just kind of me having some fun doing some research, and doing some speculation, reading what other people are saying, and kind of coming to my own conclusions. I doubt it’s actually a 4.5 Turbo model, but who knows–maybe this week, Sam Alman comes out and says, ‘Psych! It’s been .5 for the last week now, and you guys didn’t even notice,’ but I don’t think that’s going to happen. Anyway, I just wanted to make a super quick update about this because if you’re on X and you’re on YouTube, and you’re paying attention to what’s going on in the AI world right now, you’re probably seeing a lot of stuff about GPT 4.5 and GPT 4.5 Turbo right now, and I wanted to sort of clear the air and break down what we actually know, sort of separate fact from fiction right now, what people are seeing, what OpenAI themselves are actually saying, and what my sort of guess on what’s really going on is.
