I know I’m about 2 months late to the party, but I just looked into ChatGPT and I was interested in what I found. These will be some random assessments since I don’t have the energy for a full post. Obviously they keep updating the model so some of this may no longer be true, but here is what I found.

• The model says its data only goes to 2021 and is coy about exactly when its data ends, but I was to be able to pin it down to mid-2021 some time right before the Olympics. The model is unaware of anything about the 2021 Olympics, but believes Naftali Bennet is the current Prime Minister of Israel. Bennet became PM in July 13th 2021, the Olympics began July 23rd. Since the Olympics are always such a media frenzy, I find it hard to believe that ChatGPT would not have been trained on at least a few articles about the 2021 Olympics if its training dataset included dates after July 23rd, so I estimate that its dataset cuts off in mid-July 2021 between the 13th and the 23rd.
• As a topic becomes more in depth, the quality of the answers decreases. It has some amazingly esoteric “surface level” knowledge (ask it what the capital city was for some ancient, long dead nation/civilization), but has a harder time with the kind of deep knowledge that makes one an expert in a field. I asked it to explain how Isoelectric Point relates to pH, and while it gave all the right words it came up with an answer that is the opposite of reality (see image below). For reference if the pH is higher than the pI, the protein will have a positive charge due to deprotonated amino acids. This answer would be like someone giving you a a very deep description of an electron but ended up saying it always has a positive charge. Sounds good! But wrong answer.

• The math update has fixed some of the fun, dumb responses you used to be able to get, and it can now give some strong answers for your physics homework like calculating the speed and trajectory of a baseball. But it still has some weird hangups though and I don’t know why. Complex word problems usually seem ok but simple math is much trickier. I’ve heard tell that the language processor will (somehow) be hooked up to Wolfram Alpha to get solutions to math problems, but it seems like that’s not the case yet.

• As an aside, when ChatGPT gives me a wrong answer, I find myself doubtful and second guessing myself. It gives blatantly wrong answers with the exact same cadence that it tells you all the correct things, so I find myself wondering if I’m the dumb one and my college degrees are all a lie. I guess it proves the maxim that if you just say things confidently people will believe you.
• I wondered if ChatGPT would work as an ad supported model. This may be dumb, but hear me out. Say Khan Academy wants to advertise itself, and they already know students are looking up homework answer on ChatGPT (just like they used google before) so this is the perfect opportunity. A submodel could be trained using Khan Academy-approved language, such as testimonials from happy parents and children about how great Khan Academy is. Then when ChatGPT’s language model starts using words associated with Khan Academy topics (calculus, biology, physics etc) it can insert a canned tagline for Khan Academy and follow it up with words chosen based on the Khan Academy-approved text. So in my pI question above, it could add a snippet somewhere which would go “isoelectric point and pH [tagline starts here] are also taught as part of the Khan Academy course on Chemistry. Parents and students love Khan Academy because blah blah blah [end advertisement] oh by the way deprotonated amino acids are negatively charged.” Inserting ads into your searches is basically Google’s whole business model, and I’d certainly prefer this over a paid version of ChatGPT.
• The non-deterministic nature of the answers makes it sometimes hard to gauge the overall “quality” of the model. I’ve had days where every answer seemed right and days where everything was a shambles. I’ve seen people complain that certain tricks don’t work while others post snapshots showing that they do. I think the output is at least partly determined by previous parts of the conversation but it also just seems semi-random (would love to know if it IS semi-random!). Either way, it makes it hard to judge without doing some statistics that I don’t feel like doing.

Anyway those are my impressions of ChatGPT so far. Fun timewaster, MUCH less toxic than spending all day on Twitter.