Yes and no. It really depends on the model.

The newest Claude Sonnet I’d probably guess will come in above average compared to the humans available for a program like this in making learning fun and personally digestible for each student.

The newest Gemini models could literally cost kids their lives.

The gap between what the public is aware of (and even what many employees at labs, including the frontier ones) and the reality of just how far things have come in the last year is wild.

From the linked article:

It is understood at least two Danish women in their 20s have died, and at least 10 have fallen ill after drinking the tainted alcohol.

A statement from the Danish Ministry of Foreign Affairs said: “The Ministry of Foreign Affairs can confirm that two Danish citizens have passed away in Laos. For reasons of confidentiality in personal matters the Ministry of Foreign Affairs has no further comments.”

‘Nobody’ says anything about anything if you don’t bother to read anything they have to say.

In many cases yes (though I’ve been in good ones when playing off and on, usually the smaller the more there’s actual group activities).

But they are essential to be a part of for blueprints and trading, which are very core parts of the game.

You’ll almost always end up doing missions with other people other than when you intentionally want to do certain tasks solo.

A lot of the game is built around guilds and player to player interactions.

PvP sucks and it’s almost all PvE content vs Destiny though.

Let there be this kind of light in these dark times.

Oh nice, another Gary Marcus “AI hitting a wall post.”

Like his “Deep Learning Is Hitting a Wall” post on March 10th, 2022.

Indeed, not much has changed in the world of deep learning between spring 2022 and now.

No new model releases.

No leaps beyond what was expected.

\s

Gary Marcus is like a reverse Cassandra.

Consistently wrong, and yet regularly listened to, amplified, and believed.

You haven’t used Cursor yet, have you?

That’s definitely one of the ways it’s going to be applied.

The bigger challenge is union negotiations around voice synthesis for those lines, but that will eventually get sorted out.

It won’t be dynamic, unless live service, but you’ll have significantly more fleshed out NPCs by the next generation of open world games (around 5-6 years from now).

Earlier than that will be somewhat enhanced, but not built from the ground up with it in mind the way the next generation will be.

Base model =/= Corpo fine tune

Wait until it starts feeling like revelation deja vu.

Among them are Hymenaeus and Philetus, who have swerved from the truth, saying resurrection has already occurred. They are upsetting the faith of some.

• 2 Tim 2:17-18

I’m a seasoned dev and I was at a launch event when an edge case failure reared its head.

In less than a half an hour after pulling out my laptop to fix it myself, I’d used Cursor + Claude 3.5 Sonnet to:

1. Automatically add logging statements to help identify where the issue was occurring
2. Told it the issue once identified and had it update with a fix
3. Had it remove the logging statements, and pushed the update

I never typed a single line of code and never left the chat box.

My job is increasingly becoming Henry Ford drawing the ‘X’ and not sitting on the assembly line, and I’m all for it.

And this would only have been possible in just the last few months.

We’re already well past the scaffolding stage. That’s old news.

Developing has never been easier or more plain old fun, and it’s getting better literally by the week.

Edit: I agree about junior devs not blindly trusting them though. They don’t yet know where to draw the X.

Actually, they are hiding the full CoT sequence outside of the demos.

What you are seeing there is a summary, but because the actual process is hidden it’s not possible to see what actually transpired.

People are very not happy about this aspect of the situation.

It also means that model context (which in research has been shown to be much more influential than previously thought) is now in part hidden with exclusive access and control by OAI.

There’s a lot of things to be focused on in that image, and “hur dur the stochastic model can’t count letters in this cherry picked example” is the least among them.

You should really look at the full CoT traces on the demos.

I think you think you know more than you actually know.

I’d recommend everyone saying “it can’t understand anything and can’t think” to look at this example:

https://x.com/flowersslop/status/1834349905692824017

Try to solve it after seeing only the first image before you open the second and see o1’s response.

Let me know if you got it before seeing the actual answer.

It’s right in the research I was mentioning:

https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html

Find the section on the model’s representation of self and then the ranked feature activations.

I misremembered the top feature slightly, which was: responding “I’m fine” or gives a positive but insincere response when asked how they are doing.

This is incorrect as was shown last year with the Skill-Mix research:

Furthermore, simple probability calculations indicate that GPT-4’s reasonable performance on k=5 is suggestive of going beyond “stochastic parrot” behavior (Bender et al., 2021), i.e., it combines skills in ways that it had not seen during training.

• https://arxiv.org/abs/2310.17567

The problem is that they are prone to making up why they are correct too.

There’s various techniques to try and identify and correct hallucinations, but they all increase the cost and none are a silver bullet.

But the rate at which it occurs decreased with the jump in pretrained models, and will likely decrease further with the next jump too.

Here you are: https://www.nature.com/articles/s41562-024-01882-z

The other interesting thing is how they get it to end up correct on the faux pas questions asking for less certainty to get it to go from refusal to near perfect accuracy.

Even with early GPT-4 it would also cite real citations that weren’t actually about the topic. So you may be doing a lot of work double checking as opposed to just looking into an answer yourself from the start.

Part of the problem is fine tuning is very shallow, and that a contributing issue for claiming to be right when it isn’t is the pretraining on a bunch of training data of people online claiming to be right when they aren’t.