Why Frontier LLMs Are Getting Worse at Conversation

If you've been using the latest frontier large language models (LLMs), you've almost certainly noticed by now that the new models are getting worse at conversation.

They're more robotic, speak in jargon, spit out verbose text, and do things you didn't ask for.

How did this happen? Well, I'm not the one training these models, so I can't say for sure, but I do know enough evidence to have a well-informed guess, which I think is interesting to share as a crash course in modern LLM training pipelines.

So let's get started.

Let's go back to 2020. GPT-2 and GPT-3 had been released and were widely available, but they could only predict one token at a time—that's the core of an LLM.

Token prediction was available via API, but there was nothing you could "converse" with. So, while it generated a lot of excitement in academia for emergent intelligence, it wasn't widely adopted.

In 2022, ChatGPT changed all that. The research effort that led to ChatGPT was initially a model called "InstructGPT." It used GPT-3 as the intelligence base and used reinforcement learning from human feedback (RLHF) to teach the model how to "chat."

The core idea of RLHF is that you have the model generate several responses, then have real humans pick the ones they prefer. Do this repeatedly, and you get a model that knows how to converse.

It's worth noting that even in the early InstructGPT days, research found that making the model more pleasant to converse with reduced their pure academic capabilities. This was called the "alignment tax," an interesting thing we'll come back to later.

Various techniques have been used to minimize reliance on humans, but ultimately the reward is modeled on human preferences, which makes these AI assistants easy to talk to.

So keep this in mind: RLHF = training models to be liked by humans.

In 2024, Claude Sonnet 3.5 introduced a turning point, being the first model that could somewhat autonomously complete coding tasks. It sparked the first wave of viable "coding agents."

The way Sonnet 3.5 achieved this was by training the model with a harness (now called an agent) equipped with bash and file editing tools, throwing the agent into a virtual machine, giving it a task, and letting it try to complete it. These tasks all had predefined machine-verifiable outcomes, mainly through test cases, to verify whether the model actually completed the task.

Then you let the model make billions of attempts in such a virtual environment, some of which happen to succeed. You keep the successful agent sessions and use reinforcement learning to teach the model to do that more often—boom, you get a coding agent.

This is called reinforcement learning with verifiable rewards (RLVR). If you look closely, you'll see that during this RLVR process, the model's final text response doesn't matter at all, as long as the code the agent writes passes the tests. It could talk like a jerk and still be rewarded.

So keep this in mind: RLVR = training models to be accepted by machines.

In late 2024 and early 2025, we saw o1 and DeepSeek R1 released as the first wave of "reasoning models." This article is already long, so I won't dive into reasoning models now, but just know that reasoning models also heavily rely on RLVR to scale up the training process—letting the model think before acting, rewarding thought traces if the thinking leads to machine-verifiable results, and teaching the model to think that way more often.

The biggest difference between RLVR and RLHF is that RLVR is more scalable. Getting human feedback is expensive, especially in domains where only experts can have valid opinions on which result is good.

With RLHF, if we have the model generate 100 responses, then humans have to review all 100 responses to pick which one is good.

With RLVR, humans (sometimes AI) only need to define the task and verifier once, and the model can generate a million responses—the machine verifier picks which ones are good in an automated fashion.

So, as a result, RLVR is becoming increasingly dominant in the training pipelines of new models.

If you put all this together:

RLHF = training models to be liked by humans

RLVR = training models to be accepted by machines

RLVR is more scalable

The "alignment tax" says that "being liked by humans" makes models perform worse on verifiable tasks

Now do you see why new models are becoming less and less likable?

This isn't just a matter of "frontier labs messing up their model training"—it's a war between machines and humans, and humans are losing.

We chase benchmarks, and none of those benchmarks measure whether humans actually enjoy working with the models.

We use machines to decide which AI response is better because it's easier and cheaper, and we have no way to ensure those machines truly represent what we humans want.

We let AI complete predefined tasks alone in the dark in virtual environments, at all costs, while in reality, we often can't define verifiable outcomes in advance and need AI to collaborate with us along the way.

I don't have a good solution, but I want to call for awareness that we are beginning to witness the failure of superintelligent alignment firsthand.

This war of machines against humans is one we really can't afford to lose.