I've seen countless startups burn through cash trying to implement RLHF without understanding the fundamentals. Here's the painful truth about why it fails and a simple framework for getting it right from day one.
Why Most Approaches Fail
Many common approaches to RLHF are flawed at the root.
The root cause is usually one of three things:
Copying what big companies do without understanding why they do it. What works for Google doesn't work for a 10-person startup.
Over-engineering the solution when a simple approach would work better. I've seen teams spend six months building something that could have been done in two weeks.
Ignoring the human element. Technology is the easy part. Getting people to actually use it is where the real challenge lives.
The Reality Nobody Talks About
A lot of the advice on RLHF rests on assumptions that made sense five years ago. The world has moved on.
The first thing to understand is that the best solutions are often the simplest ones. I've seen this play out across dozens of companies. The pattern is unmistakable.
Many teams spend months on the wrong path before realizing that customer feedback is the only metric that matters. Once they make the switch, everything changes.
The Framework That Actually Works
Here is a simple framework for evaluating RLHF. It's not complicated, but it requires discipline.
Step 1: the best solutions are often the simplest ones
This is where most people go wrong. They skip this step entirely and jump straight to execution. Don't do that.
Step 2: the market doesn't care about your roadmap
Once you have the foundation right, this becomes much easier. I've watched founders struggle with this for months when the answer was staring them in the face.
Step 3: Iterate relentlessly
Nothing works perfectly the first time. The companies that get RLHF right treat it as an ongoing process, not a one-time project.
What I Tell Founders
Before deciding anything about RLHF, start with three questions:
What's your timeline? Because the right approach for a company with 6 months of runway is very different from one with 3 years.
What have you already tried? Most founders have tried something. Understanding what didn't work is often more valuable than knowing what might.
Who on your team owns this? If the answer is "everyone" or "no one," that's your first problem to solve.
These questions seem simple but they reveal a lot about where a company actually stands.
This connects to broader themes around RLHF, DPO, small language models that I've been thinking about a lot lately.
What's Next
Best practice in RLHF is changing fast. What worked last year might not work next year. That's both the challenge and the opportunity.
My advice: stay curious, stay humble, and stay close to the people who are actually doing the work. Read less thought leadership and do more experiments. Talk to fewer consultants and more practitioners.
If you're building in this space, the best time to get RLHF right is before you need to. Don't wait for a crisis to force your hand.
I'll keep sharing what I learn. This stuff matters too much to keep to myself.
Frequently Asked Questions
Q:What's the most common pushback you get on this?
People often push back by citing exceptions or edge cases. And they're usually right that exceptions exist. But building a strategy around exceptions rather than patterns is a losing game for most founders.
Q:What experience informs this perspective?
This perspective comes from over a decade of building companies in Silicon Valley, two successful exits (RemoteTeam to Gusto, MovieLaLa to Gfycat), and investing in 234 startups including Anthropic, OpenAI, and Scale AI. I write about what I've lived.
Q:Do all experts agree with this view?
No, and that's fine. The best ideas in business are often contrarian. I share my perspective based on my experience and data, but I encourage you to seek out opposing viewpoints and form your own conclusions.