Why train a model from scratch when you can merge the best of what's already out there? I'll walk you through the art and science of model merging, a powerful technique for creating highly specialized models with a fraction of the effort.
The Framework That Actually Works
Here is a simple framework for evaluating model merging. It's not complicated, but it requires discipline.
Step 1: timing is everything in this game
This is where most people go wrong. They skip this step entirely and jump straight to execution. Don't do that.
Step 2: you should focus on one thing and do it exceptionally well
Once you have the foundation right, this becomes much easier. I've watched founders struggle with this for months when the answer was staring them in the face.
Step 3: Iterate relentlessly
Nothing works perfectly the first time. The companies that get model merging right treat it as an ongoing process, not a one-time project.
The Counterintuitive Truth
In model merging, the best practitioners often do less, not more.
The lesson: your team matters more than your technology. It sounds simple. It's incredibly hard to execute.
Lessons From the Trenches
I want to share a few specific lessons I've picked up over the years. These aren't theoretical. They come from real companies, real failures, and real successes.
Lesson 1: The best time to start thinking about model merging was yesterday. The second best time is now. Don't wait until you have the perfect plan.
Lesson 2: Hire for attitude, train for skill. The best people at model merging are rarely the most technically gifted. They are the most curious and persistent.
Lesson 3: Your competitors are probably getting this wrong too. That's your opportunity. While everyone else is following the same playbook, you can zig when they zag.
This connects to broader themes around small language models, model merging, DPO that I've been thinking about a lot lately.
What's Next
Best practice in model merging is changing fast. What worked last year might not work next year. That's both the challenge and the opportunity.
My advice: stay curious, stay humble, and stay close to the people who are actually doing the work. Read less thought leadership and do more experiments. Talk to fewer consultants and more practitioners.
If you're building in this space, the best time to get model merging right is before you need to. Don't wait for a crisis to force your hand.
I'll keep sharing what I learn. This stuff matters too much to keep to myself.
Frequently Asked Questions
Q:How has this view evolved over time?
My thinking on most topics has changed significantly over the years. Early in my career, I held many conventional views that experience proved wrong. I try to update my beliefs when the evidence changes.
Q:What experience informs this perspective?
This perspective comes from over a decade of building companies in Silicon Valley, two successful exits (RemoteTeam to Gusto, MovieLaLa to Gfycat), and investing in 234 startups including Anthropic, OpenAI, and Scale AI. I write about what I've lived.
Q:Do all experts agree with this view?
No, and that's fine. The best ideas in business are often contrarian. I share my perspective based on my experience and data, but I encourage you to seek out opposing viewpoints and form your own conclusions.