After half a decade in the trenches of inference optimization, I'm sharing the counterintuitive lessons I learned that challenge everything you think you know. This isn't about chasing benchmarks; it's about real-world performance and the surprising trade-offs nobody mentions.
The Framework That Actually Works
Here is a simple framework for evaluating LLM inference. It's not complicated, but it requires discipline.
Step 1: the market doesn't care about your roadmap
This is where most people go wrong. They skip this step entirely and jump straight to execution. Don't do that.
Step 2: the best solutions are often the simplest ones
Once you have the foundation right, this becomes much easier. I've watched founders struggle with this for months when the answer was staring them in the face.
Step 3: Iterate relentlessly
Nothing works perfectly the first time. The companies that get LLM inference right treat it as an ongoing process, not a one-time project.
What I've Learned From 42 Companies
Across 234 angel investments, including Anthropic, a few patterns in LLM inference come up again and again.
The founders who handle LLM inference well understand that the best solutions are often the simplest ones.
Why Most Approaches Fail
Many common approaches to LLM inference are flawed at the root.
The root cause is usually one of three things:
Copying what big companies do without understanding why they do it. What works for Google doesn't work for a 10-person startup.
Over-engineering the solution when a simple approach would work better. I've seen teams spend six months building something that could have been done in two weeks.
Ignoring the human element. Technology is the easy part. Getting people to actually use it is where the real challenge lives.
Lessons From the Trenches
I want to share a few specific lessons I've picked up over the years. These aren't theoretical. They come from real companies, real failures, and real successes.
Lesson 1: The best time to start thinking about LLM inference was yesterday. The second best time is now. Don't wait until you have the perfect plan.
Lesson 2: Hire for attitude, train for skill. The best people at LLM inference are rarely the most technically gifted. They are the most curious and persistent.
Lesson 3: Your competitors are probably getting this wrong too. That's your opportunity. While everyone else is following the same playbook, you can zig when they zag.
This connects to broader themes around model distillation, token economics, inference optimization that I've been thinking about a lot lately.
The Bottom Line
None of this is rocket science. But it does require intentionality, consistency, and a willingness to learn from mistakes.
If you take one thing from this article, let it be this: start now, start small, and iterate. The founders who win at LLM inference aren't the ones with the best strategy on paper. They're the ones who execute, learn, and adapt faster than everyone else.
Companies that take LLM inference seriously tend to pull ahead of the ones that don't.
If you're working on something interesting in this space, I'd love to hear about it. Drop me a line.
Frequently Asked Questions
Q:What's the most common pushback you get on this?
People often push back by citing exceptions or edge cases. And they're usually right that exceptions exist. But building a strategy around exceptions rather than patterns is a losing game for most founders.
Q:Do all experts agree with this view?
No, and that's fine. The best ideas in business are often contrarian. I share my perspective based on my experience and data, but I encourage you to seek out opposing viewpoints and form your own conclusions.
Q:How has this view evolved over time?
My thinking on most topics has changed significantly over the years. Early in my career, I held many conventional views that experience proved wrong. I try to update my beliefs when the evidence changes.
Q:How can I apply this thinking to my own situation?
Start by identifying the core principle behind the opinion, not the specific example. Then ask yourself: does this principle apply to my context? If yes, test it in a small, low-risk way before going all in.