Building multimodal LLMs is the next frontier, but most attempts are clunky and impractical. I'm sharing my playbook for creating seamless, intuitive multimodal experiences that users will actually love, based on my experience shipping three of them.
The Reality Nobody Talks About
A lot of the advice on multimodal LLMs rests on assumptions that made sense five years ago. The world has moved on.
The first thing to understand is that customer feedback is the only metric that matters. I've seen this play out across dozens of companies. The pattern is unmistakable.
Many teams spend months on the wrong path before realizing that the market doesn't care about your roadmap. Once they make the switch, everything changes.
The Counterintuitive Truth
In multimodal LLMs, the best practitioners often do less, not more.
The lesson: you should focus on one thing and do it exceptionally well. It sounds simple. It's incredibly hard to execute.
What I Tell Founders
Before deciding anything about multimodal LLMs, start with three questions:
What's your timeline? Because the right approach for a company with 6 months of runway is very different from one with 3 years.
What have you already tried? Most founders have tried something. Understanding what didn't work is often more valuable than knowing what might.
Who on your team owns this? If the answer is "everyone" or "no one," that's your first problem to solve.
These questions seem simple but they reveal a lot about where a company actually stands.
This connects to broader themes around small language models, multimodal LLMs, on-device AI that I've been thinking about a lot lately.
Wrapping Up
I've shared a lot here, and I know it can feel overwhelming. But you don't need to get multimodal LLMs right on day one. You just need to get started and keep improving.
Founders who do well at multimodal LLMs tend to share one trait: they're relentlessly practical. They don't chase perfection. They chase progress.
That's the mindset I'd encourage you to adopt. Start where you are. Use what you have. Do what you can. And keep pushing forward.
As always, I'm rooting for you.
Frequently Asked Questions
Q:How long does it take to get multimodal LLMs right?
The timeline varies depending on your starting point and resources. For most founders, expect 2-4 weeks for initial setup and 2-3 months to see meaningful results. I've seen teams move faster when they focus on one thing at a time rather than trying to do everything at once.
Q:Do I need technical skills for multimodal LLMs?
Not necessarily. While technical understanding helps, the most important skills are clear thinking and the ability to break problems into smaller pieces. Many successful founders I've invested in started with zero technical background and either learned enough to be dangerous or found the right technical partner.
Q:How do I measure success with this approach?
Pick one or two metrics that directly tie to your goal and track them weekly. Vanity metrics like page views or follower counts rarely matter. Focus on metrics that reflect real engagement or revenue impact.
Q:What tools do I need to get started?
Start with the basics. You don't need expensive software or fancy tools. A spreadsheet, a note-taking app, and direct access to your customers will get you further than any enterprise platform. Add tools only when you hit a specific bottleneck.