The cloud is expensive and slow. I've spent the last two years focused on on-device AI, and I'm sharing everything I've learned about running powerful LLMs directly on your users' hardware. This is the future of private, personalized AI.
What I've Learned From 53 Companies
Across 234 angel investments, including Anthropic, a few patterns in on-device AI come up again and again.
The founders who handle on-device AI well understand that you need to move fast and break things.
The Framework That Actually Works
Here is a simple framework for evaluating on-device AI. It's not complicated, but it requires discipline.
Step 1: customer feedback is the only metric that matters
This is where most people go wrong. They skip this step entirely and jump straight to execution. Don't do that.
Step 2: your team matters more than your technology
Once you have the foundation right, this becomes much easier. I've watched founders struggle with this for months when the answer was staring them in the face.
Step 3: Iterate relentlessly
Nothing works perfectly the first time. The companies that get on-device AI right treat it as an ongoing process, not a one-time project.
What I Tell Founders
Before deciding anything about on-device AI, start with three questions:
What's your timeline? Because the right approach for a company with 6 months of runway is very different from one with 3 years.
What have you already tried? Most founders have tried something. Understanding what didn't work is often more valuable than knowing what might.
Who on your team owns this? If the answer is "everyone" or "no one," that's your first problem to solve.
These questions seem simple but they reveal a lot about where a company actually stands.
This connects to broader themes around inference optimization, on-device AI, small language models that I've been thinking about a lot lately.
What's Next
Best practice in on-device AI is changing fast. What worked last year might not work next year. That's both the challenge and the opportunity.
My advice: stay curious, stay humble, and stay close to the people who are actually doing the work. Read less thought leadership and do more experiments. Talk to fewer consultants and more practitioners.
If you're building in this space, the best time to get on-device AI right is before you need to. Don't wait for a crisis to force your hand.
I'll keep sharing what I learn. This stuff matters too much to keep to myself.
Frequently Asked Questions
Q:How should I work through this guide?
Don't try to absorb everything in one sitting. Read through once to get the big picture, then go back and work through each section as it becomes relevant to your current challenges. Bookmark it and return to it regularly.
Q:How often is this guide updated?
I revisit and update my guides regularly as I learn new things and as the market evolves. The core principles tend to stay stable, but specific tactics and tools get refreshed based on what's working right now.
Q:Is this guide based on real experience?
Every recommendation in this guide comes from direct experience, either from building and selling my own companies, or from patterns I've observed across 234 angel investments. I don't write about things I haven't personally tested.