I'm probably going to get a lot of hate for this, but it needs to be said: your approach to a/b testing ai is fundamentally flawed. We're all chasing shiny AI objects and forgetting the first principles of building great products. Here's the unpopular opinion that might just save your startup.
The Reality Nobody Talks About
A lot of the advice on pricing experiments rests on assumptions that made sense five years ago. The world has moved on.
The first thing to understand is that timing is everything in this game. I've seen this play out across dozens of companies. The pattern is unmistakable.
Many teams spend months on the wrong path before realizing that your team matters more than your technology. Once they make the switch, everything changes.
The Framework That Actually Works
Here is a simple framework for evaluating pricing experiments. It's not complicated, but it requires discipline.
Step 1: customer feedback is the only metric that matters
This is where most people go wrong. They skip this step entirely and jump straight to execution. Don't do that.
Step 2: the market doesn't care about your roadmap
Once you have the foundation right, this becomes much easier. I've watched founders struggle with this for months when the answer was staring them in the face.
Step 3: Iterate relentlessly
Nothing works perfectly the first time. The companies that get pricing experiments right treat it as an ongoing process, not a one-time project.
The Numbers Don't Lie
Investing early in pricing experiments can be the difference between raising your next round and running out of runway.
One of my portfolio companies went from struggling to profitable in under a year after they finally got serious about this. The founder told me later that they wished they'd started sooner.
This connects to broader themes around a/b testing ai, pricing, segmentation that I've been thinking about a lot lately.
What's Next
Best practice in pricing experiments is changing fast. What worked last year might not work next year. That's both the challenge and the opportunity.
My advice: stay curious, stay humble, and stay close to the people who are actually doing the work. Read less thought leadership and do more experiments. Talk to fewer consultants and more practitioners.
If you're building in this space, the best time to get pricing experiments right is before you need to. Don't wait for a crisis to force your hand.
I'll keep sharing what I learn. This stuff matters too much to keep to myself.
Frequently Asked Questions
Q:How can I apply this thinking to my own situation?
Start by identifying the core principle behind the opinion, not the specific example. Then ask yourself: does this principle apply to my context? If yes, test it in a small, low-risk way before going all in.
Q:How has this view evolved over time?
My thinking on most topics has changed significantly over the years. Early in my career, I held many conventional views that experience proved wrong. I try to update my beliefs when the evidence changes.
Q:What experience informs this perspective?
This perspective comes from over a decade of building companies in Silicon Valley, two successful exits (RemoteTeam to Gusto, MovieLaLa to Gfycat), and investing in 234 startups including Anthropic, OpenAI, and Scale AI. I write about what I've lived.
Q:What's the most common pushback you get on this?
People often push back by citing exceptions or edge cases. And they're usually right that exceptions exist. But building a strategy around exceptions rather than patterns is a losing game for most founders.