We thought we had it. A 99.7% accuracy rate on our diagnostic model. In the lab, it was a thing of beauty. We were huddled around a monitor at 2 AM, watching the results flash across the screen, and it felt like we were staring into the future. We were going to save lives. We were going to be rich. We were going to change the world.
Five years and $15 million later, the project was dead. The code is probably sitting on a forgotten server somewhere. The team has scattered to the winds. And I was left with a very expensive, very painful education.
I’ve been in Silicon Valley for a long time. I’ve seen bubbles come and go. I’ve had two successful exits with RemoteTeam and MovieLaLa, and I’ve written checks for over 200 startups, including some of the biggest names in AI like Anthropic and OpenAI. I’ve seen what success looks like up close. But my biggest lessons didn’t come from the wins. They came from the failures. Especially this one.
Most founders in the healthcare AI space are completely delusional. They think building a great model is the hard part. They’re wrong. That’s the easiest part. Here’s the uncomfortable truth about what it really takes to succeed in the brutal, regulated world of healthcare AI.
The Great Lie of Model Accuracy
Everyone in the AI world is obsessed with accuracy. 99%. 99.5%. 99.999%. It’s the metric we use to pat ourselves on the back and raise our next round of funding. But in a real clinical setting, it’s almost meaningless.
Our model could spot a specific type of early-stage cancer from a medical image with near-perfect accuracy. We celebrated it. We put it in our pitch deck. We thought it was our golden ticket. But when we tried to get doctors to actually use it, they couldn’t have cared less.
I remember one meeting with a senior radiologist at a major hospital. We showed him the demo, the flashing green box correctly identifying a tiny, malignant nodule that a junior resident had missed. He watched, nodded politely, and then said, “So I have to log into another system for this?” He wasn’t being a Luddite. He was being practical. His day was already a nightmare of clicking between different windows, re-entering patient IDs, and dealing with software that looked like it was designed in 1998. Our revolutionary AI was just one more piece of cognitive overhead he didn’t have time for.
Why? Because a doctor’s workflow isn’t a Kaggle competition. They aren’t just looking at a single image in isolation. They’re dealing with a patient. A human being with a messy, complicated history, a dozen other symptoms, and a terrible health insurance plan. Our “99.7% accurate” model didn’t integrate with the hospital’s 20-year-old Electronic Health Record (EHR) system. It required a doctor to log into a separate interface, upload an image manually, and wait 30 seconds for a result. No one had time for that.
The brutal truth: A B+ model that fits seamlessly into a doctor’s existing workflow is infinitely more valuable than an A+ model that requires them to change how they work. We spent 90% of our energy on the algorithm and 10% on the workflow. We should have flipped that ratio. We should have obsessed over the user experience of the doctor, not the elegance of our convolutional neural network.
Your Real Customer Isn't the Doctor
We built our tool for radiologists. We spent months shadowing them, interviewing them, and refining our UI based on their feedback. They loved it. They told us it was incredible. And then they told us they had absolutely no power to buy it.
In a hospital system, the end-user is rarely the economic buyer. The person who buys software is a hospital administrator, a value analysis committee, or a CIO in the IT department. Their concerns are not about diagnostic elegance. They care about three things:
- Cost & ROI: How much does it cost, and how much will it save us? Not in some abstract “better diagnostics” way, but in hard dollars. Will it reduce patient readmission rates? Will it decrease the average length of stay? Can we bill a higher reimbursement code for using this technology? We couldn’t answer these questions with concrete data.
- Integration & Security: Does it work with our ancient, creaking infrastructure? Can it be deployed on-premise behind our firewall? Does it use a standard protocol like HL7 or FHIR? Is it HIPAA compliant, and have you been audited by a third party? Our beautiful, cloud-native application was a security nightmare for them.
- Liability & Regulation: If the AI makes a mistake, who gets sued? The doctor? The hospital? You? What happens if the model’s performance drifts over time? Are you FDA-cleared? What class of medical device is this? We had a beautiful answer for the doctors, but we had a terrible answer for the administrators. We were trying to sell a sports car to someone who just needed a reliable, cheap minivan.
We spent all our time winning the hearts of the clinicians, but we completely ignored the minds of the people who held the purse strings. We learned the hard way that in healthcare, the sales process is just as complex as the technology.
The Data Moat is a Myth
Every AI startup pitch deck has a slide about their “data moat.” The idea is that once you have enough proprietary data, no one can ever catch up to you. It’s a nice theory. In healthcare, it’s mostly a fantasy.
First, getting the data is hell. It’s locked up in siloed systems, governed by HIPAA, and owned by institutions that are rightly paranoid about privacy. We spent two years and over $1 million in legal fees just to get our initial training dataset. It was anonymized, but the process was still a nightmare of negotiating data use agreements, getting IRB approval, and dealing with hospital lawyers who are paid to say no.
Second, the data you get is often garbage. It’s poorly labeled, inconsistent, and full of biases from the real-world clinical practices where it was generated. We found images that were scanned at different resolutions, reports with missing fields, and diagnoses that were later overturned. We had to spend another year just cleaning and standardizing the data before our model could make any sense of it. It was digital janitorial work, and it was 80% of the job.
By the time we had a working model, new and better public datasets were already becoming available. Research institutions were releasing their own curated datasets, and the advantage we thought we had was evaporating before our eyes. The moat was a mirage.
The lesson: Your competitive advantage can’t just be your data. It has to be what you do with the data. It has to be the trust you build with clinicians, the workflow you perfect, and the boring, unsexy business case you build for the hospital administrators. It has to be the service you provide, not just the algorithm.
There Is No 'Aha!' Moment
In the movies, building a startup is a series of dramatic breakthroughs. There’s the late-night coding session, the sudden inspiration, the “aha!” moment that changes everything. My experience was nothing like that.
It was a slow, painful grind. A thousand tiny steps forward and nine hundred and ninety-nine steps back. It was fighting with insurance companies over reimbursement codes. It was debugging a legacy API that was older than some of our engineers. It was sitting in a beige conference room, trying to explain deep learning to a hospital procurement officer who just wanted to know if we were compatible with Windows XP.
I remember one specific week where we thought we were finally making progress. We had a pilot program lined up. Then, on Monday, we found out the hospital’s IT department had a freeze on all new software installations. On Tuesday, our lead engineer got poached by Google. On Wednesday, our cloud provider had a major outage that took down our demo environment. On Thursday, the hospital informed us that the key physician champion for our project was leaving to go to a competitor. By Friday, I was sitting in my car in the office parking lot, just staring at the steering wheel, wondering if I had what it took to keep going.
We failed not because of one big mistake, but because of a thousand small cuts. We underestimated the complexity of the problem. We were seduced by our own technology. We were a team of builders who didn’t understand the business of healthcare.
I don’t regret the experience. The lessons I learned have been invaluable in my angel investing. When a founder pitches me a healthcare AI company now, I don’t ask about their model’s accuracy. I ask them about their integration partners. I ask them about their sales cycle. I ask them who the economic buyer is. I ask them the questions I wish someone had asked me five years ago.
Building in healthcare AI is not for the faint of heart. It’s a brutal, unforgiving landscape. But the problems are real, and the opportunity to make a difference is immense. Just know what you’re getting into. Don’t believe the hype. And for God’s sake, don’t fall in love with your model. Fall in love with the problem you’re trying to solve. That’s the only thing that will get you through the grind.
Frequently Asked Questions
What experience informs this perspective?
This perspective comes from over a decade of building companies in Silicon Valley, two successful exits (RemoteTeam to Gusto, MovieLaLa to Gfycat), and investing in 200+ startups including Anthropic, OpenAI, and Scale AI. I write about what I've lived.
What's the most common pushback you get on this?
People often push back by citing exceptions or edge cases. And they're usually right that exceptions exist. But building a strategy around exceptions rather than patterns is a losing game for most founders.
How can I apply this thinking to my own situation?
Start by identifying the core principle behind the opinion, not the specific example. Then ask yourself: does this principle apply to my context? If yes, test it in a small, low-risk way before going all in.
How has this view evolved over time?
My thinking on most topics has changed significantly over the years. Early in my career, I held many conventional views that experience proved wrong. I try to update my beliefs when the evidence changes.