I once burned $150,000 on a single AI project that went absolutely nowhere.
It was early in my journey, long before my exits with RemoteTeam and MovieLaLa. We were building a predictive analytics model that was supposed to be our silver bullet. We had a team of smart people, a mountain of data, and what we thought was a brilliant idea. Six months later, all we had was a lighter bank account and a model that couldn't predict next week's weather, let alone customer churn. It was a painful, expensive failure. And it wasn't the last.
Look, I get it. The hype around AI is deafening. You see headlines about companies using AI to achieve unbelievable results, and you feel the pressure to keep up. You hire data scientists, you invest in the latest tools, and you expect magic to happen. But most of the time, it doesn’t. The models fail, the dashboards gather dust, and the promised ROI never appears. After dissecting over a hundred datasets and burning through more cash than I care to admit, I started to see a pattern. The failures weren't about the algorithms or the tech. They were about the data. More specifically, they were about the brutal, uncomfortable truths about data that nobody likes to talk about.
Here are the five I learned the hard way.
1. Your “Perfect” Data is a Lie
At MovieLaLa, we were obsessed with building the perfect recommendation engine. We thought if we could just collect enough data points—every click, every search, every hover—we could create a flawless system. We spent months building intricate data pipelines to capture everything. The dataset was massive, and on paper, it looked pristine. We thought we had gold.
We had garbage.
What we failed to account for was the noise. A user hovering over a movie poster for 2 seconds didn't mean they were interested; it often meant they were distracted by a cat video on another tab. The clicks weren't always a signal of intent; sometimes they were just mis-clicks. Our beautiful, clean dataset was a funhouse mirror, reflecting a distorted version of reality. The model we built on it was, unsurprisingly, useless. It was a classic case of garbage in, garbage out. We had to scrap it and start over, this time by talking to actual users and focusing on explicit signals, like adding a movie to a watchlist. It was a humbling and expensive lesson in the difference between big data and good data.
Here’s the thing: there is no such thing as perfect data. Every dataset has flaws, biases, and gaps. The pursuit of data perfection is a trap that will paralyze your projects and drain your budget. Instead of trying to boil the ocean, focus on "good enough" data that is directly relevant to the problem you're solving. A smaller, cleaner dataset with strong signals will always outperform a massive, noisy one.
2. The Model is the Easiest Part
I’ve sat in countless meetings where data scientists argue for hours about the merits of a random forest versus a gradient boosting machine or a complex neural network. It’s the fun part, the intellectually stimulating part. It’s also the least important part.
I honestly had no idea what I was doing in the early days. I’d hire PhDs and trust them to figure it out. They would spend 80% of their time fine-tuning model hyperparameters and 20% on the data itself. Our results were consistently mediocre. It wasn’t until I flipped that ratio on its head that we started seeing real wins. The breakthrough came when we stopped obsessing over the algorithm and started obsessing over feature engineering.
For one of our key projects at RemoteTeam, we were trying to predict which companies were most likely to upgrade to a higher-tier plan. The initial models were barely better than a coin flip. The team was stuck. So, we locked ourselves in a room for a week and did nothing but brainstorm and build new features. We looked at everything: the number of users in different time zones, the frequency of specific feature usage, the time between inviting a new team member and them becoming active. We created dozens of new features from the raw data. When we fed those features into a simple logistic regression model, one of the most basic models out there, it blew our complex, over-engineered model out of the water. The accuracy jumped by 40%. That project alone generated a 7-figure uplift in revenue.
Data scientists love to geek out on models, but 90% of the value comes from the features you feed them. Your team should be spending the vast majority of their time on understanding the business problem, exploring the data, and creatively engineering features that capture real-world signals. The model is just the engine; the features are the fuel.
3. Vanity Metrics Will Bankrupt You
Early on, we had this huge dashboard on the wall of the office. It was covered in charts and numbers that were always going up and to the right. Daily active users, sign-ups, page views. It looked impressive. It made us feel good. But it was a lie.
Those were vanity metrics. They were easy to measure and easy to goose, but they had almost no correlation with the health of our business. We were celebrating a growing user base while our churn rate was quietly creeping up and our revenue per user was flat. We were so focused on the top of the funnel that we were ignoring the fact that the bottom was a leaky bucket. This is a common trap, and I see it in AI projects all the time. Teams build AI dashboards that track things like "model accuracy" or "predictions served" without tying them to a single meaningful business outcome.
Who cares if your model is 99% accurate if it’s not making you money or saving you money? I’ve seen companies spend a fortune on predictive maintenance models that were technically brilliant but cost more to operate than the breakdowns they were preventing. You have to be ruthless about connecting your AI initiatives to real, tangible business value. Are you increasing revenue? Are you reducing costs? Are you improving customer retention? If you can’t answer those questions, you’re just playing a game. We learned this lesson again when optimizing our user funnels, which you can read about in my post on growth hacking.
4. Your Team is Siloed and It's Killing Your ROI
You can have the best data scientists in the world, but if they operate in a vacuum, they will fail. I’ve seen this movie play out dozens of times. The data team builds a "brilliant" model and throws it over the wall to the engineering team, who have no idea how to deploy it or maintain it. Or they present their findings to the business team, who don’t understand the implications or don’t trust the "black box" they’ve been handed.
AI is not a technical problem; it’s an organizational one. To succeed, you need a cross-functional team of data scientists, engineers, product managers, and business stakeholders all working together from day one. The business people need to define the problem and the success criteria. The data scientists need to explore the data and build the model. The engineers need to build the infrastructure to deploy and scale it. It has to be a partnership.
One of the best investments I ever made was embedding data scientists directly into our product teams. Instead of being a separate service organization, they became core members of the team, involved in every stage of the product development lifecycle. It was a big deal. The communication overhead dropped, the feedback loops tightened, and the models they built were suddenly much more relevant and impactful. It required a shift in mindset, moving from a project-based approach to a product-based one. If you want to learn more about how we structure our teams, check out my thoughts on building high-performance product pods.
5. You’re Not Thinking Big Enough
The final brutal truth is a bit different. It’s not about avoiding failure, but about the size of the wins. Too many companies approach AI data analysis with a timid, incremental mindset. They look for small optimizations, a 2% improvement in conversion rates, a 5% reduction in costs. These are fine, but they are not transformative.
The real power of AI is its ability to fundamentally change the way you do business. It can help you create entirely new products, enter new markets, and build sustainable competitive advantages. When I look at my most successful investments, companies like Scale AI or Hugging Face, they aren’t just optimizing existing processes. They are creating entirely new categories.
After my initial stumbles, I realized I was thinking too small. I was using data to answer questions, when I should have been using it to discover the questions I didn’t even know I should be asking. Instead of just building a churn prediction model, why not build a model that identifies the root causes of churn and suggests proactive interventions? Instead of just personalizing marketing content, why not create a fully adaptive product experience that reconfigures itself for every single user?
This requires a shift from thinking about AI as a tool to thinking about it as a capability. It’s not about a single project; it’s about building a data-driven culture and a platform that allows for rapid experimentation and learning. It’s a much bigger, harder, and more ambiguous challenge. But it’s also where the 7-figure wins and the market-defining breakthroughs are found.
The Real Work
Getting value from AI data is not easy. It’s a messy, frustrating, and often counterintuitive process. You will face setbacks, and you will make mistakes. I certainly did. But the potential rewards are immense. By embracing these brutal truths, by being realistic about your data, focusing on features, tying your work to business value, breaking down silos, and thinking bigger, you can avoid the most common traps.
The real work isn’t in the algorithm. It’s in the trenches, cleaning the data, talking to users, and building the organizational muscle to turn insights into action. It’s less glamorous than the headlines suggest, but it’s the only path to real, sustainable success.
Frequently Asked Questions
Which item on this list has the highest impact?
It depends on your stage and context, but in my experience, the items near the top of the list tend to have the broadest applicability. That said, sometimes the less obvious items create the biggest breakthroughs for specific situations.
Are these recommendations still relevant in 2026?
Absolutely. While specific tools and tactics change, the underlying principles remain consistent. I update my thinking regularly based on what I'm seeing in the market and across my portfolio companies.
How were these items selected?
Each item on this list comes from direct experience, either from building my own companies or from patterns I've observed across the 200+ startups I've invested in. I prioritize practical, actionable items over theoretical concepts.