I had a founder pitch me last week. Super sharp, great product idea, and a solid team. They were building an AI-powered tool for architects, and they had a clear vision for how they were going to change the industry. But then I asked them about their AI infrastructure costs, and the conversation went south. They had a line item for GPUs in their financial model, but they hadn't factored in any of the other costs. I had to be the bad guy and tell them that their model was completely wrong. It's a conversation I have almost every day.
You think you know what your AI infrastructure costs? You're probably wrong. It's not just about the GPUs. I'll break down the hidden costs of power, cooling, and engineering talent that are silently eating into your runway.
I’ve seen it happen more times than I can count. A founder comes to me, excited about their new AI-powered product. They’ve raised a seed round, hired a couple of engineers, and they’re ready to change the world. They’ve even factored in the cost of renting a few A100s. But when I ask them about the other costs, I get a blank stare. It’s not their fault. The true cost of AI infrastructure is one of the best-kept secrets in Silicon Valley. The big cloud providers have no incentive to tell you that the sticker price of their GPUs is just the beginning. They want you to get locked into their ecosystem, and then they hit you with the hidden fees.
The Iceberg of AI Infrastructure Costs
Think of your AI infrastructure costs as an iceberg. The GPUs are the tip of the iceberg, the part that everyone sees. But the real danger is what’s lurking beneath the surface. The hidden costs of power, cooling, networking, and engineering talent can be 2-3x the cost of the GPUs themselves.
Let’s break it down:
Power and Cooling: A single NVIDIA H100 GPU can consume up to 700 watts of power. A rack of 8 of them can consume more power than a small house. And all that power generates a lot of heat, which means you need to spend a lot of money on cooling. I remember when we were building out the infrastructure for one of my early startups, MovieLaLa, we had to install a dedicated air conditioning unit just for our server room. It was a huge, unexpected expense that almost killed our budget. We were literally burning money to keep our servers from melting.
Networking: AI workloads are incredibly network-intensive. You need high-bandwidth, low-latency networking to connect your GPUs and your storage. This can be a major expense, especially if you’re using a cloud provider. I’ve seen startups get hit with six-figure networking bills in a single month. It's not just the cost of the bandwidth, it's the complexity of setting up and managing the network. You need to be a networking expert to get it right, and most startups don't have that expertise in-house.
Engineering Talent: This is the big one. Good AI engineers are expensive and hard to find. And you need a lot of them to build and maintain your AI infrastructure. You need engineers who can optimize your models, debug your code, and keep your systems running 24/7. When I was at RemoteTeam, we had a whole team of engineers dedicated to just managing our AI infrastructure. It was a huge investment, but it was worth it. We couldn't have scaled the business without them. The opportunity cost of having your best engineers managing infrastructure instead of building product is immense.
The Pricing Model Maze: Per-Token, Per-Seat, and Everything in Between
Now that you have a better understanding of the true costs of AI infrastructure, let's talk about how to price your own AI-powered product. This is where I see a lot of founders make mistakes. They either charge too little and leave money on the table, or they charge too much and price themselves out of the market.
There are a few common pricing models for AI products:
Per-Token: This is the most common pricing model for large language models like GPT-3. You pay for the number of tokens (words or parts of words) that you use. This model is simple and transparent, but it can be hard to predict your costs. I've seen companies get hit with huge bills because they underestimated their token usage. It's like leaving the water running; you don't realize how much you're using until you get the bill.
Per-Seat: This is a more traditional SaaS pricing model. You pay a flat fee per user per month. This model is predictable and easy to understand, but it can be hard to justify if your product is not used consistently by all users. When we were at RemoteTeam, we used a per-seat model, and it worked well for us. But we had to make sure that our product was providing enough value to justify the cost for every single user.
Usage-Based: This is a hybrid model that combines elements of per-token and per-seat pricing. You might pay a flat fee for a certain number of users, and then pay for additional usage on a per-token or per-API-call basis. This model can be a good way to balance predictability and flexibility. I'm a big fan of this model, and I've seen it work well for a lot of companies. It aligns the value that the customer is getting with the price they are paying.
Value-Based: This is the holy grail of pricing. You price your product based on the value that it provides to your customers. This can be hard to quantify, but if you can do it, you can build a much more profitable business. For example, if your AI-powered tool helps architects win more bids, you could charge a percentage of the value of the contracts they win. This is the ultimate win-win.
The Rise of Vertical SaaS and Serverless AI
So what’s the solution to the high cost of AI infrastructure and the complexity of pricing? For many startups, the answer is to move away from horizontal AI platforms and towards vertical SaaS and serverless AI.
Vertical SaaS companies are building AI-powered products for specific industries. They’re able to build deep domain expertise and create products that are much more valuable to their customers than a generic AI platform. And because they’re focused on a specific niche, they can often get by with a much smaller AI infrastructure. I’m a big believer in this trend. I’ve invested in several vertical SaaS companies, and I think they’re the future of the AI industry. One of my investments, an AI-powered legal tech company, is a great example of this. They are able to charge a premium for their product because it solves a very specific and expensive problem for their customers.
Serverless AI is another trend that’s helping to reduce the cost of AI infrastructure. With serverless AI, you only pay for the compute that you actually use. You don’t have to worry about managing your own servers or paying for idle capacity. This can be a huge cost savings for startups. I'm an investor in Hugging Face, and they are doing amazing work in making it easier for developers to build and deploy AI models without having to worry about the underlying infrastructure.
My Advice to Founders
So, what's the takeaway here? If you're a founder building an AI-powered product, here's my advice:
- Don't be afraid to get your hands dirty. You need to understand the nuts and bolts of your AI infrastructure. You don't have to be an expert, but you need to know enough to ask the right questions and make informed decisions.
- Build a realistic financial model. Don't just copy and paste a template you found online. Do your own research, talk to other founders, and build a model that is based on your specific business.
- Think carefully about your pricing model. Don't just default to the per-token model because that's what everyone else is doing. Think about what makes sense for your business and your customers.
- Consider a vertical SaaS or serverless AI approach. It's not the easy path, but it's the smart path. And in the world of startups, the smart path is the only path that leads to success.
Building an AI-powered product is hard. It's expensive, it's complex, and it's constantly changing. But it's also one of the most exciting and rewarding things you can do. If you can get the infrastructure and pricing right, you can build a business that will change the world. And I, for one, am excited to see what you build.
I want to double-click on the engineering talent part. At RemoteTeam, we were acquired by Gusto, and one of the things that made us attractive was our ability to build and manage a complex AI infrastructure with a relatively small team. But that didn't happen by accident. We spent months recruiting the right people, and we made a few costly mistakes along the way. I remember one time we hired a brilliant data scientist who was a rockstar at building models, but he had no experience with infrastructure. He ended up spending all his time trying to keep our servers from crashing, and he wasn't able to do what he did best. It was a classic case of having the right person in the wrong seat. We learned our lesson and hired a dedicated infrastructure engineer, and it made all the difference.
The Ultimate Goal: Value-Based Pricing in Action
I mentioned value-based pricing earlier, and I want to spend a little more time on it because I think it's the future of SaaS. It's not easy to implement, but if you can get it right, it's a game-changer.
Let's go back to the example of the AI-powered tool for architects. How could you implement value-based pricing for that product? Well, you could start by tracking the number of bids that your customers win using your tool. You could then charge them a percentage of the value of those contracts. For example, you could charge them 1% of the value of every contract they win. If they win a $1 million contract, you get $10,000. It's a simple and powerful model that aligns your incentives with your customers' incentives.
Of course, it's not always that simple. You need to be able to track the right metrics, and you need to be able to convince your customers that your product is actually responsible for their success. But if you can do it, you can build a much more profitable and sustainable business. I've seen it happen with some of my portfolio companies, and it's a beautiful thing to watch.
One of my investments, a company that provides AI-powered marketing automation, is a great example of this. They started with a traditional per-seat pricing model, but they quickly realized that it wasn't working. Their customers were getting a ton of value from the product, but they weren't paying for it. So they switched to a value-based pricing model. They now charge their customers a percentage of the revenue that they generate from the marketing campaigns that are powered by their platform. Their revenue has exploded since they made the switch, and their customers are happier than ever.
The Future is Vertical and Serverless
I'm more convinced than ever that the future of AI is vertical and serverless. The horizontal AI platforms are great for experimentation, but they're not the right solution for most businesses. They're too expensive, too complex, and too generic. The real value is in building AI-powered products for specific industries. And the best way to do that is with a vertical SaaS or serverless AI approach.
I'm putting my money where my mouth is. I'm actively investing in companies that are taking this approach. I'm looking for founders who have deep domain expertise, a clear vision for how they're going to change their industry, and a smart plan for how they're going to build and price their product. If that sounds like you, I'd love to hear from you.
Building an AI-powered product is a marathon, not a sprint. It's a long and difficult journey, but it's also one of the most rewarding things you can do. So don't be discouraged by the challenges. Embrace them. And if you need a little help along the way, don't be afraid to ask for it. The future is bright for those who are bold enough to build it.
Frequently Asked Questions
What's the most common pushback you get on this?
People often push back by citing exceptions or edge cases. And they're usually right that exceptions exist. But building a strategy around exceptions rather than patterns is a losing game for most founders.
Do all experts agree with this view?
No, and that's fine. The best ideas in business are often contrarian. I share my perspective based on my experience and data, but I encourage you to seek out opposing viewpoints and form your own conclusions.
How can I apply this thinking to my own situation?
Start by identifying the core principle behind the opinion, not the specific example. Then ask yourself: does this principle apply to my context? If yes, test it in a small, low-risk way before going all in.
How has this view evolved over time?
My thinking on most topics has changed significantly over the years. Early in my career, I held many conventional views that experience proved wrong. I try to update my beliefs when the evidence changes.