What if everything we assumed about AI in learning was wrong? We have the data from 10,000 students that proves it. The most effective learning strategies are the ones no one is talking about.
For the past six months, my team and I have been buried in data. We’ve been running a quiet experiment with an AI tutor we developed, and we just crossed a major milestone: analyzing over 10,000 individual student sessions. We didn’t go into this with a huge thesis. We just wanted to see what would happen if we gave students a powerful learning tool and then got out of the way. No heavy-handed curriculum, no rigid lesson plans. Just a smart AI, a student, and a chat box.
The data that came back didn’t just surprise us. It completely shattered some of our most fundamental beliefs about how people learn. The patterns we saw weren’t the ones you read about in EdTech blogs or academic papers. They were raw, counter-intuitive, and frankly, a little weird. But the numbers don’t lie.
I’ve built companies, invested in over 200 startups, including some of the biggest names in AI like Anthropic and OpenAI, and I’ve learned one thing: the truth is usually found in the messy, unexpected details. It’s not in the clean, top-down theories. It’s in the user behavior that makes you scratch your head. Here are the three most shocking patterns we discovered.
Pattern 1: The Socratic Dip
The prevailing wisdom in AI-driven education is that the tutor should be a gentle, encouraging guide. It should lead the student down a pre-defined path, correcting mistakes softly and always being agreeable. Our data showed the exact opposite.
The students who learned the most, and retained that knowledge weeks later, were the ones who argued with the AI. They challenged its answers. They told it when they thought it was wrong. They engaged in a real, Socratic debate.
We called this pattern the “Socratic Dip.” When a student started challenging the AI, their immediate scores on post-session quizzes would often dip slightly. They were spending more time arguing than just absorbing the material. But then, a week later, their retention scores were, on average, 35% higher than the students who just passively agreed with the tutor.
It makes perfect sense when you think about it. Passive learning is a myth. Real learning is an active, often confrontational process. You have to wrestle with an idea to truly own it. I see this all the time in the founders I invest in. The best ones aren’t the ones who agree with everything I say. They’re the ones who push back, who defend their vision, who have thought about the problem so deeply that they can debate it from any angle. They’ve done the work to make the idea their own.
One of our students, a high school junior named Alex, spent nearly half of a session arguing with the AI about the causes of the Peloponnesian War. The AI presented the standard historical consensus, and Alex, who had been reading a different historian, brought up a contrarian viewpoint. The transcript was fascinating. It was a genuine intellectual sparring match. Alex’s quiz score right after was just okay. But on the follow-up test a month later? He aced it. He didn’t just remember the facts; he understood the nuances. He had internalized the debate.
We’re now re-engineering our AI to be more provocative. We’re training it to occasionally take a devil’s advocate position, to push students to defend their answers, and to turn a simple Q&A into a real dialogue. The goal isn’t to be a friendly encyclopedia. It’s to be a sparring partner.
Pattern 2: Micro-Binging
Forget the 60-minute “deep work” study session. That’s another sacred cow that our data just slaughtered. We assumed the students with the longest session times would be our super-users, the ones with the best outcomes. We were dead wrong.
The students with the highest engagement and best long-term results were “Micro-Bingers.” They used the tutor in short, intense bursts, usually between 5 and 12 minutes. And they did it a lot. Some of them would have 10-15 of these micro-sessions in a single day.
What were they doing? They weren’t sitting down to “study for a test.” They were using the AI to solve an immediate, specific problem. They were stuck on a single math problem, and they’d jump in, work through it with the AI, and jump out. They had a quick question about a line of code while working on a project, and they’d use the tutor as an instant, interactive resource. It was learning on-demand, integrated directly into their workflow.
This is exactly how the best engineers I know operate. When we were building RemoteTeam, which was later acquired by Gusto, our top developers didn’t spend hours reading documentation. When they hit a wall, they’d find a solution with ruthless efficiency, often by pinging a colleague or searching a specific forum, get the answer, and apply it immediately. They were constantly learning, but in tiny, focused sprints.
Our data showed that these micro-binges were incredibly effective. The knowledge was being acquired at the moment of need, which meant it was immediately applied and reinforced. It wasn’t abstract information for a future test; it was a practical tool for a present problem. This context-driven learning sticks.
One user, a college freshman learning Python, had an average session length of just 7 minutes. But she was one of our most active users, with over 200 sessions in two months. Her project for the class, which she built while using our tutor, was in the top 5% of her course. She wasn’t “studying” in the traditional sense. She was building, and she was using our AI as her co-pilot, her on-demand expert, every step of the way.
This has huge implications for how we design learning tools. The future isn’t the hour-long, scheduled lesson. It’s the 5-minute, problem-solving sprint that happens right when you need it.
Pattern 3: The 'Wrong' Question Goldmine
This last one was the most surprising. In any structured learning environment, the goal is to keep the student on track. If the lesson is about photosynthesis, you don’t want the student asking about black holes. But our data revealed that the biggest learning breakthroughs often came immediately after a student asked a question that was completely “wrong” or off-topic.
We found that these tangential questions were a sign of a deeper cognitive process. The student was trying to connect the new information to something they already knew. They were building a mental model, and these seemingly random questions were them trying to find where the new puzzle piece fit.
And here’s where the AI shined. A human teacher, constrained by time and curriculum, might gently steer the student back on topic. But the AI could indulge the curiosity. It could find the thread connecting photosynthesis to black holes (perhaps through the lifecycle of stars and the elements they create). In these moments, the AI wasn’t just a tutor; it was a tool for intellectual exploration.
We saw this over and over. A student in a history lesson about the Roman Empire suddenly asked, “Is this like the blockchain?” A student learning about Shakespeare asked, “Did he have beef with other writers like rappers do?”
My first reaction when I saw these logs was to laugh. But then I looked at the data. The sessions where these “wrong” questions occurred had a 50% higher rate of follow-up questions and a measurably deeper level of engagement. The students weren’t being distracted; they were being inspired. They were forging connections that made the material more meaningful and memorable.
It reminds me of how innovation works in the startup world. The biggest breakthroughs don’t come from just optimizing an existing process. They come from connecting two seemingly unrelated fields. It’s the biologist talking to the computer scientist, or the artist talking to the engineer. That’s where the magic happens. It’s why I get so excited about companies like Hugging Face, which sits at the intersection of open source and advanced AI research.
These “wrong” questions are a goldmine. They are a signal of a curious, engaged mind at work. Our job as educators—and as builders of educational tools—is not to shut them down, but to follow them and see where they lead.
The Truth in the Numbers
After analyzing 10,000 sessions, I’m more convinced than ever that we are in the very early days of understanding AI in education. Most of what we think we know is based on old assumptions about classroom learning. We’re trying to fit a revolutionary tool into an outdated model.
The data is telling us to let go. It’s telling us that real learning is messy, argumentative, and happens in short, intense bursts. It’s telling us that curiosity is more important than curriculum. It’s time we stopped trying to build AI tutors that act like boring, patient teachers and started building ones that act like what they are: incredibly powerful tools for thought.
My biggest takeaway from all this? Don’t listen to the experts. Don’t listen to me. Look at the data. The students are showing us the way. We just have to be willing to watch, listen, and throw away our old maps.
Frequently Asked Questions
What's the most common pushback you get on this?
People often push back by citing exceptions or edge cases. And they're usually right that exceptions exist. But building a strategy around exceptions rather than patterns is a losing game for most founders.
How can I apply this thinking to my own situation?
Start by identifying the core principle behind the opinion, not the specific example. Then ask yourself: does this principle apply to my context? If yes, test it in a small, low-risk way before going all in.
How has this view evolved over time?
My thinking on most topics has changed significantly over the years. Early in my career, I held many conventional views that experience proved wrong. I try to update my beliefs when the evidence changes.