We’ve all seen it: an AI agent that works perfectly on a short task, only to fail unpredictably on a longer one. It feels random, but it’s not. A new model from Toby Ord explains why, and the answer is both simpler and more profound than you might think.
The paper introduces a brilliantly simple model that demystifies why AI agents struggle with longer tasks, reframing the challenge with a concept borrowed from physics: half-life.
In this deep dive, we’ll break down this game-changing framework to give you a clearer, more predictable way to think about building with AI.
Insights
1. A new unit of measurement: the “half-life” of an AI
The central thesis of the paper is that the declining performance of AI on complex tasks isn’t a mystery, but can be explained by a constant risk of failure over time. As Ord states:
The performance of AI agents on longer-duration tasks can be explained by an extremely simple mathematical model: a constant rate of failing during each minute a human would take to do the task.
This leads to a powerful new metric: the “half-life” of an agent. Just like in radioactive decay, this is the time it takes for half of the substance to decay. For an AI, “the 50% success rate time-horizon for each agent... is precisely the half-life of that agent.” It’s a single, intuitive number that tells you how long of a task a given model can handle before it has a 50/50 chance of succeeding.
2. The surprising “memoryless” nature of AI failure
Here’s the idea that will break your brain: the model is “memoryless.” An agent’s success on the first 10 steps of a task gives it zero momentum or advantage for step 11. Unlike a human who gains confidence, the AI faces the exact same risk of random failure at every single moment. Success is just the luck of avoiding a critical error for the entire duration.
Ord highlights how surprising this is:
Because the exponential distribution is the unique memoryless distribution... the chance of failing at the next moment is independent of how far you’ve come... This would be a surprising and interesting property for reasoning agents.
This frames AI not as a strategic thinker that gets tired or more confident, but as a process with a constant, ever-present risk of random failure. Success is simply completing the entire sequence without a single critical error.
3. A predictable, but brutal, pace of progress
This model is grounded in a key empirical finding: the capability of frontier agents is improving at a predictable, exponential rate. The research shows that “every 7 months, the length of task they can solve doubles.” This is the engine of progress driving the entire field forward.
However, the half-life model reveals a sobering reality about achieving true reliability. Moving from moderately reliable (50%) to highly reliable is a long, arduous journey. Ord’s analysis provides a forecast: “T99.9 reaches any particular length about 6 years after T50 does [each additional “nine” of reliability requires 2 more years...]” This quantifies the immense challenge ahead, showing that each “nine” of reliability is a multi-year effort.
4. Humans don’t follow the same curve
Crucially, this model of constant failure doesn’t seem to apply to us. The analysis found that human performance on long tasks decays far more gracefully:
Intriguingly, this human survival curve seems to be noticeably better than a constant hazard rate... the chance of succeeding over long timescales drops off more slowly for humans.
This suggests humans possess a fundamentally different, more robust error-correction capability that current AI agents lack, highlighting a massive opportunity for designing better human-AI systems.
5. A powerful rule of thumb for builders
Beyond the high-level theory, the model provides an incredibly practical heuristic for day-to-day work. It “provides simple rules of thumb for predicting success probabilities (e.g. that if you double the task duration, you square the success probability).”
If your agent has a 70% chance of success on a 10-minute task, you can quickly estimate it will have roughly a 49% (0.72) chance on a 20-minute task. This makes the abstract concept of exponential decay tangible and useful for project planning and setting expectations.
Takeaways
Measure your agent’s half-life. Stop relying on vague performance metrics. This week, identify a core task for your agent and run it 20 times. If it succeeds 10 times, you’ve just found your half-life duration! This single number will become your most important KPI for reliability.
Embrace the “memoryless” reality: Design your products assuming failure can happen at any moment, regardless of progress. This means architecting for fault tolerance, creating seamless human hand-off points, and building robust error-recovery loops.
Use the “double and square” rule: Use the insight that doubling task length squares the success probability to inform your roadmaps. Be realistic about taking on longer, more complex tasks and communicate these probabilities clearly with stakeholders.
Focus on the human-AI interface: The data shows humans are better at managing long-term tasks. Instead of aiming for full automation, focus your design efforts on creating a brilliant co-pilot that leverages human error-correction skills as a core part of the system.
Use the model as a diagnostic tool: If a new model’s performance doesn’t fit the simple exponential decay curve, investigate why. As Ord notes, “Deviations from exponential decay... might provide hints as to what they are doing wrong (or right).” This is how you spot true architectural breakthroughs.
The half-life model gives us a profoundly useful map for the journey ahead. It replaces ambiguity with predictability, providing a clear-eyed view of both the incredible pace of progress and the immense challenge of achieving true reliability. The path to building powerful, dependable AI tools is becoming clearer, lit by the steady tick of the half-life clock.
Of course, this clarity demands a new strategy. It means shifting our focus from chasing flawless autonomy to architecting resilient systems that embrace the constant probability of failure. It means treating our AI agents not as infallible minds, but as powerful, yet brittle, engines that require a brilliant human co-pilot to navigate the long journey.
This leads to the essential question we should all be asking our teams: Knowing failure is a constant risk, how do we stop designing for a perfect pilot and start building the ultimate co-pilot?
Share this post