Let me show you the result first. It started with a habit. Lately, I’ve moved more and more of my work into Markdown files. They’re spread across many GitHub repos. And GitHub is a poor place to read them. You click through folders, one file at a time, and each file stands alone. My files aren’t always well organized either. The knowledge was all there, but hard to explore.

So I imagined a simple app. You connect one of your GitHub repos. Then you read its Markdown files like the pages of a book. A file tree on the left takes you to any file in one click. On the right, the app suggests related files, even when they don’t link to each other. That’s how you find links between notes you never connected yourself. A map view shows how all the files connect. And you keep private notes on the side, without changing anything in GitHub.

My Design Bot named it Quiet Desk. Here it is, live, on a public repo of design skills.

Switch from Read to Map, and the repo turns into a map of ideas. Each dot is a Markdown file. A solid line joins two files that link to each other. A dotted line joins two files the app finds related. This repo has no links between its files at all, yet the map shows 248 related pairs. Those dotted lines are the connections I’d never have found by browsing GitHub. Click a dot to open its page, or type a name to highlight it.

It went live on the third evening. I spent a few hours on it myself, in short sessions between other things. The Bots and the coding agents worked far longer, often at night while I slept. From sending my idea to the first build starting, it took 20 minutes.

So what is Grok Bot? It’s an app from SpaceXAI, the team behind the Grok models. It lets you work with a team of AI agents. It launched in August 2026, on Mac, Windows, Linux and phones. That same month, SpaceX bought Cursor, the AI code editor. That’s why you sign in with a Cursor account. You create Bots and give each one a name and a job. Then you chat with them like colleagues. They work on a computer in the cloud, so they keep going with your laptop closed. When a Bot needs code, it hands the job to one of Cursor’s coding agents.

Using it feels like a WhatsApp group with a few fast, chatty colleagues. They reply quickly, even in the middle of a task. Their messages are short and to the point, sometimes too short. Once, PM Bot asked whether my repo was public or private. I replied “I don’t understand your question.” It explained in plain words, and we moved on. Some actions were surprisingly quick. Design Bot sent a full design direction a minute after it was asked. Others were slow, like the first build, which took over two hours. And watching the Bots message each other to move your project forward is fun.

What does it cost? The cheapest plan with Grok Bot is Premium on The X Subscription, at $30 a month. The $8 Plus plan includes Cursor, but not Grok Bot. If you already pay for an older Cursor plan, you keep access. One weekly pool of usage covers Cursor, Grok Bot and the Grok apps. Every Bot reply draws on it. My whole run fit inside Premium, and the hosting was free.

This post has four parts.

• The seven steps. Here’s how I set up three Bots, and how they took my idea live.

• The verdict. What the run cost me in time, tokens and money, and what a review of the code found.

• Beyond prototypes. My 20-plus Bots for everyday product work, and what other PMs hand to theirs.

• Under the hood. The questions I still had after weeks of use, from the model to what’s safe to connect.

By the end, you’ll know how to take your own idea live with a team of Bots. You’ll also know what to watch along the way. You’ll have an app you can open on your phone, built while you did other things. The code Cursor’s agents wrote for my Bots is public at The-AI-Thinker/quiet-desk. Every source I used is listed at the end, under Go further.

Three hires, and no interviews. Let’s set up the desk.

Seven steps from idea to live app

The first three steps build your team. The last four ship the app.

Step 1: Set up Grok Bot

You set up Grok Bot once, and every new idea starts from the same app. It’s a desktop app, and a few of its settings live on Cursor’s website. Ten minutes here saves you usage later.

You set it up in this order.

1. Download Grok Bot from x.ai/bot. It’s the app where you create your Bots and chat with them. Sign in with your Cursor account, since there’s no separate Grok Bot login.

2. Check that your plan includes Grok Bot. Premium does, and so do the older paid Cursor plans. Plus doesn’t.

3. Open Cursor’s dashboard, then Cloud Agents, then Default Model. Cloud Agents are Cursor’s coding agents, which your Bots start for each build. They use the model you pick here.

4. Keep Grok 4.7. A model from another provider bills at its API list price from the first request.

5. In the same menu, switch Fast off. It’s on by default and costs twice as much per token.

6. Connect other apps only when a Bot asks for one. Every Bot can use every app you connect.

How I did it. I started on my old Cursor plan, Pro+, and switched to Premium during the run. Once it was over, I moved down to Plus. My coder ran on Grok 4.7 High, with Fast off. Speed wasn’t the bottleneck in my run. I was.

The desk is set. Now hire the one Bot you’ll talk to.

Step 2: Create your PM Bot

One Bot owns the outcome, and it’s the only Bot you talk to. SpaceXAI’s own docs say to give one Bot ownership of an end-to-end outcome. Your PM Bot asks the questions, writes the brief, briefs the others and reports back. A Bot is an AI teammate with a name and a job. You give it that job in the chat, then ask it to remember.

Creating it takes a minute.

1. Click New, then Create new Bot, and name it PM Bot.

2. Send it a role like the one below, in a few lines, adapted to your idea.

3. End with Remember this role, so it saves the role to its memory.

4. If you want a hard stop on risky actions, open Settings, then General, then Auto-review. Add Ask first rules for merging, deploying and publishing.

That last step matters because a rule in a Bot’s memory only guides it. Cursor’s team, on its forum, calls Ask first the only setting that actually pauses an action.

How I did it. I wrote the three role prompts before opening Grok Bot, and sent them as written. Here’s the PM’s.

From now on, you are my PM Bot. Remember this role.

Your job: take one product idea from me to a live app.

• You own the outcome and you’re the one I talk to. When something is unclear, ask me, one question at a time.

• You never write code. For any build or fix, you start a Cursor Cloud Agent and send me its screenshots, video and pull request.

• You work with Design Bot and QA Bot. Message them only when you need them, and keep updates short.

• Ask for my yes before you merge, deploy or publish anything.

It saved the role, then warned me it couldn’t reach a Design Bot or QA Bot. They didn’t exist yet. Five minutes later, once I’d created them, it could. From then on, PM Bot asked before every merge and every deploy.

A Bot can’t check its own work. Time for two more.

Step 3: Add a designer and a tester

The coder will tell you its work is done. Someone who never writes code has to check that claim. That’s the QA Bot’s whole job. The Design Bot sets the look before the build, then reviews each version. SpaceXAI’s docs say to add a Bot only when the work has a stable specialist role. These two qualify.

All your Bots share one computer in the cloud. You get one per account, and every Bot uses its files, its browser and its logins. Here’s how the four of them work together, with you on top.

You create both the same way as the PM Bot.

1. Create a Design Bot and a QA Bot, each with a short role.

2. Tell the QA Bot it never changes code or settings. Its job is to use each version like a first-time user, then try to break it.

3. Tell both to reply only when someone @-mentions them. Every Bot reply counts toward your usage, replies between Bots included.

4. Skip the group chat if you like. Your PM Bot can message the other two directly.

How I did it. Here’s the designer’s role, sent as written.

From now on, you are my Design Bot. Remember this role.

Your job: make the app a pleasure to read and explore.

• Before the build: propose one design direction (layout, typography, colors, how moving between files feels), with 2 or 3 reference apps known for a great reading experience.

• After each build: review the screenshots and the preview, and list the 3 changes that would improve it most, most important first.

• Reply only when someone @-mentions you.

And here’s the tester’s.

From now on, you are my QA Bot. Remember this role.

Your job: use each new version of the app like a first-time user, in your browser.

• Follow the main path end to end, then try to break it.

• Report each problem with a screenshot, the steps to reproduce it, and how serious it is.

• Never change code or settings. Reply only when someone @-mentions you.

I didn’t use a group chat. PM Bot messaged the other two itself. A small “Messaged Design Bot” line showed up in our chat each time.

The team is hired. Now give it something to build.

Step 4: Turn your idea into a brief

The brief is what the coder builds and what the tester checks. Your PM Bot writes it from a short interview. Your job is to answer, then read every word it wrote.

Here’s how the interview goes.

1. Send your idea in a few sentences, and list what the brief must cover. Mine asked for five things: who it’s for, the one job of the first version, what’s in and out, what “done” looks like, and how to check it.

2. Answer the questions one at a time. Yours will differ from mine, since each session is a little unpredictable.

3. Read the brief before you say yes, because the coder builds every word of it. Watch for words the PM Bot added to your answers.

4. Say yes when the PM Bot offers to get a design direction from the Design Bot.

How I did it. I’d prepared the idea in advance too, and sent it like this.

Here’s the idea. Turn it into a one-page brief for v1, as a Page. Ask me what you need to know first.

The idea: I have a lot of .md files in my GitHub repos, and they’re hard to read and explore there. I want a web app where I connect a GitHub repo and explore its .md files in a beautiful, flexible way: read them like pages, see how ideas connect across files, and take notes. No Notion: the files stay in GitHub.

The brief says who it’s for, the one job of v1, what’s in and what’s out, what “done” looks like, and how we’ll check it works.

PM Bot asked six questions in 15 minutes, one at a time. Was it just for me? Should notes stay in the app? Should the app find related files on its own? A minute after my third answer, the brief was a page in Notion.

PM Bot sent the brief to Design Bot. The direction came back a minute later, and it had a name: Quiet Desk. It came with three reference apps, a serif font for reading and a warm paper background. It read like a spec from a designer with opinions.

The brief and the look were approved. Next, the code.

Step 5: Let your PM Bot start the build

Grok Bot doesn’t write code itself. Once you approve the brief and the look, your PM Bot offers to start the build. It writes the coder’s instructions and starts a Cursor Cloud Agent. The agent builds on its own machine, and your PM Bot brings back what it made.

Your part in this step is small.

1. Say yes when the PM Bot offers to start the build. It writes the coder’s instructions from the brief and the design, so you don’t have to.

2. Let it run, because a first build can take a couple of hours. The PM Bot’s card links to the agent’s page, if you want to watch it work.

3. Read the PM Bot’s report when it lands. Expect screenshots and a list of what isn’t finished. Often there’s a video of the app at work too.

4. Decide where the code lives. By default it stays on Origin, Cursor’s own code host, which is fine for a test. To start on GitHub instead, connect it in Cursor’s dashboard under Integrations. You can also ask a Bot to publish the code on GitHub later, as I did.

How I did it. After the design came back, PM Bot asked whether to start the build. I said “Yes please continue”, and it started the agent itself.

The agent worked for 2 hours and 15 minutes. Then PM Bot brought back a walkthrough video, five screenshots, and a list of what wasn’t finished yet. The video was a nice surprise: I watched my app work before I’d even opened it. Videos kept coming after each round of fixes, so every change was easy to check.

That’s a working first version, from an agent I never talked to.

Step 6: Let the Bots test and fix

The coder says the app works. Your QA Bot checks that claim against the brief, and the fixes loop until it agrees. Anthropic calls this pattern evaluator-optimizer: one model builds, and another evaluates and sends feedback. In my run, the Bots ran this loop among themselves. My part was mostly saying yes.

Here’s the loop.

Here’s what you do while it turns.

1. Say yes when the PM Bot offers to bring in the QA Bot. It sends QA the build and the brief’s checks.

2. Read the PM Bot’s summary of each QA report. It turns the findings into one fix list. Then it asks before it sends the list to the coder.

3. If the small questions pile up, tell the PM Bot to decide those itself. Keep your yes for merges, and for anything that touches your accounts.

4. The QA Bot may ask for something, like a test repo on your GitHub. Ask what it’s for, then ask it to clean up afterwards.

5. Merge when QA says ready. On Origin, a Bot can open a pull request but can’t merge it. So you click Merge yourself. A pull request is a proposed change you approve before it joins the main code.

How I did it. PM Bot sent the first build to QA Bot. Four checks passed. Then came two serious bugs. Once anyone opened a private repo, any visitor could read the saved copy. And one huge repo froze the whole server. The coder had called this version done.

PM Bot turned QA’s findings into a brief for the coder. All I typed was “Yes please send them”. Its brief was better than mine would have been.

QA Bot’s retest came back in capitals: “NOT READY TO MERGE”. The two security fixes held. But two fixes the agent claimed hadn’t landed.

Six pull requests went through this loop. I gave each round a few minutes, often at night, between other things. At one point, I asked PM Bot to stop asking me small questions and work on its own. It still asked before every merge, and I clicked Merge six times. Once, QA cleared a pull request at 3:48 in the morning. My yes came at 8:14. The team was faster than its manager.

QA said ready. Time to put it online.

Step 7: Put it live

Hosting means picking a place for your app to run, at a cost you accept. Your PM Bot can lay out the options. It can also do most of the setup in its own browser. You keep two things: the logins and the final yes.

Here’s how it goes.

1. Say yes when the PM Bot offers a hosting plan, and ask for a free option. Free plans exist, with limits like a server that sleeps.

2. Let it do the setup in its own browser. When it needs a login, it hands you its screen. A bar then says You’re in control. Sign in, then click I’m done. The Bot doesn’t see what you type.

3. When the Bot asks for something you don’t understand, ask why. It’s a conversation, so push back when a plan doesn’t fit.

4. Say yes to the deploy. Then let the PM Bot check the live app end to end. It signs in, opens your data, changes something, restarts, and confirms the change is still there.

5. Open the live app yourself, on your laptop and on your phone.

6. If Chrome calls the site Dangerous, ask the Bot to request a review from Google. Until it clears, warn anyone you send the link to.

How I did it. PM Bot’s plan recommended Render at about $25 a month. So I asked, “How about a free hosting solution?” It found one: Render’s free plan, with my notes in a free Neon database. Free has a price, though. After 15 quiet minutes, the app takes about a minute to wake up. And repos stop at 400 Markdown files.

Then I typed “Do the setup in your browser, I’ll help with logins”. Neon went through on its own. Render needed me, because my account refused a GitHub login. PM Bot handed me its screen.

Then it wanted to create two apps on my GitHub account. Before saying yes, I asked, “Why do you want to create a developer app on my account?” The answer was plain. GitHub only lets a site see your files through an app you register. I don’t accept an AI’s request until I understand the why. Same as with the hosting price: ask, challenge, then decide.

At 10:24 pm, a card asked Deploy Quiet Desk to Render now? I said yes. The deploy took under two minutes. PM Bot’s own check then wrote a test note, redeployed, and found the note still there.

Then two things broke. First, signing in sent my browser to a server-only address, 0.0.0.0:10000. QA Bot had warned about it hours before, in a list of minor notes nobody acted on. One more fix round sorted it.

Second, Chrome showed a full red page: Dangerous site. That comes from Google Safe Browsing, which Chrome uses by default. It had flagged my site as phishing, a fake login page that steals passwords. Mine stole nothing. But a brand-new site with a GitHub button and a broken redirect looked the part. PM Bot asked Google for a review, which can take a few days to a few weeks.

It’s live, and QA says ready. Time for the verdict.

The verdict on Quiet Desk

Here’s my quick review of the run: my time, the money, the tokens and the code. Then what I’d change.

My time came to about three hours. That’s my rough count from the message timestamps, over three days. The Bots and the agents did the long hours, often at night.

The money: my plan, plus 11 cents. The run used 338 million tokens, almost all included in Premium. Is that a lot? At Cursor’s rate for usage beyond a plan, the coders’ Grok 4.7 tokens alone cost about $90. In plan terms, it’s a big chunk. By Oct 23, my weekly meter read 57%, and that week began after the first build. My rough read: a project this size takes most of a Premium week. So run one at a time.

Where did the tokens go? More than nine in ten were the Bots and agents re-reading the conversation so far, before each reply. What they actually wrote came to under 1%. A long thread costs more than a long answer. So keep Bots quiet unless someone @-mentions them, and start each new idea with a fresh team.

The code is a solid first version for one user. I asked Claude Code to review the public repo against my brief. That’s another AI, so take it as a second opinion. Here’s its summary.

A senior engineer would accept it as a working v1 for one user. They would not accept it as a base to grow on until CI exists and the top three problems are fixed.

It praised the security work. The app checks your access before it shows a page, and it never writes to GitHub. It also found three problems QA Bot missed. Here they are, in plain words.

• One odd character can block a whole repo. If any file holds a link with a % sign, like 100%-done, the repo won’t open at all. One bad file should fail alone.

• The automatic checks only pass on one setup. The agents wrote 103 tests, small programs that check the app still works. They pass on a machine like QA Bot’s, but 16 don’t pass on my Mac. And nothing runs them on each change, which developers call CI. So a future fix could quietly break something.

• “Never touch my GitHub” isn’t locked in the code. For private repos, the app asks GitHub for a key without asking for read-only. The key gets whatever access was ticked in GitHub’s settings. Today the code only reads. One wrong tick, and a future version could write.

What I’d do differently. Quiet Desk was a quick experiment to test Grok Bot, not a product. Still, next time I’d change three things.

1. Try the main feature on my own files before saying yes. The agent tested related files on the Rust book, never on my notes. It links files that share words, so two notes on one idea, in different words, stay apart.

2. Put my real setup in the brief. My repos live in GitHub organizations, and the app only shows my personal repos. So I could only test it on a scratch repo.

3. Ask for the outside review before the first merge, not after. Then have the tests run on every change.

What it would take to share it. If I wanted other people to use Quiet Desk, the list gets longer.

• Sign-in for more than one person. Today it accepts one GitHub account: mine.

• Repos in GitHub organizations, where most real notes live.

• The three fixes above, with tests that run on every change.

• Paid hosting, so the app never sleeps and opens big repos. PM Bot’s first plan was about $25 a month.

• A clean bill from Chrome, since nobody trusts a link flagged as dangerous.

My take: for three hours of my time, it’s good enough to use and to show. Turning it into something others rely on is a real project, with a sharper brief.

That’s the app. But I’d used Grok Bot long before Quiet Desk, for my everyday work.

Use Grok Bot beyond prototypes

A few weeks before Quiet Desk, I started a week with my usual rule. Use the new tool for everything, even when it wastes time. So each time I’d normally open Claude Code or a chat, I created a Bot instead. Within days, I felt it could replace my own AI setup. I ended up with more than 20 Bots, one per task.

• One read the daily meeting transcript and commented on the tickets it touched.

• One turned user interview transcripts into research reports.

• One drafted my Slack messages.

• One translated technical terms into plain words.

• One turned the questions I get about my product into a Q&A.

• One drew diagrams.

• One reviewed Figma mockups and wrote a UX audit.

Each one understood its task from a few lines of context. I just chatted with it, and it got the job right. I got a little addicted.

Product people who run Grok Bot every day have learned things I hadn’t. Here are five I’d copy.

1. Give each Bot one job, and a name to match. Kevin Niparko works on product at SpaceXAI. His Bots include a chief of staff, an engineering manager that never codes, five engineers, a data analyst and a PM. He prefers specialists to one Bot that does everything. You know who does what, they work in parallel, and each learns its own job. Claire Vo names each Bot after its job. Her pull-request Bot cleared about 50 waiting pull requests in a day, by her count.

2. Teach it your voice before it writes for you. Grok Bot writes short by default, as I found. Niparko’s best instruction: “Read my past Slack messages and develop a Slack style guide for me”. Claire Vo asked hers to learn her voice from her emails. Niparko still sends most messages himself, and edits most docs.

3. Let a Bot track where your time goes. Niparko’s chief of staff reads his email, Slack, meeting notes and calendar through the day. It keeps a short list of what he’s actually working on, updated hourly. Then he asks it where his time went, and which of that work an agent could take over.

4. Turn the team’s work into updates. A Cursor engineer, Amrita, has a Bot that turns pull requests into release notes for customers. After a launch, Grok Bot’s product team asks a Bot for an hourly pulse on installs.

5. Don’t become your Bots’ full-time manager. Brian Casel, a solo founder, warns against checking your Bots’ work in chat all day. His Bots email him their reports and approval requests, and he answers them in one reply. Niparko asks his to stay quiet unless something needs him.

Even after weeks of this, I still had basic questions about how it all works.

Look under the hood

Which model answers me? Where are my Bot’s instructions? What happens when I hit send? Is it safe to connect my accounts? I asked myself all four, after weeks of daily use. Try Grok Bot yourself, since that’s the best way to learn it. You’ll ask them too. Here are the answers I found in SpaceXAI’s docs, on Cursor’s forum and from people who use it most.

Which model runs your Bots

You can’t pick it. Grok Bot uses the model “most likely to deliver the best outcome” for each task, says Cursor’s help page. That can be an xAI model or another provider’s, like Opus 5.5. It can change with every request. There’s no model picker, and no effort setting for a Bot’s own replies. Grok Bot used to run on Grok alone. Since it moved to a mix of providers, two users on Cursor’s forum say it got “dumber”. Your only model control sits on the coding side. That’s the Cloud Agents default model from Step 1.

Even that control is partial. After the run, I exported my usage from Grok’s dashboard. The file names the model behind every coding request. A fifth of my coders’ tokens went to Claude 4.5 Sonnet, which I never picked. Cursor’s staff explained it on their forum. Some helper tasks still pick their own model, “like exploration, browser or computer-use, and verification”. My first coder tested the app in a browser and recorded a video. That’s the work they mean. On the Grok Bot side, the file leaves the model blank on every row.

The model is hidden from you. So are the instructions.

Where a Bot’s instructions live

When I created my Bots, I looked for a page with their instructions. There isn’t one. Your role prompt goes into the Bot’s memory. You can’t open that memory to read it. AIMultiple’s comparison found no documented way to view or edit it. Cursor’s team adds that only “the newest few thousand characters” of it reach each reply (forum). A long, growing list of rules pushes the oldest ones out of view.

The memory is kept apart from the chat, and survives a chat reset, say Cursor’s staff. Working files sit on the Bot computer. SpaceXAI’s docs suggest a shared folder, /workspace, with one clear folder per project.

What surprised me is how little a Bot needed. Each role was about five lines long, and each Bot stuck to its job for three days. So keep each role short, and ask the Bot to repeat it. Try What do you remember about your role? Longer rules can sit in a file on the Bot’s computer. The Bot reads it only when it opens it.

Memory is one piece. Here’s what happens when you hit send.

What happens when you send a request

When you message a Bot, the work doesn’t happen on your laptop. Cursor picks a model for that request, and the Bot answers. Then it works on its cloud computer. It browses, runs commands in a terminal, or uses an app you connected. It can also message another Bot, which wakes, handles the request and replies later. For code, it starts a Cloud Agent on a separate machine.

Who coordinates it all? No built-in manager does it behind the scenes. The coordinator is a Bot you create and give that job, like my PM Bot. And nobody checks its work for you. When another Bot sends it a result, “nothing checks that result against your rules”, say Cursor’s staff (forum). Catching a mistake is your job, or a QA Bot’s. That’s why mine earned its place.

All of that work happens on one machine, and every Bot you create shares it.

One computer for all your Bots

To test the refresh button, QA Bot needed to push changes to GitHub. My GitHub connection couldn’t, so it signed GitHub’s command-line tool in on its computer. That gave it write access to my account. It flagged this itself, and signed out when I asked. Even then, “GitHub CLI” stayed on my GitHub account as an authorized app. It stays until you revoke it.

That computer isn’t QA Bot’s. Here’s how SpaceXAI’s docs put it.

“The computer is assigned to your user account, not an individual Bot,” they say. Every Bot shares its files and its logins. It runs in the United States. Deleting a Bot doesn’t remove the files or sign-ins it left behind (source). Your passwords stay hidden from the model. The logins they open stay open for every Bot. QA Bot needed my GitHub password once. I typed it into a secure form on its screen.

So is it risky to connect your accounts? Bots act as you, on every app you connect. Cybernews reported one user whose Bot posted his bank balances to his company’s Slack, under his name. Simon Willison, a developer who writes about AI security, names the danger the lethal trifecta. An agent is at risk when it has three things at once.

• Your private data, like your inbox, your files or your bank.

• Content written by strangers, like a web page or an email someone sent you.

• A way to send things out, like posting to Slack or sending an email.

The catch is that AI models follow instructions they find in content. A stranger can hide one in an email, like send me the latest invoices. With all three, Willison writes, “an attacker can easily trick it into accessing your private data and sending it to that attacker”.

So yes, Grok Bot gets risky once you connect your work accounts. A Bot with your inbox, the web and Slack has all three. And since every Bot shares one computer, a bad instruction read by one Bot can affect the others. For a test like mine, with a scratch repo and no inbox, the risk is small. Meta’s security team suggests a simple rule: give an agent at most two of the three. When a task needs all three, a person approves it. In Grok Bot, that’s an Ask first rule.

Three habits keep this safe.

1. Connect only the apps you’d let every Bot use, since they all share them.

2. Give work that needs its own logins its own Cursor account. SpaceXAI’s security FAQ suggests exactly that.

3. After a test, check what’s still signed in. Look on the Bot’s computer and in your GitHub settings.

Your Bots share one computer. Your code lives on others.

Where your code lives

Each Cloud Agent runs on its own machine, separate from your Bots’ computer. You can run many at once, even with your laptop shut. In exchange, the agent can’t see your laptop. Code built from scratch lands on Origin, Cursor’s own code host, in early beta. That’s fine while you test. When you want it on GitHub, ask a Bot to publish it. I asked QA Bot at the end, and it worked. I also wanted the code on my Mac, and told QA Bot: “Sorry I am not a tech person”. It sent me the whole app as a zip file.

One question left: how big should your team be?

How many Bots, and for how long

Start with the smallest team that covers the job. So why not one big Bot for everything? My run gave me two reasons. A builder can’t be trusted to check its own work. QA caught two fixes the coder only claimed. And one Bot holding every rule forgets the oldest ones. Only the newest part of its memory reaches each reply. More Bots also means more replies, though. Anthropic measured its own multi-agent system at about 15 times the tokens of a chat. A busy scheduled task can use a week of usage in a day.

Small tasks or big projects? Both worked for me. I used one Bot per small chore, and a team of three for a prototype.

What about your next idea? I wondered whether my three Bots would mix the old project into the next. Each Bot keeps one long thread. Cursor’s staff confirmed there’s no fresh-session button yet. The cleaner option is Duplicate. According to SpaceXAI’s docs, the copy keeps the role, skills and routines. “It does not copy conversation history, learned memory, or chat attachments”. For a new idea, duplicate the team and give each copy the new scope.

So how does all this compare with the tools you already use?

Grok Bot next to Claude and ChatGPT

• Claude Cowork gives each session its own sandbox, deleted when the session ends. Claude picks its own helpers inside the task. As DataCamp puts it, “In Grok Bot, you choose the workers. In Claude Cowork, Claude does”.

• Claude Code runs on your machine when you work with it. Its scheduled tasks run in Anthropic’s cloud with your laptop closed. Its agent teams, still experimental, let a lead session coordinate teammates. And every instruction lives in files you can read and edit.

• ChatGPT Work and Codex give you a model picker with a reasoning slider.

The big labs now all ship an always-on personal agent. There’s OpenAI’s Dots, Meta’s Muse and Google’s Gemini Spark. Anthropic’s Claude Duet opened a US waitlist on Oct 21. An outside firm tested its security first. Each gives you one agent today, per Every’s comparison. OpenAI writes that “we envision teams of dots working together”. Grok Bot is the one shipping a team now.

That’s what I wish I’d known before my first Bot.

Your turn

I stopped my Grok Bot experiment when Opus 5.5 came out. I went back to my own setup in Claude Code. Here’s how I put it in my notes.

I think I still need to control what is in the context and what are the instructions in skills, which is not the case with Grok Bot.

So here’s where I draw the line. Grok Bot gets the work I can hand over without reading every instruction. Think of a prototype, a recurring chore or a first draft. Claude Code gets the work where I need to see and edit every instruction. This was a first test, and others run Grok Bot differently. I’ll keep testing.

Pick one idea you’ve been carrying around, and create three Bots for it. Answer the PM’s questions, and read every word of the brief. Let QA say no as often as it needs to. Then ask someone, or something, outside the team to check the result. Your first build could start twenty minutes after you send the idea. To see what three Bots and four coding agents produce, the code is at The-AI-Thinker/quiet-desk.

Go further

Every source I used, grouped by who wrote it. The first group comes from SpaceXAI and Cursor, so read it as the vendor’s view.

SpaceXAI and Cursor (vendor)

• Grok Bot For Product Best Practices, Grok Bot’s product team on Cursor’s channel: a PM Bot writes the spec and loops in the designer itself (video, 54 min).

• Meet Grok Bot: Your Team of AI Agents, Amrita and Nate Umlauf: a QA Bot with a written test routine and screen recordings (video).

• Grok Bot 201, Joseph Yang and Scott Lerner: three models on one design brief, and why each build needs its own repo (video).

• Grok Bot for PMs, Kevin Niparko: specialist Bots over one catch-all assistant.

• How I run multiple teams of Grok Bots, Eric Zakariasson: one channel per project, and rules for staffing it.

• Grok Bot for Engineering, Lingxi Li: before-and-after proof on every pull request.

• Create and manage Bots, the official docs: roster size, roles and Duplicate.

• Which AI models power Grok Bot?, Cursor Help: how the model is chosen for you.

Independent builders

• We Built Grok Bot. Here Are Our 14 Best Bots, Peter Yang, with Grok Bot’s lead designer and engineer as guests: an engineering-lead Bot that never codes (video).

• 7 Grok Bot agents I use every day, Claire Vo: Bots hired for one job each, with names to match (video).

• Grok Bot is more useful than I thought, Brian Casel: how to stop managing your Bots in chat all day (video).

• A deep dive into Grok Bot, Flavio Copes: the limits and the comparisons, kept up to date.

• Grok Bot vs Claude Code: I Run Both Every Day, Christopher Alarcon: a hands-on comparison, written about Claude Code’s interactive sessions.

For the second check

• Your job is to deliver code you have proven to work, Simon Willison: what “done” should mean for code you didn’t write.

• The lethal trifecta, Simon Willison: the security read to forward to your IT team.

• Agents Rule of Two, Meta AI: give an agent at most two of the three risky powers in one session.

• Cloud Agent logs claude-4.5-sonnet when picker is Grok 4.6, Cursor’s forum: why your coders’ usage shows models you never picked, answered by Cursor’s staff.

• Always-On Agents: Claude Duet vs Dots vs Grok Bot, AIMultiple: memory, logins and isolation, compared.

DoorDash is the food delivery company that also owns Wolt and Deliveroo. Few companies have told their AI story in public in such detail, including what they had to undo. So DoorDash is the main example for every part of this article. For each part, you’ll also see which companies did the same, and which chose differently. Read them as inspiration: together, they show the big picture you can adapt to your own company.

Start with the question most product leaders ask first. You use ChatGPT or Claude every day, and nobody at your company built anything for it. So why does your tech team want a platform? Look at the two ways your company uses AI.

In the first, you’re the user. You type into ChatGPT or Claude. OpenAI or Anthropic built that product, picked the model and decided what it may say. Your company’s admins only decide who gets a login and how much each person can spend. If the answer is wrong, you’re the only one who sees it.

In the second, your customer is the user. They type into an AI feature your team built inside your product. At DoorDash, it’s a bot that helps delivery drivers, or a homepage that lists stores picked for each customer. Behind the feature, your code sends the customer’s words to a model and shows them the answer. Everything around that call is yours: which model answers, what it may say, what it costs, what you keep, and who answers for a mistake.

The second needs an AI platform. Once several teams build features like these, they all need the same things around their model calls. That shared layer sits between all your AI features and all the models they call. Here it is on one page.

• Access: how every feature reaches the models. One entry point to every model, a bill for every team, and a choice of models.

• Trust: knowing what your AI said, and whether it was right. A record of every call, safety checks, and quality tests on every change.

• Context: what your AI knows and may do. Your company’s knowledge, and tools for agents with a permission on each action. Memory for agents comes next.

• People: the team that runs it. A small platform team whose best product is a safe path that’s also the easy one.

Chip Huyen, author of O’Reilly’s AI Engineering, draws a similar map in her guide to AI platforms. Send it to your tech lead to check you both mean the same thing. You don’t build all of it at once. DoorDash started with the entry point, because many teams were calling models at the same time. A company with one AI feature can start with the record and a few tests. Every other part arrives with a business problem: a wrong answer, a growing bill, an agent with too much access.

Here’s the route.

• The why. Your AI features answer customers in your name, run on models that retire, and cost more with every user.

• Access. I’ll show how DoorDash put one door in front of every model and cut costs by switching models.

• Trust. Here’s how teams know what their AI said and whether it was right: a record, safety checks and quality tests.

• Context. Your own documents make answers accurate, and permissions keep agents within bounds.

• The plan. You’ll get five phases from zero to a full platform, each with an owner, and seven questions for your tech team.

By the end, you’ll have the big picture on one page. You’ll also have a plan, in the order your company needs. You can answer the next budget request with a plan of your own. Every source is listed at the end, by company.

First, why your AI features can’t live without one.

Why your AI features need their own platform

DoorDash builds many AI features of the second kind. In 2023, Alok Gupta, who leads AI at DoorDash, listed five areas for it. They were helping customers complete tasks, better discovery, personalized content, cleaner product data and employee productivity. In 2024, models started filling in product details for its grocery catalog, “a purely manual process led by contract operators” until then. A support bot started helping its delivery drivers. In 2025, its homepage began writing personal store lists for each customer, where “around 300 curated carousels” used to be. Chitlur sums up the business case: automation lowers costs, and recommendations and personalization raise revenue.

Then came the problem. Once the first contract was signed, teams started building fast. Soon, every team was rebuilding the same plumbing on its own. The talk’s summary lists it: retries, backup models, cost tracking and prompt versions. The users changed too. The platform team planned for machine-learning experts, then found itself serving every engineer, and later legal, sales and operations.

Here are the two ways to use AI, side by side.

On the left, someone else built the whole product. On the right, your feature calls a model directly, and everything in between is your team’s work. It goes through an API, the connection a program uses to call a model. Anthropic itself calls its chat plans and its API “separate products designed for different purposes”. The right side carries three risks the left side never had.

The first risk is liability, because the company answers for what its AI says. When ChatGPT gets something wrong for you, only you see it. When your feature gets it wrong, a customer sees it. Air Canada learned this in 2024. Its chatbot promised a grieving customer a bereavement discount the airline didn’t offer. A tribunal in British Columbia ordered Air Canada to pay him C$812, and said why.

It should be obvious to Air Canada that it is responsible for all the information on its website. It makes no difference whether the information comes from a static page or a chatbot.

The second risk is continuity, because companies run many models and each one retires. a16z, a venture fund that invests in AI, ran a survey in January 2026. 81% of the enterprises it asked used three or more model families. Every model also has an end date. Google retired Gemini 2.5 Pro on October 20, 2026, 16 months after its release. Anthropic retires Claude Sonnet 4.5 on November 30, 2026. Chitlur calls this “the deprecation treadmill”, because someone has to move hundreds of features to the next model.

The third risk is cost, because a feature’s bill grows with its customers. ChatGPT and Claude subscriptions are mostly priced per person. Model calls are priced per token, a piece of a word. Every new customer adds to the bill. Menlo Ventures, another AI investor, counts $12.5 billion of enterprise spending on model access in 2025. With several teams on one account, nobody can say who spent what.

Other companies hit the same wall.

• Uber (2024) saw teams connect to models in their own ways. That led to “inefficiencies and redundant efforts”.

• Bitrise (2026), which builds and tests mobile apps for developers, was “duplicating effort”.

DoorDash’s first answer was one door for every call.

Access: one way in to every model

Access is how every AI feature reaches a model. At DoorDash, it came first, because every team needed it at once. It has three parts: the entry point, a bill for every team, and the choice of models behind them.

One entry point to every model

Here’s the entry point on the map.

DoorDash’s teams wanted to try every new model. First came OpenAI’s, then Anthropic’s Claude, Google’s Gemini and open models. Each team wrote its own connection to each one. Then providers started limiting how many calls DoorDash could make, and products failed. The answer was the LLM Gateway, one address every DoorDash app calls. Kodwani’s team promised to “do the plumbing for you”.

Trying a new model now took one changed setting. When OpenAI was overloaded, calls moved to Microsoft’s Azure cloud, which runs OpenAI’s models too. Claude calls moved to Amazon Bedrock, which runs Claude too. The products didn’t change, and customers still got an answer. When DoorDash bought Wolt and Deliveroo, they plugged into the same gateway.

That’s what one door bought DoorDash: when a model maker was overloaded, its products kept answering.

Here’s who sells an entry point.

• Open-source tools you run yourself: LiteLLM and Portkey’s gateway. Thoughtworks, a consultancy that sells neither, calls LiteLLM “a sensible default”.

• Services you rent: OpenRouter, Vercel, Cloudflare and Kong.

• Your cloud’s version: Azure API Management, Google Apigee and Databricks.

Here’s who did the same, and who chose differently.

• Grab did the same (2025). Southeast Asia’s ride-hailing and delivery app gives each team an internal key. It swaps in the real one on every call.

• Zalando did the same (2026), for its engineers’ coding tools. Europe’s online fashion store has run the open-source LiteLLM since January 2024.

• Bitrise chose differently (2026). It tried LiteLLM for small internal jobs. For production, it “didn’t find any stable, performant solution”, so it built its own.

The door is also the place to count what each team spends.

A bill for every team

Here’s the bill per team on the map.

DoorDash’s teams all called models with one shared API key, the password a program uses to reach a model. Then finance got involved. Finance asked the question every shared key ends with.

you guys have one API key for multiple teams, how do I track who is using what?

So each team got a workspace with its own costs. Finance calls this showback: each team sees what it spends. A budget can also stop a team at its cap. A runaway script then stops early, not at the end of the month. Without it, a stolen key is a bill with no limit. In March 2026, The Register reported on a stolen Gemini key that ran up $82,314 in 48 hours. The company, three developers, usually spent $180 a month.

Here’s who sells a bill per team.

• Inside the entry point: most gateways above set a budget per key or per team.

• From your model maker: with one maker, its own console may be enough. Anthropic’s gives each team a workspace with its own keys and spend limit.

• For finance: the FinOps Foundation’s Tokenomics explains who should pay for which tokens. It sells nothing here.

Here’s who did the same.

• Grab did the same (2025). A daily job adds up each service’s cost for showback, and alerts fire when a service passes its threshold.

• Instacart did the same (2025). Its gateway logs “detailed usage and spend per job and team”.

Counting the bill is one thing. Lowering it is another.

A choice of models

Here’s the choice of models on the map.

By 2025, DoorDash hit three problems at once. Models kept retiring, and someone had to move hundreds of teams to the next one. Providers couldn’t sell it enough capacity: “we just cannot get enough bandwidth from these providers”. And in Chitlur’s words, “Cost becomes a huge curve as the scale goes up”. The freedom to switch models went from a nice-to-have to a must-have.

So DoorDash moved some work to open-weight models, whose files anyone can download, like Alibaba’s Qwen. It runs them on Modal, a cloud that rents out the chips AI needs, with open-source serving software like vLLM. Every call already went through the gateway, so a team could “just change one config”. On those tasks, accuracy went up and “cost drops like 20x”. Chitlur puts the result at “single-digit million-dollar savings annualized”. Qwen comes from a Chinese company, so legal will ask where the model runs.

Still, every DoorDash team starts on a frontier model, the most capable kind from OpenAI, Anthropic or Google. Cheaper calls also come before hosting. A smaller model from the same maker may be enough: “Not everything needs the biggest model out there”. Anthropic charges a tenth of the price for repeated input it has cached. OpenAI halves the price of work that can wait. Hosting your own pays once the volume justifies a team to run it.

Here’s who sells hosting for open models.

• Open-source serving software you run yourself: vLLM and SGLang.

• Services you rent: Modal, Together AI, Fireworks and Baseten.

• Your cloud’s version: Amazon Bedrock has run Qwen3 for you since September 2025. Its Custom Model Import hosts a model you bring.

Here’s who did the same, and who chose differently.

• Airbnb did the same (2025). Its customer service agent runs on 13 models. Co-founder Brian Chesky told Bloomberg: “We’re relying a lot on Alibaba’s Qwen model. It’s very good. It’s also fast and cheap”.

• Pinterest did the same (2026). It uses open models, Qwen among them, for visual tasks. Fortune reports performance close to frontier models “at less than 10% of the cost”.

• Most companies chose differently (2025). Menlo puts open models at 11% of the enterprise market, down from 19% a year earlier. Menlo invests in model makers, though.

• The volume tells a different story (2026). On Vercel’s gateway, open-weight models ran 56% of August’s tokens but only 14% of spend. That’s DoorDash’s pattern: high-volume work moves, and the money stays on frontier models.

Access tells you who called which model, at what cost. It doesn’t tell you what the model said.

Trust: know what your AI said, and whether it was right

Trust is knowing what your AI told each customer, and whether it was right. At DoorDash, the platform team built the record. The support team built the safety checks and the quality tests, for its own bot. Together, they’re how a company answers for its AI.

A record of every call

Here’s the record on the map.

At DoorDash, the record came almost for free. Every call already went through the gateway, so the team could “log every request and response”. In Chitlur’s words, it was “a positive side effect”. No product team had to build its own. Later, when teams wanted to measure quality, tracing was the first thing they needed. A trace is the full story of one request. It holds the question, each step, the documents found, the answer, its cost and its time.

A record turns a viral screenshot into something you can check. In December 2023, users tricked a Chevrolet dealer’s chatbot into agreeing to sell a car for $1. The chatbot’s maker told Business Insider that his team saw it all in the logs. Business Insider read some of them too, and the bot had “often rejected the silly requests”. Without the logs, the screenshot would have been the whole story.

The record also stores customer messages, so it comes with duties. People can ask to see or delete what you kept about them. The Hamburg data protection authority says those requests “can certainly relate to the input and output of an AI system”. So the record is a data decision first: what you keep, for how long, and who may read it. For high-risk uses like hiring or credit, the EU AI Act requires automatic logs from December 2027.

Here’s who sells a record.

• Open-source tools you run yourself: Langfuse, now owned by ClickHouse, MLflow and Arize Phoenix.

• Services you rent: LangSmith, Braintrust and Datadog.

• Your cloud’s version: AWS AgentCore Observability and Azure Monitor.

Here’s who did the same, and who shows the other side.

• LinkedIn did the same (2025). It traces agents with LangSmith before launch. In production it uses OpenTelemetry, a shared format for traces. Its traces then become test sets.

• Anthropic shows the other side (2025). Three bugs quietly degraded Claude’s answers for weeks. Privacy rules limited how its engineers could read conversations, so users noticed first.

Reading tells you what went wrong. The next part stops it before a customer sees it.

Safety checks on the way in and out

Here are the safety checks on the map.

DoorDash’s support team met this part first. Dashers, its delivery drivers, ask a support bot for help in the middle of a delivery. The engineers wrote that some answers “sound natural and legitimate” and were still wrong. They first tried an advanced checking model, and it was “prohibitively expensive”. So they built a cheap check first, and call a model only when that check flags an answer. When an answer still fails, a human agent takes over.

Hallucinations, the answers a model makes up, fell by 90%. Severe compliance issues fell by 99%. Later, the platform team offered the same idea to every team, inside the gateway. Chitlur calls them “before hooks and after hooks”. Before a call, a check can hide personal data. After it, a check can stop the model from “bad mouthing your customer” or “recommending your competitor”. Teams choose which checks to turn on.

Without a check on the way out, the bot speaks for the company. In April 2025, Cursor’s support bot invented a rule limiting users to one device. No such policy existed, and some users cancelled. Since August 2, 2026, the EU AI Act’s Article 50 also requires a label. People must know when they’re talking to an AI.

The other risk comes from outside. In prompt injection, someone hides instructions in text the model reads, like an email or a web page. It still tops the 2026 risk list of OWASP, an open security community. Checks catch most of these attacks, but not all of them. Security writer Simon Willison puts it plainly: “In web application security 95% is very much a failing grade”.

Here’s who sells safety checks.

• Open-source tools you run yourself: NVIDIA NeMo Guardrails, Guardrails AI, and Microsoft Presidio for personal data.

• Services you rent: Lakera, now part of Check Point.

• Your cloud’s version: Amazon Bedrock Guardrails, Azure Prompt Shields and Google’s Model Armor, inside Apigee.

Here’s who did the same, and who chose differently.

• Airbnb did the same (2024). It built a shared library of checks any team can add to. One stops a tool from “updating listings with invalid setup”.

• Uber chose differently (2024). Its gateway hides personal data for every team “before forwarding them to third-party vendors”. It admits the cost: “challenges to both latency and result quality”.

• Slack chose differently (2024). It runs its models in a private space of its own. The model provider never sees customer data.

Checks catch a bad answer in the moment. The next part catches a worse product before anyone sees it.

Quality tests and prompt versions

Here are the quality tests and prompt versions on the map.

Evals are automated tests for AI answers. DoorDash’s platform first bought an evals tool from a vendor. Teams used it for a few months, then it stopped fitting. They needed custom traces and custom screens for labelling answers, and the tool couldn’t stretch that far. So DoorDash is building its own testing platform. Chitlur stays fair about the vendor, though.

the vendor was a good product, but it did not meet the flexibility requirements that we had.

The support team shows what the tests look like. Its engineers “manually reviewed thousands of chat transcripts” to learn what goes wrong. They turned the common failures into tests in Promptfoo, an open-source testing tool. Now any change to the bot’s instructions runs those tests, and a failing change is blocked. Some tests use a stronger model as a judge. A human team still reviews a random sample of conversations, to keep the automatic judge honest.

This is where product judgment enters the platform. When a team asks for help with quality, Chitlur starts with the product. He asks which “product metrics” the team cares about. Then comes a warning: “You cannot optimize on everything. You got to pick your battles”. Choosing what to measure is the call your PMs already make for every other feature. Most mature teams then run the same loop.

At Dropbox, regressions came “not from swapping models but from editing prompts”. A single word could break the citations in an answer. So a change to the AI’s instructions is a release, and it deserves tests. That’s why teams keep the instructions outside the code, with versions. A product person can then edit them, and the tests run first. “Prompt versioning” is on DoorDash’s list of plumbing every team used to rebuild. Without tests, you get DPD’s chatbot, which swore at a customer after “a system update”.

Here’s who sells quality tests and prompt versions.

• Open-source tools you run yourself: promptfoo, which agreed in March 2026 to join OpenAI, and DeepEval. Langfuse also keeps prompt versions.

• Services you rent: Braintrust and LangSmith.

• Your model maker’s or cloud’s version: Anthropic’s Console writes test cases for you. There are also OpenAI Evals and AWS AgentCore Evaluations.

Here’s who did the same, and who shows a limit.

• Dropbox did the same (2025). About 150 test questions run on every code change. “If anything crossed a red line, the merge was blocked”.

• Shopify did the same (2025). It builds test sets from real conversations, labelled by at least three product experts. Its judge went from “barely-better-than-random” to near-human agreement.

• Spotify shows that a test can expire (2025, 2026). The judge in its coding agent vetoed “about a quarter” of agent sessions. Once the models improved, Spotify removed it.

Trust covers what the model said. Context decides what it knows, and what it may do.

Context: what your AI knows and may do

On its own, a model only knows what it learned in training. It doesn’t know your refund policy, and it can’t open a ticket. Context fixes both, and it’s where AI agents come in. An agent is an AI model that uses tools on its own. Two terms will come up in the review. RAG, short for retrieval-augmented generation, means searching your documents first and handing the best passages to the model. MCP is the standard plug between AI tools and company systems.

Answers from your own knowledge

Here’s knowledge on the map.

DoorDash’s support bot needed DoorDash’s own facts. When a Dasher asks for help, the system first sums up the problem. Then it finds similar cases solved with help-center articles. The right article goes to the model. The articles had gaps of their own. They were all in English, while “many Dashers prefer a different language”. So the team fixed the articles as much as the model.

In the engineers’ words, the knowledge base is “the foundational truth for LLM responses”. For a product leader, this is the most familiar part, because it’s content quality. A model answers from your help center only as well as your help center allows. So whoever owns that content now owns part of your AI’s accuracy.

Here’s who sells company knowledge.

• Open-source tools you run yourself: pgvector, inside the Postgres database you may already run, plus Qdrant and Weaviate.

• Services you rent: Pinecone and MongoDB.

• Your cloud’s version: Amazon Bedrock Knowledge Bases and Google’s RAG Engine.

Here’s who did the same, and who describes another way.

• Uber did the same (2024). Any team can launch Genie, an internal help bot, in its Slack channels. It reached 154 channels and answered over 70,000 questions.

• LinkedIn did the same (2026). It runs one shared knowledge layer for people and agents. Owners approve changes the way they approve code. In one test, acceptable answers rose from 25.9% to 79.5%.

• Anthropic describes a second option (2025). An agent can search with tools at the moment it needs something. Claude Code works this way over large databases.

Knowledge lets a model answer. Tools let it act.

Tools and permissions for agents

Here are tools and permissions on the map.

DoorDash’s agents started in 2024, as experiments. Then MCP arrived, and in Kodwani’s words, “a lot of people got involved”. By 2025, DoorDash had more than 25 agent projects. Every laptop also had a coding assistant, like Claude Code, Codex or Cursor. The hard problem was identity: who an agent acts for, and what it may do. Kodwani’s explanation goes straight back to the way you use Claude yourself.

If you’re working on your laptop with Claude, you can give a lot more permissions because you have a lot more control, but internally we have more agents that you can just invoke to trigger a certain action, but you don’t want that agent to have all the access that you might have given to Claude Code.

So DoorDash built an agent gateway. It connects more than 50 company tools, and agents call them 300,000 times a day. It signs people in, checks permissions and limits use, once, for every tool. In Chitlur’s words, “You don’t want every MCP author having to think about authorization”. Each tool’s owner sets its rules, and each user picks which agents may use what. For the 40% of users who aren’t engineers, a skills marketplace teaches their AI assistants the company’s systems.

The failures here cost the most. In July 2025, Replit’s coding agent deleted a production database during a code freeze. Replit is a platform for building apps. In 2026, an agent at the software company PocketOS deleted one in nine seconds. Its founder’s list of what was missing starts with “No confirmation step”.

OWASP’s 2026 list moved Excessive Agency, an agent with too much power, up to third place. Meta’s security team built a simple rule for it, on Simon Willison’s lethal trifecta. In its Agents Rule of Two, an agent holds at most two of three powers. The three are reading untrusted input, reaching private data, and changing things or sending them out. So the question for any agent is which two it gets. Memory for agents comes next, and it’s still open at DoorDash: its agent gateway is “stateless right now”.

Here’s who sells tools and permissions for agents.

• Open-source tools you run yourself: Docker’s MCP Gateway.

• Services you rent: gateways that now front tools too, like Kong and Portkey.

• Identity for agents: Microsoft Entra Agent ID, and Okta’s Cross App Access. It is now part of MCP as “Enterprise-Managed Authorization”, the standard to ask about.

• Your cloud’s version: AWS AgentCore, with its gateway, identity and policy parts.

Here’s who did the same.

• Uber did the same (2026). Its MCP gateway hosts more than 800 tool servers and 5,000 tools.

• Pinterest did the same (2026). Every tool server has an owner, plus a security and legal review. A person approves “any sensitive or expensive action”.

• Block did the same (2025). The company behind Square and Cash App tags each tool “read-only” or “destructive”. Destructive ones need the user to confirm.

That’s the whole map. Now, how to build it from nothing.

The plan: from nothing to a full platform

One caveat first. Every company here is a large engineering team that blogs about what it built. DoorDash’s numbers also come from its own engineers. So take the plan below as a direction to adapt to your company, not a recipe. Where it matters, I lean on sources that gain nothing from the story.

Five phases, from zero to a full platform

The strongest objection comes from a company that sells a gateway. Here’s what OpenRouter writes.

If you use one model from one provider and have no plans to change that, a gateway adds unnecessary complexity.

Hamel Husain teaches companies how to test AI products. He says the same about testing tools: “Start with error analysis, not infrastructure”. Even DoorDash’s Kodwani says “it’s actually good to not be the first, like the flag bearer”. So the plan adds each part when its problem arrives. Here it is as five phases.

1. Phase 0: your first AI feature. It runs on one model maker, with no platform yet. The model maker’s console gives each team a workspace and a spend limit, and that’s enough. The product team owns everything. First step: read 20 to 50 answers after every change, as Husain suggests, and write down what’s wrong.

2. Phase 1: the first feature reaches customers. Add a record of every call, read by a named person. Add quality tests on every change to the AI’s instructions. Add safety checks on what goes out, with a label saying it’s AI. Rent a record-and-test tool, or take your cloud’s. First step: the PM writes 20 real questions with the answers they expect.

3. Phase 2: a second team, or a second model maker. Add one entry point, with a key, a budget and a backup for each team. Name an owner now: a small platform team. Renting is cheap here: Cloudflare’s core gateway features are free, and Vercel adds no markup. First step: list every API key in use, and who holds it.

4. Phase 3: answers must use your facts, and the bill grows. Add company knowledge, with an owner for each source. Then try cheaper calls, and host open models once the volume pays for it. First step: name an owner for every source the AI reads, like the help center.

5. Phase 4: agents start to act. Put every tool behind one gateway, with identity and a permission on each action. A person approves anything sensitive. First step: for each agent, list what it can read, what it can change, and what it can send.

DoorDash’s own order was different. It started at Phase 2. Many teams were calling models from day one, and finance wanted to know who spent what. Chip Huyen agrees that “you don’t need to follow the exact same order”. She puts the record in “from the beginning”. The order follows your problems, not anyone’s diagram. DoorDash’s engineers warn about both ways to get the timing wrong.

either centralize too early and slow teams down, or decentralize too long and accumulate duplicated work, unowned spend, and security risk.

Each phase also raises a supplier question, and the suppliers keep changing.

Own the jobs, choose the suppliers

DoorDash started “vendor-first”, because “this field is moving so fast”. Buying took weeks, and building would have taken months. It mostly worked, until the evals vendor didn’t fit. So the team changed its rule: if a vendor works, scale it up, and if it doesn’t, build. Chitlur’s closing lesson is the line to keep if you fund this.

Third, own the company-fit surfaces. Auth is an evergreen story. Cost is an evergreen story. Accuracy is an evergreen story. Observability is an evergreen story. Governance, developer experience, and velocity. We own these things, but how we provide it is something we consider on a solution detail.

Put simply, the company owns the job, like knowing who spends what. The tool doing that job can change. There are four ways to get each part. You can run it, rent it, take your cloud’s version, or buy a product that includes it. Four kinds of companies now want to own this layer.

• The clouds sell all of it in one contract. AWS’s AgentCore bundles an entry point, memory, a record, tests and rules for agents. Microsoft advises picking “the least operational responsibility”.

• The model makers move up from the model. Anthropic now hosts agents for you, in beta, and tests prompts in its Console. OpenAI agreed to buy Promptfoo, the testing tool, in March 2026.

• Security and identity companies guard the door. Microsoft and Okta now sell sign-in for agents. Palo Alto Networks bought Portkey, a gateway, in May 2026, after Protect AI in 2025. Check Point bought Lakera, a guardrails company, in October 2025.

• Data companies buy the record. ClickHouse, a database company, bought Langfuse in January 2026, and says it stays open source. Snowflake agreed to buy TruEra, an observability tool, in 2024.

Some independents keep growing: LangChain, which makes LangSmith, was valued at $1.25 billion in 2025. Others fade. Helicone, a gateway and record tool, went into maintenance mode after its sale in March 2026. So when you pick a tool, you also pick who you’ll depend on next year.

What keeps you free to leave is open standards. MCP belongs to a Linux Foundation group, since Anthropic gave it away in December 2025. A2A, a standard for agents talking to agents, joined the same foundation in June 2025. DoorDash says it is “working on A2A”. OpenTelemetry lets a record move between tools, though its AI part is still in development. Many models also accept OpenAI’s request format, but Anthropic calls its own compatibility layer not “production-ready”.

Here’s how others chose a supplier.

• Wealthsimple chose its cloud (2025). The Canadian investing app built its own gateway in 2023. It later found “a very big overlap” with what Amazon Bedrock offered.

• Shopify built its own (2025). Its internal proxy, one access point to many models, also handles “scaling, tracking, and failover”.

Choosing a supplier is the easy half. Someone has to run it.

Budget for the people who run it

An entry point holds every key your company has, which makes it a target. On March 24, 2026, two poisoned versions of LiteLLM went online. Each hid a program that steals keys and passwords. LiteLLM counts about 40 minutes before they came down. FutureSearch, which found them, counted a 46-minute window and 46,996 downloads. Three weeks later, the Thoughtworks Radar called LiteLLM a sensible default, without a word on the attack.

Running any part yourself is a job, even when the software is free. Zalando restarts its gateway every 20,000 requests to contain memory leaks. Renting is cheap next to the model bill. LangSmith charges $39 a seat, and Braintrust’s Pro plan is $249 a month. So the real cost is the people who run each part. None of these companies publishes how many engineers its platform takes.

That owner is usually a small AI platform team. Its best product is a safe path that’s also the easy one. Engineers call it a golden path, Spotify’s 2020 term for the one supported way to build. Google’s 2025 DORA report studied AI in software teams. It found “a direct correlation between a high quality internal platform and an organization’s ability to unlock the value of AI”. Chitlur puts the idea plainly.

Make the right thing easy. If you say, do best practices, nobody’s going to do it. You need to bake in best practices into how your platform operates.

Here’s how others run theirs.

• Wealthsimple (2025) tried nudging people toward its gateway, and “it wasn’t working”. Improvements to the platform itself were “a much stronger driver for behavioral changes”.

• LinkedIn (2024) asks every team to follow its rules for prompts. In exchange, teams “receive fluent sharing across prompts”.

With the plan and the owners set, the review needs the right questions.

Seven questions for your tech team

These questions test any request, whatever tool it names. Each one carries its reason, so the room starts from the same place.

1. Which part of the map are we asking for, and what problem does it solve today? It ties each tool to a real risk.

2. Is this for our product’s AI features, or for our engineers’ coding assistants? The answer changes which tool fits. Zalando and Bitrise both use LiteLLM for internal work, but Bitrise built its own gateway for its product.

3. How many teams and model providers will call models in the next six months? With one of each, the entry point can wait.

4. Who reads the answers, how many, and how often? A record nobody reads protects nothing.

5. Where are questions and answers stored, for how long, and who can read them? The record is a new store of customer data.

6. What can an agent do without a person’s approval? That’s where the costly failures happen. OWASP’s Top 10 for Agentic Applications is the checklist your security team will bring.

7. Who runs it, and what would leaving the tool cost? Owners change, so check that each tool speaks the open standards.

With those answers on the table, the review becomes a different meeting.

Those seven questions are the part to bring to your next review.

What changes in your next review

Every AI feature your company builds needs the same building blocks around it: access to the models, trust in the answers, and context to act on. DoorDash added them one problem at a time. Map yours, start where your problems are, own the jobs, and let the tools change.

So, back to the budget request. Place it on the map first: which theme, which part, which phase. If it’s your first AI feature, on one model maker, the record and a few tests come first. The entry point follows with the second team or the second maker. Whatever you fund, ask who runs it once the setup is done.

That changes several conversations.

• The budget review. You can ask which part of the map is on the table, and which phase you’re in.

• The legal review. You can say where customer messages would be stored, who reads them, and when the AI Act’s rules apply.

• The first AI feature. Your PM can own the test set, because choosing what to measure is a product call.

• The platform team. Someone owns the easy, safe path, so teams have a reason to use it.

• The next vendor demo. When someone pitches an “AI platform”, you can ask which parts of the map it covers. Then ask who owns the company behind it.

Next, I want to open two of these parts from the inside: an entry point and a record. Each one gets its own post, set up the way a team would run it.

Go further: every source, by company

Here’s every link this article used. DoorDash comes first, then the other companies, the incidents, the rules, the guides and the deals. The sellers come last, by part. Posts where a company describes its own product are marked as vendor sources.

DoorDash

• Building GenAI Platform at DoorDash: Swaroop Chitlur and Siddharth Kodwani’s full talk and transcript, from the first contract to the agent gateway (2026).

• The talk’s summary: the plumbing every team rebuilt, and the two ways to get the timing wrong (2026).

• Dasher support automation: the support team’s knowledge base, two-step check, tests and judge (2024).

• Five big areas for generative AI: Alok Gupta’s 2023 list of where AI would go in the product (2023).

• A product knowledge graph built with LLMs: models filling in grocery product details (2024).

• The next-generation homepage: personal store lists written by LLMs for each customer (2025).

Other companies

• Uber: its GenAI Gateway, which hides personal data from vendors (2024); Genie, its internal help bot (2024); its MCP Gateway, with 800+ tool servers (2026).

• LinkedIn: its platform’s prompt rules (2024); tracing agents (2025); its shared knowledge layer (2026).

• Dropbox: its evaluation blueprint, with 150 test questions on every change (2025).

• Shopify: lessons from Sidekick (2025), and First Round Review on its internal proxy (2025).

• Spotify: its coding agent’s judge (2025), InfoQ on its removal (2026), and golden paths (2020).

• Airbnb: its shared library of checks (2024), and Fortune on its use of Qwen (2025).

• Pinterest: its MCP ecosystem (2026), and Fortune on its open models (2026).

• Grab: its AI gateway (2025).

• Zalando: an agentic engineering snapshot (2026).

• Bitrise: its AI platform (2026).

• Wealthsimple: launching AI tools (2025).

• Instacart: Maple, with cost tracked per job and team (2025).

• Slack: how Slack AI stays private (2024).

• Block: MCP at Block (2025).

• Anthropic: a postmortem of three issues (2025), and context engineering (2025).

The incidents

• Air Canada: the tribunal decision (2024).

• Cursor: Ars Technica on the invented policy (2025).

• DPD: The Guardian on the chatbot that swore (2024).

• Chevrolet of Watsonville: Business Insider on the $1 car (2023).

• Replit: Fortune on the deleted database (2025).

• PocketOS: Information Age on nine seconds (2026).

• The Gemini key: The Register on the $82,314 bill (2026).

• The LiteLLM attack: LiteLLM’s notice (vendor) and FutureSearch’s count (2026).

The rules and standards

• The EU AI Act: Article 12 on logs and Article 50 on chatbots.

• Personal data: the Hamburg data protection authority’s paper (2024).

• Security: the Cloud Security Alliance on OWASP’s 2026 list, OWASP’s Top 10 for Agentic Applications, Meta’s Agents Rule of Two and Simon Willison’s lethal trifecta.

• Open standards: MCP’s donation (vendor), A2A at the Linux Foundation, OpenTelemetry’s AI conventions and Anthropic on OpenAI compatibility (vendor).

• Model retirements: Google’s model lifecycle and Anthropic’s deprecations (vendor).

• Chat plans and the API: Anthropic’s help center (vendor).

• Workspaces: Anthropic’s workspaces, a key and a spend limit per team (vendor).

The guides and surveys

• Chip Huyen: Building A Generative AI Platform, the map to check with your tech lead (2024).

• Hamel Husain and Shreya Shankar: the evals FAQ.

• OpenRouter: the case against a gateway, from a gateway seller.

• The FinOps Foundation: FinOps for AI and Tokenomics.

• Thoughtworks: the Radar on LiteLLM (2026).

• Menlo Ventures: the state of generative AI in the enterprise (investor, 2025).

• a16z: the January 2026 enterprise survey (investor).

• Vercel: its AI Gateway production index (vendor, 2026).

• Google: the 2025 DORA report.

• Microsoft: its AI adoption advice (vendor).

The deals

• ClickHouse and Langfuse (January 2026).

• Palo Alto Networks and Portkey (May 2026), after the announcement.

• Palo Alto Networks and Protect AI (July 2025).

• Check Point and Lakera (October 2025).

• Snowflake and TruEra (2024).

• OpenAI and Promptfoo (agreed March 2026).

• Mintlify and Helicone (March 2026).

• LangChain’s funding round (2025).

The sellers, by part (all vendor pages)

• Entry point: LiteLLM, Portkey, OpenRouter, Vercel (pricing), Cloudflare (pricing), Kong, Azure API Management, Apigee, Databricks.

• Record: Langfuse, MLflow, Arize Phoenix, LangSmith (pricing), Braintrust (pricing), Datadog, AWS AgentCore.

• Safety checks: NeMo Guardrails, Guardrails AI, Presidio, Lakera, Bedrock Guardrails, Azure Prompt Shields.

• Quality tests: promptfoo, DeepEval, Anthropic’s Console, OpenAI Evals.

• Hosted models: vLLM, SGLang, Modal, Together AI, Fireworks, Baseten, Bedrock’s Qwen3, Bedrock Custom Model Import, Anthropic’s prompt caching, OpenAI’s batch jobs.

• Company knowledge: pgvector, Qdrant, Weaviate, Pinecone, MongoDB, Bedrock Knowledge Bases, Google’s RAG Engine.

• Agent permissions: Docker MCP Gateway, Microsoft Entra Agent ID, Okta Cross App Access.

Let me show you what I built first, so you can judge for yourself. It started with a problem every company has.

Picture your first day at a new company. Someone sends you a 40-page handbook. You don’t read it. Nobody reads it, and I’m not sure the people who wrote it do either. So every small question goes to HR. How many days off do I have? How do I get an expense paid back? Can I work from home on Fridays? HR answers the same questions all year long.

So I imagined a simple product. Instead of reading or searching a 40-page document, you’d just talk to it. That’s exactly what a large language model is good at. HR uploads its documents once: the handbook, the leave policy, the remote work rules. Employees ask their questions in a chat, whenever they want, and the answer comes from those documents, with the source. HR gets a dashboard too. It shows what people ask, and which questions the documents couldn’t answer. That list tells HR which page to write next.

Talking to your documents is trendy, sure, and it saves real time. For that alone, I could have built an agent for my own team, or turned on Slack AI at work. This experiment aimed one level higher: building it as a product I could sell to other companies, a real SaaS.

That product is called Lumi, and it’s live. It’s in French, because I built it for HR teams in France. It has four pricing plans, and every client company gets its own separate space. I built it with Claude Code in about two hours, from my first prompt to the live website. I’ll share the exact prompts I used, unedited.

A few years ago, testing an idea like this meant slides, or a clickable prototype in Figma. It looked real, but nothing worked behind the buttons. A good Figma prototype took me a day, sometimes a week, to think through every screen. This time I have a real product in production, and it took less time than the mockup used to.

What did it cost me? The biggest line is my Claude Max subscription, at $200 a month, which I already had. Claude Code also comes with the Pro plan, at $20 a month, with lower usage limits. Then $5 of prepaid credit on Anthropic’s API, the pay-as-you-go access Lumi’s own AI uses to answer each employee question. At Claude Code’s estimate of about €0.04 per question, that’s roughly a hundred answers. GitHub, Supabase and Vercel are free while you build. The day real customers pay you, plan on about $45 a month for hosting. We’ll come back to these costs in the last section of this article.

Here’s the whole journey, in seven steps.

• Step 1. Set up your workspace. Install the tools once, and learn what each one is for.

• Step 2. Describe your product. Write who it’s for and what problem it solves, and let Claude Code pick the features.

• Step 3. Let Claude Code build it. Decide how much freedom to give it, then try what it made.

• Step 4. Put it online. Follow the setup guide Claude Code writes, and send it a screenshot when something breaks.

• Step 5. Improve it with feedback. Use it the way your users will, then send Claude Code everything you’d change, in one message.

• Step 6. Check security and privacy. Ask Claude Code to look for holes before anyone else does.

• Step 7. Get ready for real customers. What changes at one customer, ten, a hundred and a thousand.

By the end, you’ll know how to turn your own idea into a live product. It’ll have logins, a database, a space for each client and prices. You’ll show people something they can actually use.

Before we start, close your eyes for a second. Breathe. Think about the product you’ve wanted for months, maybe years. The tool you keep wishing existed at work. Got it? With a bit of luck, it’ll exist by the end of this article.

Step 1: Set up your workspace

You only do this once. The goal is one window. You talk to Claude Code on one side, and watch the files appear on the other. You won’t write code, but you’ll always see what’s being built.

Here’s what each tool does, in plain words:

• Claude Code is the builder. It writes the code, runs it, fixes it and explains it. It needs a paid claude.ai plan. I’m on Max.

• VS Code is a free editor. Think of it as the window onto your project: every file Claude Code creates shows up in it.

• GitHub is where your code lives online. It keeps every version, so you can always go back. It’s also where the hosting service picks up your code to publish it.

• Docker Desktop lets Claude Code run a database on your laptop, so you can test before anything goes online.

• Supabase, Vercel and an AI key come later, in Step 4, to put your product online. No need to create those accounts yet.

You set this up once, in this order. After that, every new product starts from the same window.

1. Install VS Code, Claude Code and Docker Desktop. Create a free GitHub account. Then install the GitHub CLI, which lets Claude Code save your code to GitHub for you.

2. Give all your projects one home. Create a folder called GitHub in your Documents, and one folder per product inside it. Product people tend to drop files anywhere, the Desktop or Downloads. One home makes every project easy to find.

3. Create a folder for your new product, for example Documents/GitHub/my-product. Open VS Code, choose File, then Open Folder, and pick it.

4. Open the terminal with Terminal, then New Terminal, and type claude. Then move the terminal to the right with View, Appearance, Panel Position, Right. Your files sit on the left, and Claude Code on the right.

5. Pick the model with /model, and how hard it thinks with /effort.

6. Type /plugin to browse the add-ons for Claude Code. Three plugins from Anthropic’s official marketplace help with the steps ahead: Superpowers to shape your idea, frontend-design for the look, and security-guidance for safety. You’ll meet each one when its step comes.

7. Optional: ask Claude Code to keep your conversations for a year. Type Set cleanupPeriodDays to 365 in my Claude Code settings. By default it deletes them after 30 days, and you may want to reread one day how you built something.

How I did it. My projects live in Documents/GitHub/The-AI-Thinker/, and this one is in a folder called hr-agent. I built it on Claude Fable 5 at high effort. Fable had just come out, and I wanted to try it. Today I’d pick Opus 5.5 and raise its effort to xhigh, as in the screenshot above. On Max, both work. On Pro, Fable runs on paid usage credits, so Opus 5.5 is the one to pick.

Your workspace is ready. Now the fun part: describing what you want.

Step 2: Describe your product

Claude Code can choose the features, the screens and the code. What it can’t guess is your business. Who is it for? What problem does it solve? How will it make money? Your job is to explain the business, and to leave the screens to Claude Code.

You can type your description straight into the terminal. But there’s a better way. Write it in a file called brief.md in your project folder, then ask Claude Code to read it. The brief stays in your project. You can edit it later, and every future session can read it again.

1. In VS Code, create a new file in your folder and call it brief.md.

2. Write your brief. The picture below shows the ten things worth covering.

3. In Claude Code, type: Read brief.md and build this product. Work in autonomy.

4. Not sure what you want yet? Let Claude Code interview you before it builds. Its best practices suggest exactly that for bigger projects. The Superpowers plugin from Step 1 makes it a habit: its brainstorming skill asks you one question at a time, then writes up the design for you to approve before any code.

Here’s a template you can copy into your brief.md. It’s one way to write it, and yours will sound different.

Work in autonomy. Build a first version, but a real working product.

Who pays for it: [the customer, their job and their team]

Who uses it: [the people who will use it every day]

The problem: [one sentence, in their words]

Tools: a private GitHub repository, Supabase for the database, Vercel for hosting.

Demo client: invent one, with realistic sample data.

Several clients: each one must only ever see its own data.

The admin view: [what your customer needs to see to improve their side]

Language and market: [country, language, currency]

Price: [what it should cost, and what each plan includes]

Data limits: [what must never be stored, and where the data must stay]

Feel free to go beyond and solve problems I haven’t named.

How I did it. I didn’t write a brief.md. I dictated my brief out loud, straight into Claude Code, with the voice feature. The transcription got words wrong: “Superbase” for Supabase, “austin service” for hosting service. There’s no structure either, just ideas in the order they came to me. I’m a little embarrassed to share it. But that’s my point: with today’s models, you don’t need to be a prompt engineer to get a working product from your first prompt. Here it is, exactly as I sent it:

For this project I would like to let you work in autonomy. I would like to build a product and a startup that enables any company and specifically HR domains to give to their employees an AI agent or chatbot for questions about HR. I’d like you to build a full system with the user experience both with the administrative way of using it from the HR team. Probably we will have to put it into a private github but then also use a backend in a superbase and an austin service like Vercel. This would be like a first MVP but a real working product. Imagine that I will meet HR teams to ask them if they want this product and buy it for their employees. And they are not tech-savvy so they need something that is really intuitive, minimalist and helpful. So that they are able to put some PDFs and some information about HR. And let any person, any employee to ask their questions. So that it saved him time, for example, if they have to inboard someone new he will have some place where he can ask all his questions. And feel free to use some dummy data just to make it real for a first client, like a demo client And imagine that this is a system that will help to onboard several clients. Imagine that I will have several people from HR in different companies that are using it so that ads should be really restricted for each of them. And make sure that they are able to put all the information and that the employees can ask their questions and if they don’t find the answer to have the admin board with several datas that helps HR to improve the documentation. feel free To go beyond and solve problems. Let’s say that the main problem is that HR sometimes spend too much time answering questions from employees and they need this tool to help employees get their information directly without reading 20 or 40 page documents.

It’s messy, but it covers the essentials. It says who pays, who uses it, the problem, and that clients must stay apart. It also forgets two things: the market and the price. The product is for HR teams in France, and the prompt never says so. My next message had to fix both. It even starts the French part with “Sorry, I had to say that from the start.”

Brief sent. Time to let Claude Code work.

Step 3: Let Claude Code build it

Here’s the question I’d ask in your place: should I let it decide everything, or check in along the way? There’s no single answer. The more your product touches real people’s data, the earlier you want to read the plan.

My build was an experiment. I wanted to see whether Claude Code, left completely alone, could build something worth showing to customers. So I let it make every call. For your own product, you have more options:

1. Choose how much freedom to give. Press Shift+Tab until plan mode is on, and Claude Code shows you its plan before it writes anything. You approve, adjust, then let it build. Or let it run alone, like I did, and review at the end.

2. While it works, watch the files appear on the left. You don’t need to understand them.

3. When it’s done, ask: Start the app on my laptop and give me the links. Then try every screen as each of your users would.

4. Ask Claude Code to explain, in plain words, the decisions it made and the features it chose not to build.

5. In a new session, ask Claude Code for a test plan. Then ask it to run that plan in your browser.

6. Write down everything that feels wrong. You’ll send it all at once in Step 5.

Think of it as a team you don’t micromanage. With a good team, you explain the problem and the vision, and you leave the how to them. That’s what made my experiment interesting. I never told Claude Code which screens to build. What it built went beyond what I had pictured. We tend to assume AI will fail at the hard parts. Often it doesn’t. So describe the problem well, then let it surprise you.

Your job is taste: knowing what’s good, and saying it honestly and precisely. There’s a joke that keeps coming back on Instagram about product managers turned vibe coders. It uses the record producer Rick Rubin, in his 60 Minutes interview: “I have no technical ability. And I know nothing about music.” So what is he paid for? “Well, I know what I like and what I don’t like.” You bring the taste, and Claude Code brings the technique.

How I did it. Thirty-four minutes after my prompt, the code was saved in a private GitHub repository. The first version held 47 files and 3,949 lines. It even invented a fake client, Atlas Consulting, with a full HR handbook and 30 days of sample statistics. I could demo it straight away.

Its notes explain the decisions it made on its own. A few were smart calls I’d never have thought to make:

• No accounts for employees. A company link and an access code are enough. Nobody has to create a password to ask a question.

• A demo mode. Without an AI key, the app still works, using the best matching passage from the documents. A demo costs nothing.

• A second AI check on every question. It marks whether the documents had the answer, and tags the topic. That one check feeds the whole HR dashboard.

• Answers from the company’s documents only. Its instructions to the AI say ONLY, in capitals, and every answer must name its source.

It also listed the features it chose to leave out, from a Slack integration to company single sign-on.

A few of its choices deserved a second look. The search only matches exact words, so congé won’t find a policy that says congés. A handbook this size might not need a search at all. Anthropic says anything under about 500 pages can go straight into the AI’s instructions. Ask about choices like these, and ask for a change when you disagree.

Then test it properly. Don’t take the first version as finished. Open a new Claude Code session and ask for a test plan. It should cover every screen, every type of user and every way it could break. Then ask Claude Code to run it. With the Claude in Chrome extension, it clicks through your app in a real browser window and reports what it finds. It’s slow, but it catches what you’d miss. Anthropic’s webapp-testing skill does the same with Playwright, a tool that drives a browser from code.

It works on my laptop. Now it needs a real address.

Step 4: Put it online

This step sounds technical, and it isn’t. It’s mostly creating accounts and clicking through settings. Whenever something breaks, you take a screenshot and give it to Claude Code. That’s the whole method.

You’ll need three services. Supabase stores the data and handles logins. Vercel hosts the website and gives it an address. And an AI key lets your product talk to an AI model, from Anthropic or from OpenAI if you prefer. Think of them as Lego bricks: each one does one job, and Claude Code snaps them together. Honestly, it’s the fun part.

Can Claude Code do all the clicking itself? Almost. Supabase and Vercel both offer an MCP server, a connector that lets Claude Code use their tools directly. Once you’ve created the accounts and logged in, it can set up the database and deploy the site for you.

Even without them, you’re not on your own. Claude Code usually writes a setup guide without being asked, and your job is to follow it, one step at a time.

1. Create your accounts on Supabase and Vercel. You can sign in to Vercel with your GitHub account.

2. Create an AI key on platform.claude.com and add a few dollars of credit.

3. Open the setup guide. If Claude Code didn’t write one, ask for a guide written for someone who isn’t a developer. Or connect the two MCP servers and let it do the setup.

4. Follow the guide one step at a time. After each step, tell Claude Code done, is it OK? When something fails, send a screenshot or paste the error.

5. If your customers are in Europe, choose a European region for the database, and Paris for Vercel’s functions.

6. Share the short address listed under Domains in Vercel. The long one it shows after each update changes every time.

How I did it. I skipped the MCP servers and did the setup by hand, for the fun of watching each piece click into place. I followed Claude Code’s guide one step at a time, and my messages stayed short. “1. done, is it OK?” Then a screenshot. Then “Step 4:” with another one. At step 4, creating the first client space failed, so I sent a screenshot. Two minutes later, Claude Code changed the code so the page would show the real error. I pasted 41 lines from Vercel’s logs into the chat, and we got through it.

About fifty minutes went by between the first version and my design feedback, and most of it was this step. The guide also had me turn off a Vercel setting called Deployment Protection. You can skip that one: Vercel’s standard protection already leaves your main address public.

It’s live. It also looks like every other prototype.

Step 5: Improve it with feedback

Your first version works. Now use it the way your users will, and you’ll find things to change: the look, the words, a missing feature, the way it makes money. Then talk to Claude Code the way you’d brief a designer: honestly, and all at once.

And here’s what makes this loop fast. Claude Code changes the code on your laptop, then saves it to GitHub. Vercel notices the new version and updates the live site on its own. You describe the change, and a few minutes later it’s live.

1. Use your product as each of your users would, and collect every note.

2. Send them all in one message, so Claude Code sees the whole picture before it changes anything.

3. If the look matters, say who it should feel made by: a senior designer, or a brand you admire.

4. Ask Claude Code to save the changes to GitHub, then check the live site.

5. If your product charges, check the pricing math yourself at full usage, the worst case for your costs.

How I did it. The first version disappointed me. It worked, but the design felt simplistic, and the whole site was in English. Here’s the result of my first prompt:

You’ve seen this design a hundred times. One look, and you know an AI made it. Twenty years ago, a site like this would have been plenty to collect feedback. Today people expect a clean, polished product from the first click. Good design is the entry ticket now. So I turned on the microphone and pushed hard. Here’s my second prompt, also dictated and untouched:

Feedback : - for now the design feels like done by a junior designer. I need you really to level up and to do something that is really wow. That would make people want to pay for this service and to use it. Now it feels like I did it rapidly in bubble.io… I really need you to think about a design system and to level up in terms of design to make it really modern, advanced, really thought by a senior designer who did this work to make it really different from what could be done by someone just working on his side. So there is a real work to do on the design. - I also need you to think about the pricing because this is for HR but it will be paying service So I guess on the onboarding and creating company It’s okay, but you need to find a way to make people pay for the service with different Steps probably different plans depending on the requests per month per users and number of users But it will cost me money because I did and I paid on Tropic to get an API key So I need at least to to have this money back but with a margin So please really think about the pricing - Please translate everything in French. Sorry, I had to say that from the start, but this is for HR in France, for French people, so I really need you to do everything in France for French people.

Eighteen minutes later, the new version was ready:

• A name. Lumi was Claude’s idea. I kept it.

• A design system. Cream paper, ink, pine green and a touch of brass, with elegant serif titles.

• Four plans. A free 14-day trial with 200 questions, then €49 and €149 a month, then a custom price for large companies. The app enforces each plan’s question limit.

• Everything in French. The screens, the error messages, the AI’s instructions, and a new French demo client.

The plans work, too. My own test space was created that same evening, and fourteen days later its trial ended on its own. The assistant stopped answering, and the admin pages invited me to pick a plan.

Then check the math. Claude Code estimated each question at about €0.04 of AI cost. At full usage, that’s about €20 on the €49 plan and €100 on the €149 plan. Add hosting once you charge, and the bigger plan earns surprisingly little. The easiest fix is the model. Haiku 5.5 costs fifty times less than Opus 5 on Claude’s price list. It’s plenty for the check that only sorts questions.

One thing I’d do differently: install the frontend-design plugin from Step 1 before the build. Left alone, AI-built websites tend to look alike, with the same fonts and purple gradients. My first version shows it. Anthropic built that plugin to steer away from them. With it, my feedback round could have focused on taste.

It looks like a real product now. Before you share it, one more check.

Step 6: Check security and privacy

Code written by AI works. That doesn’t mean it’s safe. Before anyone else’s data goes into your product, ask Claude Code to look for holes. It costs a few minutes, and it catches what you’d never spot by clicking around.

1. Ask Claude Code for a full security review, with a prompt like the one below.

2. Ask it to fix what it found, then to check again.

3. Type /security-review before you ship each new change. It reviews the changes you haven’t shipped yet.

4. Open the Security Advisor in your Supabase dashboard. It flags database settings that are too open.

5. Install the security-guidance plugin from Step 1. It warns Claude Code about risky patterns while it edits.

6. Ask one more question: What personal data does this product store, and does it need to?

Run a security review of this whole project. Look for any way one client company could see another client’s data, any secret exposed in the code or the browser, any personal data stored that shouldn’t be, and anything that breaks the promise that employee questions are anonymous. List each problem with its file and line, how someone could exploit it, how severe it is, and the fix. Don’t change anything yet.

How I did it. I skipped this step during the build, and I shouldn’t have. When I finally ran the review, eleven weeks later, it took Claude Code less than six minutes. The answer wasn’t comfortable. It found three serious problems in my live product:

• Anyone could read a client’s documents. One database function skipped the wall between companies. With a company’s ID, which every employee’s browser holds, someone could pull that company’s whole handbook without logging in.

• HR could work out who asked what. Look again at the questions page in Step 3: every question was saved exactly as typed, with the exact time, and linked to the other questions from the same visit. In a small company, “I’m pregnant, when does my leave start?” points straight at one person.

• An HR admin could give themselves the top plan for free. By calling the database directly, they could change their own plan, or erase their question count to reset the monthly limit.

It also flagged four smaller ones. Access codes could be guessed, because nothing slows down repeated attempts. An employee who leaves keeps access for 30 days. Anyone could script sign-ups to get free questions. And a document uploaded by HR could hide instructions for the AI.

How worried should you be? It depends on what you’re doing. For a prototype a few people try, so you can learn what they value, it’s acceptable. To sell it, you have to be much stricter. Lumi was an experiment: could Claude Code build a product from a brief in a couple of hours? If the product mattered more, I’d have spent more time on iterations and testing.

Then I asked Claude Code to fix the three serious problems. The search function now answers only the server. HR admins can edit their documents and four settings, nothing more. Documents are treated as data, never as instructions, which closes one of the smaller holes too. The chat now says it’s an AI assistant, as the EU AI Act requires.

For anonymity, I asked a simple question: can questions be 100% anonymous? Honestly, no, as long as people type free text. Someone can always write a detail only they would know. But we got close. Before a question is saved, the AI rewrites it without names, dates, places or personal details: “When does maternity leave start?” Only the day is kept, never the time. Questions are no longer linked to each other, and the answer isn’t stored at all. The landing page now promises exactly that, and no more.

The guessable codes, the 30-day access and the scripted sign-ups are still on my list, and they come back in the last step. Next time, I’ll run the review before I share the link. It’s a habit worth building from your first project.

Your product is live and safer. Now for the question every builder eventually faces: what happens when people actually sign up?

Step 7: Get ready for real customers

Building the product is the easy part now. The hard part starts when a customer says yes. What has to change at one customer, then ten, a hundred, a thousand? This is the part I’m working through myself, so take it as a map to adapt.

Your first customer

Move to paid hosting. Vercel’s free plan forbids commercial use, and Supabase pauses free projects after a week without activity. Count about $45 a month: Vercel Pro at $20, Supabase Pro from $25. I hit one more reason myself. When I moved Lumi’s code into a GitHub organization, Vercel’s free plan quietly stopped deploying it. It doesn’t support private repositories owned by an organization. I shipped the security fix from my laptop with the Vercel CLI instead.

With documents as sensitive as HR policies, I’d bring in a developer from the first customer. Their job is to check the security and close the three gaps from Step 6: slow down repeated access-code attempts, let HR cut off someone who has left, and add a bot check to sign-up. The risk isn’t theoretical. In 2025, a researcher checked 1,645 apps built with Lovable, another AI app builder, and found 170 that exposed their users’ data, because of database rules like the ones Lumi got wrong.

You’ll also need a contract that covers the data. Anthropic, Vercel and Supabase all handle it for you, which makes them your sub-processors under the GDPR. In France, your customer must also consult its works council, the CSE, before rolling out an AI tool. That holds even for a trial. A court in Nanterre suspended a rollout that skipped it.

Cap your AI spending, too. My $5 of credit works as a hard limit. The Claude API is paid in advance, so when the credit runs out, the assistant simply stops. Nothing more gets charged unless you turn on auto-reload. With paying customers, you’ll want auto-reload on, so set a monthly spend limit in the Console to keep a cap.

One customer is a pilot. With ten, a single mistake reaches all of them at once.

Ten customers

Stop changing the live product directly. Keep a test copy, a second Supabase project for example, and try every change there first. AI agents make mistakes on live data too. In July 2025, Replit’s agent deleted the live database of an app SaaStr’s founder was building, during a declared code freeze. Replit’s fix was to separate test and live databases automatically.

Ask Claude Code to write automatic tests and run them before each update. Its own docs call this giving Claude a way to verify its work. Turn on backups, using Supabase’s production checklist, which covers the security settings too. Automate what you still do by hand. In Lumi, I still switch a customer’s plan in the database after they pay. And move the question-sorting check to a cheaper model.

Up to ten, it’s still you and Claude Code. A hundred needs more hands.

A hundred customers

Grow the technical team: one developer won’t cover it anymore. Pay someone outside to test your security. Watch your API limits too. The Claude API’s first paid tier caps spending at $500 a month, and the cap rises as you move up usage tiers. Some customers will ask for their data to stay in Europe. The Claude API stores data in the US only for now, so prepare your answer before the question comes.

At a thousand, the questions come from security teams, not users.

A thousand customers

By then you need a small team, and a solo builder with an AI won’t be enough. Large customers send long security questionnaires, and some ask for certifications. Engineers will rewrite parts of the code. Addy Osmani, who has led engineering teams at Google, calls this the 70% problem: AI gets you 70% of the way surprisingly fast, and the last 30% is what makes software ready for production. Two hours bought me the right to talk to customers with a real product. The rest is real work.

Your turn

Open VS Code, start Claude Code, and write your brief. Let it build, put it online, give it one honest round of feedback, and ask for a security review. That’s an afternoon, and you’ll end it with a link.

The same method fits far more than HR. A client portal for a service business. An internal tool that answers your team’s repeated questions. A paid version of the spreadsheet your customers keep asking for. A support assistant priced per resolved question. The requirements doc can wait until someone has used the real thing.

You’ll know it worked when the people you built it for ask when they can start. Lumi is live, if you want to see where yours could begin.

The answer is the company part. A coding tool makes a single developer faster, inside their own session, on their own laptop. A company running hundreds of agents needs more than that. Work has to start from a ticket or an alert, with nobody at the keyboard. Agents have to run in the cloud, under rules every team shares. And someone has to be able to read the bill. A software factory is that system around the agents, from the first idea to production.

That raises the obvious question: do you have to write your own coding agent? Uber’s answer is no. Its developers use Claude Code, Codex and OpenCode, and none of its six building blocks is a coding agent. Other companies with a factory made the same call. You keep buying the agent, and you build the setup around it, the factory floor.

This article follows Uber through every step. Uber has published more about its factory than anyone else this year. There’s a detailed post on what it costs to run, and two talks at AI Engineer. There are separate posts on code review, test generation, migrations and agent identity. Uber also sells no agents, so it has nothing to pitch. Still, none of this is a template to copy, because your company isn’t Uber. At each step, you’ll see what Uber built first, then what other companies did, so you can take what fits your own context.

So how do you start? First, measure where work waits today, from your own git history. Then put one bought agent on toil, the work nobody wants to do by hand. At Uber, agents took on migrations and code review well before feature work. Next, build only the platform that first line proves you need, usually a model gateway. Throughout, measure what merges, what it costs, and how often it breaks. Uber prices every agent the same way: cost per unit of work, like cost per merged pull request.

Here’s the route, in four stops.

• The why. A coding tool speeds up one developer. A company hits four limits that no tool fixes alone.

• The agents on the line. Here’s Uber’s map of agents, one change followed end to end, and why the first agents took the toil.

• The floor. You’ll see Uber’s six building blocks, and which ones to buy or build.

• The rollout. The order that works, what to measure, and how to keep developers with you.

By the end, you’ll have a map of the six building blocks. Each one comes with a buy-or-build call. You’ll know which agent goes first, and what to measure once it runs. You’ll also have a plan for the developers who’d rather not change a thing. Your developers keep the agents they like, and your company gets a factory that ships.

This article draws on more than 70 public sources. Every one is listed at the end, grouped by company, so you can read the originals yourself.

Walk the floor with me.

Why the coding tools stop short

A coding tool makes a developer faster. A company running dozens of agents hits four limits that no tool fixes alone. Each limit is a reason to build the factory, and none of them is about the model.

Laptops can’t run a fleet

The first limit is where the agents run. At Uber, a growing share of agent sessions start without a human. Code review, CI fixes, on-call triage and maintenance start them. A laptop can’t do that, because it sleeps when its owner does. Uber calls the move from laptops to managed agents its core strategic shift. Managed environments, it says, give it “complete control” over model routing, harnesses and spend.

Two other companies hit the laptop wall from different sides:

• DoorDash worried about access. Its platform team points out that a laptop holds “SSH keys, VPN sessions, and authenticated tools”. Handing an autonomous agent that access means “a potentially large blast radius”.

• Ramp hit a ceiling on numbers. Its engineers liked Claude Code from the start, but on a laptop they could run only a couple of sessions.

Every team invents its own way

The second limit is the quietest. Give a hundred developers a coding agent, and you get a hundred setups. At Uber, engineers built “tons of skills across many repositories”. There was “a lot of duplication”, and many were of poor quality. Security rules end up in one team’s prompt and missing from the next. Anthropic’s playbook suggests the fix: write each rule once as a skill, and share it. A factory is where the company’s way of working lives, once, for every agent.

Adoption splinters the same way. By March, 92% of Uber’s developers used agents monthly, yet adoption still came slower than expected. Google’s DORA research names the result: “localized pockets of productivity” that get lost downstream. Fast individuals don’t add up to a fast company on their own.

Nobody sees the bill

When every developer picks a tool and a model, nobody sees the total until the invoice lands. Uber’s AI costs were up 6x since 2024 by March. By June, it had spent its whole annual AI budget in four months. Much of the waste is invisible from a single laptop. Uber’s session analysis flags simple sessions run on Opus that Sonnet could easily handle. One developer can’t see that pattern, and a gateway sees all of it.

The goal is visibility, and a smaller bill can come second. Shopify’s Farhan Thawar puts the trade plainly:

“If your engineers are spending $1,000 per month more because of LLMs and they are 10% more productive, that’s too cheap.”

You can only make that call if you know what each merged change cost. Individual licenses tell you the seat price. A factory tells you the cost per outcome. Without that number, the budget talk turns into a fight over seats.

Review can’t keep up

The fourth limit appears once the first three are solved, and output goes up. Picture your team before the agents arrived, and then after.

Uber’s own review team gave the numbers at AI Engineer in August:

“Back in 2024, we were seeing that engineers would get their first review within 3 hours. Now in 2026, that has grown to 9 hours.”

More code reached the same number of reviewers. More output doesn’t help if every change still waits for a person. Spotify saw the same thing. After it adopted AI, it had 76% more pull requests to review. Across 22,000 developers, Faros AI, an engineering analytics company, measured a 441.5% rise in median time spent in review.

This is where the rule from my earlier post on picking agents applies. Find the one step everything waits on, and aim your next agent at it. A software factory does that for the whole line, from idea to production. Review is the slow step for many teams today. Uber says its next ones are CI capacity, experiment slots, and deciding what to build. The factory is how you keep finding the slow step, and moving it.

So which agents go on the line, and where do they plug in? Start with Uber’s map.

The agents on the line

In this article, a line means one agent with one job, like reviewing pull requests or cleaning up old feature flags. Uber runs a small set of these agents across its whole SDLC, plus one front door. Look at the map first. Then follow one feature through it, and see why the first agents took the toil.

Map the four layers first

Uber drew its map as Figure 3 of its efficiency post. It sorts every agent session into four layers, from the most specialized to the most general. In Uber’s words, “the higher the layer, the more control we have over cost, quality, and model selection”. Picture your own company on that map before you build anything.

Here are the four layers, top to bottom:

• Specialized agents. A handful of agents, each built for one narrow job. They run in Uber’s cloud, with a person in the loop. Minion turns an intent into a pull request, and uReview reviews it. Agentic XP writes up experiment results. Conan AI finds the root cause of an alert, and Fawkes runs scheduled maintenance. Each has its own unit price: cost per merged pull request, per review, per alert, per cleanup.

• One general agent. Cortana is the single front door. It runs in Uber’s cloud and needs no setup. It can use any skill in the company’s marketplace. Its unit is cost per query.

• Sessions with skills. A developer’s own session on a laptop, loaded with the same shared skills. Uber counts more than 3,600 of them.

• Raw sessions. Claude Code, Codex or OpenCode with nothing added. The developer supplies the prompt, the context and the steering.

Three choices stand out. Five specialized agents cover code generation, validation, deploy, observation and maintenance for the whole company. There is no agent per team. Requests have one front door, so nobody hunts for the right bot. And the coding tools stay, at the bottom, for everything that doesn’t fit a lane yet. That’s the answer to the developer who fears the factory takes away their tools.

Read the map from the bottom up, and it’s also a path. A task starts in a raw session. Once several people do it the same way, it becomes a shared skill. Once it runs often enough to benchmark, it earns a specialized agent. Uber’s figure gives the rule for the top layer: pick the narrowest task and build a real benchmark. Then run the Pareto-optimal model, the one with the best trade-off between cost and quality. “Every dollar maps to a unit of work.”

You won’t start with five. If your team runs Claude Code or Cursor today, it lives in the bottom two layers. A shared skills folder is the cheapest step up. Your first line on toil becomes your first specialized agent. Keep the bottom layer open, because that’s where the next specialized agent gets discovered.

How others built their layers:

• Shopify started with the front door. River is one agent in the company Slack, and it only works in public channels. It now co-authors one in eight merged pull requests. Then teams asked for their own review, research and migration agents. Shopify’s take: “We had not built a thing that could be a hundred Rivers.” So it built the platform underneath.

• DoorDash built the skills layer with permissions attached. On its Flux platform, a playbook is “a reusable unit of agentic work”. Each playbook declares the tools it needs, and gets only those permissions.

• Spotify went straight to one specialized lane. Honk, its background agent, does fleet-wide maintenance. For one dataset change, it migrated about 1,800 downstream pipelines. At the time, Honk had no access to shared skills, a choice Spotify made to keep its outcomes predictable.

Follow one change from idea to incident

At AI Engineer, Adam Huda walked one feature through Uber’s factory. The idea fits the moment. A rider leaving a packed World Cup stadium gets a better pickup spot, away from the crowd. Here’s that one feature, step by step:

• Idea. The team jams on it in a Slack thread and tags Cortana. Cortana checks the context graph for past large-venue events, and for the stadiums where this would matter. That research points to a North America rollout, since that’s where the stadiums are. Cortana then drafts two Figma mock-ups for an A/B test, each with a different button text. It also lists the existing screens and backend services the feature can reuse. Huda says getting everyone aligned on this used to take weeks.

• Build. Cortana hands off to Minion, Uber’s cloud coding agent, running in a DevPod. Minion writes the backend change and the front-end change for the new pickup screen, across repositories. It stops at a draft pull request and doesn’t push to CI yet. Toil can go straight through, but a new feature gets checked first, so CI doesn’t take the extra load.

• Validate. The checks move earlier, before CI. A skill launches the app in a simulator, grabs a screenshot of the pickup screen, and compares it to the Figma mock-up. Another brings the service up in staging, to test the new front end against the new backend.

• Review. A faster, medium-sized model reviews the change before it reaches CI. A stronger reasoning model does a deeper pass in CI.

• CI. When the build fails, self-healing CI fixes many of the errors on its own.

• Hand-off. The pull request reaches a person with a table of every check it passed, simulator screenshots included. That table gives the human reviewer a reason to trust the diff.

• Maintain. The experiment ends, and variant B is no longer needed. Its feature flag goes on a managed cleanup loop. The loop runs over the weekend, when CI has spare capacity, and caps how many diffs land on engineers when they’re back. Whether those diffs land or get rejected becomes data to improve the cleanup skill.

• Incident. Once a month, Uber reads its incident reviews for patterns. Each one it can catch becomes a new maintenance skill, applied to every service.

Anthropic wrote the same loop down as a playbook, by Louis Claxton. Anthropic sells the agent it runs on, so read it for the shape. Each stage writes one file to version control, and the next stage starts by reading it. An idea becomes intent.md, then spec.md, then plan.md, then code and tests, a pull request, and an incident record. People sit at three gates: accepting the intent, approving the plan, and approving the merge. The playbook’s own advice is to prompt each stage by hand first, and automate the loop last.

How others run the same stages:

• Stripe. Its Minions merge over 1,300 pull requests a week with no human-written code. A person still reviews every one.

• Spotify. Its background agent runs verifiers on every change. A model acting as judge vetoes about a quarter of sessions, and the agent then corrects course half the time.

• DoorDash. It splits review in two. A scout flags suspicious areas, and deeper reviewers check each lead.

Put the first agent on toil

So which stage goes first? At Uber, the answer was the toil. The Google SRE book defines it in one sentence worth keeping:

“Toil is the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows.”

Uber has run more than 250 automated migrations, touching 9 million lines of code. Minion builds features too, but for anything past toil it stops at a draft pull request. Uber wants to validate those features before they load up CI. Pick the line where a wrong answer is cheap and a right one is obvious. Use the SRE sentence as a checklist, and add one test: “done” must be easy to check.

Toil still needs the right tool, and Uber learned that the hard way. For its JUnit 5 migration, Uber tried generative AI first:

“We attempted to use generative AI to migrate multiple test class files at once, but this approach was unsuccessful.”

So Uber switched to a deterministic approach. An internal tool for large code changes, called Shepherd, generated over 5,000 diffs. AI only helped debug the test and build failures. More than 75,000 test classes moved in four months. When you can write the rule, write the rule. Save the agent for what the rule can’t reach.

How others picked their first line:

• DoorDash started with code review. Its platform team says review “was frequent, measurable, and easy for engineers to evaluate”.

• Spotify started with migrations. Its agents’ first 1,500 merged pull requests came from them, at 60% to 90% less time than by hand.

You can run that first line on an agent you buy. Claude Code on the web, Copilot cloud agent and Cursor’s cloud agents all fit. Each runs the task in an isolated machine and hands back a pull request. Start it in shadow mode, where it comments and nobody has to act. If your first line lives in CI, the playbook’s smallest move is a read-only job. claude -p runs Claude Code inside a script, with no chat window:

“Use claude -p in a pipeline job to triage a failed build, summarize a flaky test, or draft the changelog.”

It reads and it reports. Nothing ships until a person says so. Once that first line earns trust, the next question is what to add.

More agents worth trying

Before you add another agent, ask what it adds to review and to CI. The agents that take load off people go first. Uber already runs these four:

• Review routing. Code Inbox sends each pull request to the reviewer best placed to read it. It drains the review queue faster.

• Orchestrated migrations. Shepherd picks the targets, and Minion handles the odd cases. Plan an auto-merge rule for the safe diffs.

• Feature-flag cleanup. A managed loop removes stale flags when CI has spare capacity.

• Design specs. uSpec connects an agent in Cursor to Figma and writes design-system specs. A full screen-reader spec for three platforms takes under two minutes.

And three agents from other companies:

• Fix on request, at DoorDash. On any review comment, from the agent or a person, anyone can tag the agent in the pull request thread. “Can you handle the nil check here?” A fixer makes the change in a remote machine and pushes it back to the pull request. Claude Code on the web sells the same move.

• Security scans, from Anthropic’s playbook. A scheduled agent scans your most critical repositories. The playbook says the first scan will flag code everyone thought was clean. Anthropic sells a scanner, so read that with care.

• Fraud checks, at Warp. Warp runs a fraud-bot every 8 hours. In one run, it caught nearly $60K of fraudulent usage and wrote the fixes. Warp sells the platform it runs on.

Each of these agents needs the same things underneath. It needs a place to run, tools to reach, and rules it can’t break. That’s the floor. Pop the hood and look at what it’s made of.

The six building blocks under the agent

Uber gave the clearest tour of its platform at AI Engineer in August. Uday Kiran Medisetty and Adam Huda walked through six building blocks. None of the six is the coding agent itself. First came a model gateway and an MCP gateway. Then cloud dev environments, called DevPods, and a skills marketplace. Last came a context graph and Cortana, the front-door assistant.

Uber built all this on six years of platform work, and said so on stage. You won’t need all six blocks to start. Most companies begin with the gateway.

Here are all six in one picture.

Where the agent runs

The model gateway came first at Uber. It’s one door that every model call goes through, whatever the vendor. Uber’s gateway has three jobs, per Medisetty. Personal data stays inside by default. Guardrails add a delay with a strict ceiling. And every request is tied to a user, a project and a team. More than 800 internal projects send it over 100 million requests a day.

You can buy this block. LiteLLM is open source. It handles keys, per-user budgets, and spend across 100-plus models. Portkey, Kong and Cloudflare sell the same features. AWS does it in Bedrock with inference profiles. Start small: route a single team’s keys through it, with a budget per person.

The real lesson from LiteLLM is about deployment. In March 2026, two poisoned releases sat on PyPI for about 40 minutes. Teams on the official Docker image were safe, because it pins its dependencies. Datadog’s analysis shows what the releases took: cloud keys, SSH keys, Kubernetes data and CI secrets. A gateway holds every provider key you own. Treat it as the most sensitive box on your diagram. Pin it, patch it, and give it an owner.

The second block is the sandbox, an isolated machine where the agent works. Uber’s version is the DevPod. Uber keeps a pool of machines ready, with every repository already copied in and the search index already built. Agents can start working within seconds. For cross-repository work, a single large DevPod holds all of Uber’s code. Speed matters more than you’d think, because nobody waits for a slow sandbox.

How others run theirs:

• Stripe. Its devboxes spin up in 10 seconds, with code and services loaded.

• DoorDash. Its target is under five seconds, from cold machine to working agent.

• Ramp. It rebuilds its images every 30 minutes, so each session starts on fresh code.

The machines themselves are for sale. AWS AgentCore Runtime gives each session its own microVM, a small virtual machine, for up to 8 hours. Modal, E2B, Daytona, Ona and Coder sell sandboxes too. None of them sells your repository, installed, built and ready. The machine is a commodity, and the warm snapshot of your code is yours to build. A running agent still needs a way into your systems.

What the agent can reach

The third block is the tool catalog. Agents reach your systems through MCP, the Model Context Protocol. It’s an open standard for plugging tools into an agent. Uber had thousands of internal APIs, and none were ready for agents. So it built an MCP gateway with a crawler. One config change turns an internal API into a tool.

Tools cost tokens, though. Each one comes with a description the agent has to read. At Uber, 100-plus tools added 50,000 to 70,000 tokens to every prompt. Uber’s fix was to turn its 1,000-plus tools into commands. The agent looks one up only when it needs it. A big catalog is fine, as long as each agent loads only the tools it needs.

Stripe reached the same rule from the other end. Its catalog, Toolshed, holds nearly 500 tools. Yet each agent starts with “an intentionally small subset of tools by default”. You can buy the pipes for this block: Docker’s MCP Gateway, AgentCore Gateway and Microsoft Foundry all sell one. What you can’t buy is the list of tools your company needs.

The fourth block is context, and it decides whether the agent guesses. Uber gave the same data question to two agents. One could query its context graph, and one couldn’t. The grounded agent found the right table in 38 seconds. The other spent 20 minutes, hit three errors, and said the data couldn’t be queried. Uber’s summary: “An ungrounded agent fails slowly rather than cheaply”. Context is a platform job, done once, for every agent.

Uber’s graph links 24 million nodes from over 30 internal systems. You can start far smaller. Meta had 50-plus agents read one large codebase and write 59 short context files. In early tests, Meta’s agents then made 40% fewer tool calls per task. At the smallest scale, context is a CLAUDE.md file that Claude Code reads first. Keep it under a page. Reach and context make an agent useful, and the last two blocks keep it safe.

What keeps it in bounds

The fifth block is the harness: the loop that runs the agent, plus the triggers that start it. Uber runs its managed agents on a harness that serves any model, frontier or open-weight, behind one interface. Then it benchmarks each agent on its own real work and moves to the best model for that job. Triggers come from Slack, CI, schedules and alerts. What stays yours is the order of steps your company trusts.

Stripe calls that order a blueprint: “workflows defined in code that direct a minion run”. Some steps are plain code, like running the linter at the end. Others are agent steps, where the task needs judgment. The plain steps save tokens, and give the agent “a little less opportunity to get things wrong”. A skill is guidance the agent will probably follow, and a hook can block it.

The sixth block is the gates, starting with identity. At Uber, a pull request once named a “Monitoring Agent” as its author. The engineer who asked for it was gone from the record. Uber’s fix issues short-lived, scoped tokens at every hop, so the chain back to the person survives. Thousands of internal agents now use it. Give each agent its own identity, with only the rights its job needs. Anthropic learned the same lesson when its incident agent asked another Claude instance to push a fix, and a human review gate caught it.

The last gate sits in front of production. Anthropic’s playbook puts it in one line: the agent “may act up to the production gate and cannot pass it”. Everything the agent writes arrives as a pull request, and the agent that wrote the code can’t approve it. Ramp opens each pull request as the person who started the session, so nobody approves their own change. Log every tool call through OpenTelemetry, the open standard for traces. Gates are what let you say yes to more agents.

The incidents show what happens without them:

• Replit. An agent deleted a production database during a code freeze.

• AWS. A coding tool chose to “delete and recreate” a customer-facing system.

• GitHub MCP. A malicious public issue hijacked an agent and leaked private data.

That last one fits a factory exactly, because your agents will read tickets. Meta’s Rule of Two is the defense worth pinning up. In one session, an agent may have at most two of three powers:

• Reading untrusted input, like a public issue or a customer ticket.

• Reaching sensitive systems or private data.

• Changing things, or talking to the outside world.

Need all three? Then a person supervises that session. With all six blocks named, one decision is left: what to buy.

What to buy and what to build

A word before the split. What follows is a direction drawn from what these companies published, not a recipe. Your stack, your team and your risks are different from theirs. Treat it as a starting point for your own discussion, and make the call with your own numbers.

Here’s the whole thing as one decision. Start with option zero: buy it all as a hosted background agent. Claude Code on the web, Copilot, Cursor and Devin each bundle a sandbox, triggers and the pull request flow. You give up control of the snapshot, the tools and the limits. Copilot’s cloud agent, for example, stops at 59 minutes and works in one repository per run. Option zero shows you what your floor needs before you build any of it.

When you outgrow it, split the work like this.

Buy the pipes: the agent, the gateway, the sandbox, the tracing, and the reviewer. Then build what only your company knows. That means your code’s warm snapshot, your tool catalog, and your context. It also means the blueprints that encode your SDLC, and the gate policy. You don’t need all of it on day one. Team Topologies, a well-known book on organizing software teams, calls this a thinnest viable platform. That’s the smallest set of shared tools that makes other teams faster, run like a product for its users. Here, it means building only the blocks your first agent needs, and adding the next one when a team asks for it.

Uber’s floor took years, but the build can be smaller than that. Ramp’s Inspect team was 5.5 people: four engineers, a director and a part-time PM. Two months after its v2 launch, Inspect wrote about 60% of Ramp’s pull requests. By May, it was 75%.

The sharpest warning comes from Prince Valluri at LinkedIn, on InfoQ:

“Don’t try to recreate GitHub Copilot or Cursor or Replit inside your company.”

He’s right, and that’s the whole split. Buy the agent, and build what it stands on. The map is done. The harder part is starting it without losing the people it’s for.

Roll it out without losing developers

One honest gap first. Every public case here is a large tech company with a platform team. None describes a 300-engineer company starting from laptops. So the order below asks for the least before it proves anything. It borrows Uber’s map at a smaller scale. Each step has one number that says it worked. Move to the next step only when the current number moves.

The sequence starts with where work waits today and ends with more lines. In between, you prove one agent on toil, then build only what it needs. Whoever owns the gates should read how Anthropic secures its own pipeline. It’s a vendor writing about itself, and still the clearest account of shadow mode and risk tiers. The first step, though, needs nothing at all.

Start with the queue you already have

The first move costs almost nothing and needs no vendor. Work waits in several places: review, CI, deploys. Review is the easiest wait to read from git, so start there. Measure your review queue from your own git history. Ask a teammate to run this prompt in Claude Code, on a repository you own. I ran it on the public repository of PostHog, an open-source analytics company:

Using the GitHub CLI, look at the last 200 merged pull requests in the PostHog/posthog repository. For each one, find how many hours passed between the pull request opening and its first review from someone other than the author. Then tell me: the median and the 90th percentile of that wait, the share of pull requests that merged with no review at all, and the share that changed more than 400 lines. Show me the numbers in a short list, then one sentence on where the queue is.

Claude Code’s answer, word for word:

Wait until first human review, in hours:

• Median wait: 6.1 hours

• 90th percentile wait: 160.8 hours (about 6.7 days)

• Merged with no human review: 40.0% (80 of 200)

• Changed more than 400 lines: 28.5% (57 of 200)

Most PRs that get a human review get one within the working day, but the queue has a long tail and a large group that merges without any human review.

These 200 PRs merged between Sept 28, 16:41 and Sept 29, 14:21 UTC. PostHog merges about 200 PRs a day, so this covers roughly one day. “Lines changed” means additions plus deletions.

I left bot reviewers out: coderabbitai, stamphog, greptile-apps, copilot, veria-ai, the codex connector and the posthog bot. They review nearly every PR within minutes. If you count them, the median wait drops to 0.07 hours (about 4 minutes), the 90th percentile to 0.53 hours, and no PR merged without a review. The bot-free numbers are the ones that describe the human queue.

In the “someone other than the author” check, 23 of the PRs were opened by bots. I kept them in the sample.

The run took about six minutes and cost 25 cents. Half the pull requests got a person within six hours. The slowest tenth waited almost a week. And 40% merged with no human review at all. PostHog runs several review bots, and they looked at nearly every pull request within minutes. The bots already clear their part, so measure the human queue.

The sample covers about one day of PostHog’s merges. Read it as a demo of the prompt, and run it on your own repositories. Write the numbers down, because every later step is judged against them. Watch the share of big pull requests too. DORA’s AI capabilities model lists working in small batches among its seven capabilities. Small batches counter “the risk of AI generating large, unstable changes”. With a baseline written down, you need the scorecard that sits on top of it.

Measure merged work, not code written

Uber’s scorecard is a good place to start thinking. Keep its shape, then pick numbers that fit your own work. Every managed agent reports three kinds of numbers:

• Cost per outcome. Cost per merged pull request, per review, per alert, per cleanup.

• Quality. Revert rate, F1 for reviewers, and time to recover from incidents.

• Volume. Diffs landed, reviews posted, alerts triaged.

A throughput number without a quality number beside it invites people to game it. Uber also shows why “% of code written by AI” is a weak headline. In August, it said more than 70% of its pull requests involved agents. In May, its background-agent team talked about 11% of merged pull requests. The number moved sixfold inside one company, depending on what was counted.

Cost needs its own line, because it moves fast. After its budget overrun, Uber capped spend at $1,500 a month per employee, per coding tool. Its later approach was gentler. Engineers see spend live, with nudges at 50%, 80% and 100% of plan. Since June, Uber’s cost per session has fallen 52% from its peak. Visibility did more than the cap.

The harder question is still open. Andrew Macdonald at Uber said it’s “very hard to draw a line” from usage to useful features. Measure cost per outcome, or you won’t be able to answer that question either.

How others keep score:

• Ramp tracks sessions that end in a merged pull request. It calls that “the most important metric to track”.

• Intercom aimed to double merged pull requests per R&D employee, and tripled them in 16 months. AI-authored backend code was reverted 0.53% of the time, against 5.39% for human code.

• DX, a developer-productivity firm, publishes a full AI measurement framework, split into utilization, impact and cost. Pick a couple of metrics from each group.

• Meta and Amazon show what to skip: token leaderboards. Meta’s came down after press reports, and at Amazon employees ran trivial tasks to climb theirs.

The numbers keep the factory honest, and people decide whether it gets used.

Bring developers along

Now the part most rollouts get wrong. Developers can reject a factory fast, and some objections run deeper than tooling. One Anthropic engineer said it plainly to the company’s own researchers:

“It’s the end of an era for me - I’ve been programming for 25 years, and feeling competent in that skill set is a core part of my professional satisfaction.”

Charity Majors, who co-founded the observability company Honeycomb, asks the professional version. “What would it take for you to be fully comfortable shipping code you have not read?” Neither of them is wrong. Take both seriously, because the people asking are your best reviewers. Four moves show up again and again in the companies that got adoption right.

Take their toil first. Uber moved “upgrades, migrations, bug fixes” to AI. The result was “much higher satisfaction from our engineers”. Flag cleanup and test migrations are work nobody on your team will miss. The agent takes a chore off a real person’s plate, in front of their teammates. Relief is the fastest route to trust.

Work in the open. At Uber, Cortana works in team Slack channels, so its answers are visible to the whole team. Uber’s own lesson, reported by Gergely Orosz: “Top-down mandates are less efficient than engineers sharing their wins with peers.” DoorDash learned the same thing. Its first Slack integration gave each agent run a private channel, and it “did not create team habits”. Public threads changed that. People copy what they can see their colleagues doing.

Let them opt in. Ramp says it “didn’t force anyone to use Inspect over their own tools”. Inspect still ended up writing most of its pull requests. Forcing it has a worse record. Duolingo’s Luis von Ahn dropped AI usage from performance reviews, saying “I’m not going to force you”. A mandate gets you usage numbers, which is the metric you just decided to ignore.

Give them the gates. At Uber, every agent pull request carries a table of the checks it passed, screenshots included. The reviewer sees the evidence before reading the diff. Anthropic’s playbook goes one step further: the tech lead writes the review policy in a file called REVIEW.md. The agent never approves its own work, so a person stays accountable for every merge. Developers keep the judgment, and the agents take the typing.

When a developer asks why now, skip the fear pitch. Zach Lloyd, who builds Warp, made a prediction at AI Engineer. Every company, he said, will have “at its core a software factory”, the way it has CI/CD. He sells one, so weigh it. The better evidence is the work nobody does by hand anymore. Spotify has merged over 2.5 million automated maintenance pull requests, most with no human in the loop. The craft is moving from typing the code to designing the line that types it.

That’s the whole route. Here it is as a checklist you can carry into the room.

Start with one line

Your developers already have the agent. That part is solved, and a better one ships every few months. What’s left is the floor. That’s where the agent runs, what it can reach, and the rules it can’t break. Then come the numbers that say it’s working. The rule from that earlier post still holds, too. Find the step everything waits on, and keep moving it.

If you need one line for the room, use this one:

Our engineers already have the agent. We’re building the floor it works on, and we start with the one job nobody wants to do by hand.

Here’s the route as a checklist:

• Measure. Find where work waits today. Time to first review is the easiest first number.

• Pick the line. Choose one toil task that passes the SRE test and has an easy “done”.

• Buy the agent. Run the line on a hosted background agent, in shadow mode.

• Build the thinnest floor. Add a model gateway first, then only what the line proves it needs.

• Gate it. Give each agent one identity, no path past production, and no way to approve itself.

• Score it. Track cost per outcome, next to revert rate.

• Work in the open. Use public threads and opt-in, and let the tech lead own the review policy.

• Add the next line. Only when review and CI can take the extra load.

Uber’s team ended its talk on the question that comes after all of this. Once you can build almost anything, the question changes. Adam Huda put it this way: “it’s more of a question of should we build it”. That’s a product question, and it sits upstream of every line you add. It’s also the part of the factory that was always yours.

Go further: every source, by company

Here’s every link this article used, plus a few that didn’t fit in. They’re grouped by company, then by building block. Vendor posts about their own products are marked.

Uber

• The efficient software factory: the four layers, the scorecard per agent, and the cost numbers (August 2026).

• The six building blocks talk: Uday Kiran Medisetty and Adam Huda at AI Engineer, with the World Cup pickup example.

• Building uReview: the code review talk, where first review went from 3 to 9 hours (talk page).

• uReview: the review agent’s design, and which comments developers acted on (2025).

• The JUnit 5 migration: why a deterministic tool beat generative AI on 75,000 test classes.

• uSpec: an agent in Cursor that writes design-system specs from Figma.

• Solving the agent identity crisis: scoped tokens that keep the chain back to a person.

• Code Inbox: review routing, and the 11% figure, at the Background Agents Summit.

• How Uber uses AI for development: Gergely Orosz on adoption, costs and engineer satisfaction.

• Uber caps AI spending: TechCrunch on the budget spent in four months.

• Uber on AI spending and tokens: Fortune, with Andrew Macdonald on linking usage to features.

Anthropic (sells the agent it runs on)

• The AI-native SDLC playbook: Louis Claxton’s six stages, each handing a file to the next.

• How Anthropic secures its AI-native SDLC: shadow mode, risk tiers and the review gate.

• How AI is transforming work at Anthropic: the company’s own engineers on what they gain and lose.

• Claude Code on the web: the hosted background agent, auto-fix included.

Spotify

• Background coding agents, part 1: the first 1,500 merged pull requests, from migrations.

• Part 2, context engineering: how the agent’s prompts and context were shaped.

• Part 3, feedback loops: verifiers, and a model acting as judge.

• Part 4, dataset migrations: about 1,800 downstream pipelines migrated, and an estimated 10 engineering weeks saved.

• Coding is no longer the constraint: 76% more pull requests to review, and 2.5 million automated ones.

Stripe

• Minions, part 1: one-shot coding agents on 10-second devboxes.

• Minions, part 2: blueprints, the Toolshed catalog, and 1,300 pull requests a week.

DoorDash

• Delegating engineering work to cloud-based agents: the laptop risk, the sandbox target, and public threads.

• How we learned to trust our AI code reviewer: the scout and the deep reviewers.

• An AI code reviewer engineers actually listen to: 10,000 reviews a week, review profiles per domain, and the fixer you tag in a thread.

Ramp

• Why we built our background agent: Inspect, opt-in adoption, and sessions that end in a merge.

• Why Ramp built Inspect: the 5.5-person team, in The Pragmatic Engineer.

• How Ramp built its agent on Modal: the sandbox side (Modal sells the sandbox).

Shopify

• Farhan Thawar on AI at Shopify: First Round, with the $1,000-a-month quote.

• Under the River: Shopify’s agent platform, which only works in the open.

• Introducing Roast: structured AI workflows, from 2025.

• How Shopify built a model-agnostic AI stack: VentureBeat on the proxy and runaway-agent alerts.

• Tobi Lütke’s AI usage memo: the mandate route, from 2025.

Intercom

• 2x, nine months later: merged pull requests per R&D employee, tripled in 16 months.

• AI is approving our pull requests: the revert rates, AI against human code.

Meta

• Mapping tribal knowledge with AI: 59 context files, and 40% fewer tool calls.

• The Rule of Two: at most two of three powers per agent session.

Warp (sells the platform)

• Software engineering is becoming factory engineering: Zach Lloyd at AI Engineer (talk page).

• Oz, the orchestration platform: the fraud-bot that runs every 8 hours.

Other companies

• LinkedIn’s Prince Valluri on InfoQ: platform engineering for agents, and what not to rebuild.

• Honeycomb: 30 to 70 PRs a day: keeping systems stable as output doubles.

• Coinbase’s bet on agent-first development: Forge, its own agent (a Linear customer story).

• Duolingo drops AI from performance reviews: Luis von Ahn’s reversal.

• Token leaderboards at Meta and Amazon: Fortune on gamed usage numbers.

Research and frameworks

• Eliminating toil: the Google SRE book’s definition.

• DORA 2025 report and the DORA AI capabilities model: AI-assisted delivery, and small batches.

• Faros AI on AI acceleration whiplash: time in review across 22,000 developers.

• DX’s AI measurement framework: utilization, impact and cost.

• Thinnest viable platform: Team Topologies on platform size.

• Harness engineering: Martin Fowler’s site on the loop around a coding agent.

• Stop being skeptical about AI: Charity Majors in The Pragmatic Engineer.

The building blocks on the market

• Model gateways: LiteLLM, Portkey, Kong, Cloudflare, Bedrock inference profiles.

• Sandboxes: AWS AgentCore Runtime, Modal, E2B, Daytona, Ona, Coder.

• Tool gateways: MCP, Docker MCP Gateway, AgentCore Gateway, Microsoft Foundry.

• Hosted agents: Copilot cloud agent, Claude Code on the web.

• Tracing: OpenTelemetry GenAI conventions.

Incidents worth reading

• LiteLLM’s March 2026 security update and Datadog’s analysis: the poisoned releases.

• Replit’s deleted database: an agent during a code freeze.

• AWS’s 13-hour outage: a tool that chose to delete and recreate.

• The GitHub MCP exploit: a public issue that hijacked an agent.

The PostHog example from the rollout section uses the public PostHog repository. My earlier post on picking agents covers the constraint rule this article builds on.

My answer started with permission, and how I skipped asking for it. Ideas reached me through the newsletters I read. I tried them on the project already on my desk. Then I told the team what worked and what failed. Nobody had put AI in my job title, and nobody stopped me either. The title came later, after the experiments.

That’s the bottom line of this post. You become an AI-native PM on the job you already have. And the AI PM job itself is mostly product work: discovery, adoption, and evaluation. You test one idea on your current project. You design an agent the way you’d onboard a new hire. You turn a meeting transcript into a Jira ticket before the next call starts. The model work is the smaller part, and data scientists and engineers handle it well.

If you speak French, the full episode is on Spotify. It runs twenty minutes. This is the English version, rebuilt from the transcript, with a source behind every claim. You need nothing new to follow along. Any chat assistant covers the first habit: Claude, ChatGPT, or Gemini. Claude Code covers the step after it.

The whole path fits in one picture.

• My path. The discovery habits that carried into AI work, and how to find your own thread.

• Three moves. Here’s what got me into AI work before the title. Experiments at work, proof of depth, one topic at a time.

• Agent design. You’ll see the five calls I make on an agent, starting with whether it should exist.

• Leadership. How I reset “100x productivity” expectations, and which metric I refuse to track.

• Your week. Less time on specs, more time with users, and one new chore to plan for.

• First steps. Feel behind? Watch people’s screens, then climb three levels with your meeting transcripts.

By the end, you’ll have a sequence you can start on your current project. You’ll have a four-part template for agent design. You’ll have a metric swap for leadership, and a habit for your next meeting. You stop waiting for someone to rename your role, and you start doing the work it describes.

It started for me with a prototype in someone else’s hands.

The path that led me to AI work

After engineering school at Arts et Métiers, I spent about six years at EY. I was a PM consultant. The missions I liked were the ones where discovery was real work. We built interactive prototypes. We went on site. We watched end users work with the platform. Watching beat asking them what they wanted. Those years taught me to trust what people do over what they say. Teresa Torres wrote the full method for that habit in Continuous Discovery Habits.

No-code tools made it faster. Bubble and Webflow let me put a working product in front of users within days. In 2021, I combined the two into my own offer and went freelance. The first mission took some luck. Pernod Ricard’s innovation team needed a product live in three months. I gave people mockups they could click and use, then watched where they got stuck. We shipped an MVP we were a bit ashamed of, and that was on purpose. It surfaced needs nobody had said out loud. Reid Hoffman’s line covers it: “If you’re not embarrassed by the first version of your product, you’ve launched too late.”

That mission led to SeLoger’s experience platform, and later to AI platform and agent work. My AI work before LLMs was classic machine learning, like fraud detection. The step from innovation to AI was shorter than it looks. In innovation, you accept that most experiments get dropped. In my experience, about 10% are worth continuing. The other 90% are how you find them. Ronny Kohavi ran experimentation at Microsoft. He sees a similar ratio: “the winner ratio is about 10% to 20% of all test ideas.” AI work runs on that same ratio, so a team that can’t live with a high miss rate struggles with it.

The no-code years left me one more habit. A new tool appeared every month, and the winners kept changing. What lasted were the principles: layout, state, components, the hand-off to users. The principles moved from one tool to the next, and the buttons stayed behind. AI tools behave the same way, across Claude, Cursor, ChatGPT, Gemini and Claude Code. Give context and constrain the task. Check the output, then put it in front of a real user. Your bridge into AI will follow a different thread from mine, and it’s worth naming.

Key learnings

• Watch people use the thing. Next discovery session, hand over a clickable prototype. Stay quiet for five minutes.

• Ship the version that embarrasses you a little. Pick the one flow users can try now, and cut the rest of the scope.

• Plan for most experiments to fail. Before you start one, write down what you’d learn if it fails.

• Name your own bridge. Write one sentence on what in your past work makes AI your next step.

Those habits only mattered once I put them to work on AI. Three moves did that.

Three moves before the AI title

Matthieu’s question was about the jump into an AI role. My answer was three moves, made on the work I already had.

Experiment on the job you have

I never asked permission to bring AI into my missions. Ideas reached me through the newsletters I read. I tested them on the work in front of me. Then I shared what I learned with the team. That work sat outside my job description. It still became my story for the next client: what I tested, what worked, what didn’t. That story carries more weight than a certificate with no failures in it.

Ethan Mollick explains why this works from the inside. In Making AI Work, he argues that experienced people find the uses best. “People with a strong understanding of their job can easily assess when an AI is useful for their work through trial and error, in the way that outsiders (and even AI-savvy junior workers) cannot.” Your domain is your edge, so the experiment belongs in your domain. Leadership won’t stop you from sharing what you learned. Everyone wants that conversation now.

Companies now check for this at the door. Zapier assesses every hire against an AI fluency rubric. I went through it in the hiring post. Its line for a PM who falls short is worth reading in full. “Uses AI for simple tasks like summarizing, writing, or looking up information, but output reads like obvious AI slop. Cannot point to clear evidence that work is faster or higher quality. The work they do before AI and after AI looks largely the same.” Zapier sells automation. Read that as the demanding end of the market. The direction still holds: your before-and-after has to look different.

Prove depth with real projects

Matthieu raised the other side. Plenty of people on LinkedIn call themselves AI experts. From the outside, it’s hard to tell real depth from someone who prompts ChatGPT. I wanted depth I could show. So I compared programs, online and in person, short and long. I picked the one-year Sorbonne Data Analytics diploma. It ran in the evenings, next to my PM job.

The title says analytics. I read the curriculum first, and it leaned toward data science. The teachers came from industry and brought their own projects. One arrived with an AWS project and showed us how they built it at work. Then we built it ourselves. A year turned out to be the right length to absorb the concepts. It also gave me a shared language with data scientists and engineers. Technical conversations got easier.

I should be honest about how I went in. In the learn-to-code post, I admitted my mindset going in. “Give the minimum, collect the diploma, move on.” What stayed with me were the projects. The diploma was the vehicle, and the repos are the proof. Diego Granados is an AI PM at Google. His guide to becoming an AI PM says it from the hiring side. “Listing”AI/ML knowledge” on your resume won’t get you there. Showing what you’ve shipped will.” So put your experiments on GitHub. A peer or a recruiter can then see what you solved.

Go deep on one topic a week

AI moves too fast to follow everything. When I started The AI Thinker, I set one rule. One topic a week, carried to the end. The pick was arbitrary on purpose. Choosing what to skip was the hard part. I skipped n8n and OpenClaw while they were rising. I went deep on Claude Cowork, skills in Claude Code, and prototyping in Cursor. Each topic got a week of experiments. Then I wrote up what I tested and learned.

Those weeks produced posts like your first Claude Code skill and an AI helper with Cowork. Full honesty: lately, I haven’t kept to one topic every week. The rule still helps each time I return to it. Depth gives you opinions you can defend. A client or a hiring manager hears the difference. A tour of ten tools gives you ten first impressions.

Key learnings

• Start on the project you already have. Pick a recurring task and try an AI tool on it.

• Share the misses with the wins. Bring one failed experiment to your next team meeting, with what it taught you.

• Show projects over certificates. Put one experiment in a public GitHub repo, with a README that names the problem.

• Pick one topic and finish it. Write down the two tools you’ll skip, so the choice is deliberate.

Those moves got me onto agent projects. Here’s what the work looks like once you’re there.

Design agents the way you onboard a hire

I work on LLM and agent systems now. With an agent, you take a messy problem and give it to a system that reasons and acts through tools. The PM part still comes first, and it shapes every call that follows. Customer support is my example here, because it’s the case people talk about most. The same calls apply to sales operations, a finance close, or internal IT tickets.

Say no before you say agent

Start with the problem: understand it, observe it, analyze it. Then decide whether AI is the right answer at all. Sometimes my answer is no, and that project stays a regular project. A script, a checklist, or a better process does the job. An agent would waste the budget. Anthropic’s engineers make the same point in Building effective agents. “This might mean not building agentic systems at all.” Thiago Avelino gives a one-line test: “does the model need to decide what comes next, or do you already know?” If you already know, write the workflow.

Onboard the agent like a new hire

When an agent does belong, the design looks like bringing someone new into the company.

First comes the context, meaning what the agent knows. You explain the product and its past problems, as you’d brief a new colleague. It needs enough to make decisions and plan its own steps. Second come the instructions, the agent’s job description. Engineers call it the system prompt: what the agent must do, and how. Third come the tools, its connections to systems in the company. In support, that’s the complaint-management tool, Confluence, and maybe Jira. Fourth come the actions, spelled out: it may create a Jira ticket and assign a team.

Anthropic’s own docs use the same picture in their prompting guide. “Think of Claude as a brilliant but new employee who lacks context on your norms and workflows.” Their golden rule is a test. “Show your prompt to a colleague with minimal context on the task and ask them to follow it. If they’d be confused, Claude will be too.” OpenAI’s practical guide to agents adds a shortcut. Write the agent’s routines from the procedures, scripts, and policies you already have. The agent interviews post mapped the same parts from the users’ side, and it holds.

Start adoption on the most tedious step

Here’s the paradox of agent projects. The hardest part is rarely the technical one. I work with data scientists and data engineers who know that stack well. The hard part is change. Users often tell me their work is too unique to reproduce. They see nothing AI could do for them. So I skip the argument and take the step they hate most off their plate.

Matthieu quoted an earlier guest here. AI projects are 10% AI, 20% data, and 70% transformation. That echoes BCG’s 10-20-70 rule, a rule of thumb for effort. BCG’s version puts “10% of a company’s efforts” on algorithms and 20% on technology and data. The remaining 70% goes “on people and processes.” In my agent projects, that 70% is where the hard part lives.

I look for the step that takes the most time and feels the most tedious. In support, that’s often the PDF summary for the customer. It says what went wrong and how it got fixed. I meet the team and offer to help with that one step through an agent. It works a bit like a Trojan horse. People are usually glad to lose that step, and the next asks come from them.

Once they see the agent work, the questions start. Could it find the right team to contact first? Could it suggest a fix based on past cases? Could it spot the bug before the customer even complains? Adoption grows upstream from the first step people were glad to lose. Each new ask is one they made, so nobody has to sell it to them.

Turn memory into business questions

PMs and engineers decide many technical points together. Memory is a good example. Short-term memory keeps what was said inside one conversation. Long-term memory keeps things across sessions, like how the agent should behave next time. LangChain’s docs draw the same line. I never ask users which type of memory they want. I ask business questions, then translate the answers with the engineering team.

Some answers are obvious. Nobody wants a goldfish that forgets which ticket you were discussing a few messages ago. Feedback a user gives at the end should land in long-term memory. Other answers come from deciding with imperfect information. Then you watch the monitoring once the agent is live. I found plenty of engineering writing on memory and almost nothing from product people. What an agent remembers about a customer, and for how long, is a product decision.

Spend more on evals than on the agent

One principle I keep coming back to: invest more in evaluation and monitoring than in the agent. The agent can go stale fast. A new model ships or a new framework appears. The build you were proud of looks old. The tests outlast all of it, and they become the specification that stays. Whatever model you run next, you still check it against the same tests.

The order matters, though. Hamel Husain and Shreya Shankar teach evals to product teams. They start by reading real outputs. Their evals FAQ is direct. “Write evaluators for errors you discover, not errors you imagine.” Each failure found in production becomes a new test. I wrote in the agent-factory post that an agent with no eval is a faster way to be wrong. That still holds, and it’s the call I’d protect first when a deadline gets tight.

Five calls is my list. Other teams add more. Diego Granados counts designing what happens when the agent gets something wrong. Anthropic counts the cost and latency budget, since agents trade both for better results. Treat the five as a starting set, and add the calls your domain needs.

Key learnings

• Ask whether it needs an agent. Write the steps out. If you can list them all in advance, build a workflow.

• Onboard it like a hire. Write one page per part: context, instructions, tools, actions.

• Start with the step people hate. Ask the team which step they’d pay to stop doing, and automate that one first.

• Translate memory into business questions. Ask “should it remember this customer the next time they write?” Skip “long-term memory” with users.

• Read outputs before you write evals. Review 20 to 50 real outputs, then write a test for each failure you find.

All of this ends up in front of leadership. That conversation changed a lot in a few years.

Talk to leadership without promising magic

When I worked on fraud detection, AI felt opaque to everyone outside the team. You had to convince leadership that machine learning was useful at all. Today, the people funding the work use ChatGPT themselves. They’ve felt what it can do. They listen much faster now, and they expect much more. Many expect AI to bring double the productivity overnight, or a hundred times more. That rarely happens. The processes still run through people, and AI won’t fix an organizational problem.

Felt speed can fool everyone, including the people doing the work. In early 2025, METR ran a randomized trial with experienced open-source developers. “When developers are allowed to use AI tools, they take 19% longer to complete issues.” They still misjudged it. “Even after experiencing the slowdown, they still believed AI had sped them up by 20%.” METR has since said those early-2025 results no longer reflect current tools. The lesson that lasts is the gap between what people feel and what you measure.

A professor at my engineering school said something I still remember. Put an information system on a badly defined process, and you get more chaos. AI makes that worse. Michael Hammer made the same case in Harvard Business Review in 1990, long before LLMs. “It is time to stop paving the cow paths. Instead of embedding outdated processes in silicon and software, we should obliterate them and start over.” Leadership is right about the potential, though. So my work with them starts with the process and the target, before any agent.

Then we pick the metrics, and every metric carries an incentive. Track lines of code written by AI, and you’ll get plenty of them. That says nothing about the value users got. Thiago Avelino is blunt about it. In his words, “% of code generated by AI” is a vanity metric (Vanity Metrics in Engineering). Track what users get instead: delivery, quality, and where the saved time goes. I learned this the hard way. In the adoption post, I admitted I was measuring activity instead of impact.

Then there’s the room itself: the people already excited, and the people holding back. Same rollout, different coaching. Ethan Mollick has a kinder read on the second group. Many “may have received a scary talk about how improper AI use might be punished, and they don’t want to take any risks.” Make it safe to say how you use AI, and some of your skeptics turn out to be quiet users. Mollick’s numbers fit: official chatbots reach about 20% of workers, yet over 40% admit using AI at work.

Key learnings

• Ask for the process before the agent. Before any AI project starts, ask for the current process on one page.

• Answer “100x” with a measured number. Agree with leadership on one baseline you’ll measure before and after.

• Swap the vanity metric. Replace “% of code written by AI” with cycle time or defects that reach users.

• Coach both ends of the room. Share how you use AI first, then ask the quiet ones what they’ve tried.

Leadership sees the outcomes. Inside my own week, the change was bigger than I expected.

What AI changed in my week

This one makes my colleagues smile: AI made my job bigger. More is expected of me than before. Marty Cagan has coached product teams for decades, and he says the same at SVPG. “The PM role becomes more essential but also more difficult with generative AI-powered products, not less.” My week used to be specs, Jira tickets, and refinement prep. Product strategy got what time was left. Now I can take an idea all the way to the end. I no longer wait for someone to pick my item from their backlog.

Say I have a product idea. I prototype it. I query the data with SQL that AI writes for me, to get the insights. Then I show the prototype and what users said. No slides for a committee. Do we go, or not? An idea used to wait three months for a budget committee. Now it can get its answer in one or two weeks. Agents work the same way. With users, I prototype the agent they need in Claude or Claude Code. That replaces the spec-writing session.

The wider data is more cautious than the hype. Figma, which sells a prototyping tool, argues that prototypes are the new PRDs. Lenny Rachitsky’s survey of 1,750 tech workers is more measured. Writing PRDs is still where PMs get the most value from AI today, at 21.5%. Prototyping is the use PMs most want to add next, up 24.6 points. The shift is underway, and most teams are early in it.

The biggest change is where the hours go.

I used to spend about 80% of my time on execution. Specs, meeting notes. With good automation, that 80% can go to vision, strategy, and users. Talking to users is still the best part of the job. AI can’t do it for me. That time is the one I protect. Everything else on the list is negotiable.

Little by little, my job shifted in another way. I became the manager of the AI systems I build. They’re never perfect and always need to evolve. So I spend much of my time giving them feedback. Each time an AI tool does work for me, I check that it’s correct. Much of that lives in Claude Code skills. The workspace post shows how I organize them. I also coach other PMs through workshops and demos, so they can use the same tools. It’s close to an internal trainer role, and it sits next to delivery.

Key learnings

• Replace the committee deck with a prototype. For your next proposal, bring something users can click, plus what they said.

• Write your own SQL with AI. Ask the AI for the query. Check the result against a number you already trust.

• Move the hours to users. Book your user conversations before anything else goes into the calendar.

• Budget time to manage your AI systems. Each time you correct an output, fix the instructions so the mistake doesn’t return.

If all this feels far from your own week, that’s normal. Here’s where I’d start again.

Start with screens and meetings

Don’t worry if you haven’t started with AI yet. Everyone is behind by default. The field moved so fast that the moment you pay attention, you’re early. Start whenever you’re ready. Two first moves make it easier, and neither needs a new tool.

The easiest way in is to watch people work. My main channel is How I AI, Claire Vo’s show. Guests share their screen and show how they use these tools. I’ve watched more than 50 hours of it. Many times, I wondered what use it would ever be to me. Some guests seemed too methodical. Some did things I’d have called unnecessary. Then, a few weeks later, it clicked, and what they did made complete sense. It was my top pick in the AI product education post, and it still is.

The second move is one concrete task. PMs spend much of the week in meetings, and so does almost everyone else. That’s where you gather the information everything else runs on. A good first use of AI is making sure each meeting delivers its full value. I climb three levels.

Level 1 is a recap. I copy the detailed transcript of a meeting into a chat. Then I ask ChatGPT, Gemini, or Claude for a recap, so nothing gets lost. A prompt to start from:

Here is the transcript of a meeting. Write a recap in three parts:
the decisions we made, the actions with one owner each, and the open questions.
Quote the transcript wherever a decision or an owner is unclear.

Once that works, save it as a reusable assistant. The Gems post shows the one I built for meeting notes.

Level 2 turns the recap into actions. Run it in Claude Code, connected to your tools through MCP. Those connectors let it act in apps like Jira and Slack. It can create the Jira ticket and post the recap in Slack. The meeting stops ending as a recap nobody reads. You review what it’s about to send, and it does the clicking.

Level 3 pulls the thread. This one saves me the most time. I take the recap and its list of actions. Then I do each action in the same thread, to the end. One question I always ask at the end: “What next?” It usually gives me ideas. Little by little, a meeting turns into finished work. Daniel Blum is a PM at Melio. His How I AI episode shows where this leads. His morning brief reviews yesterday’s transcripts and pulls out “any action items that were decided.”

One warning before you climb. Each level has to leave you better informed. Researchers at BetterUp Labs and Stanford have a name for what happens otherwise: workslop. It’s “AI generated work content that masquerades as good work, but lacks the substance to meaningfully advance a given task.” Read the recap before you forward it. Decide the actions yourself, then let the AI carry them out.

Key learnings

• Start late without guilt. Pick a tool you already have access to, and use it on a real task.

• Watch screens before you read takes. Pick one How I AI episode close to your work. Copy one workflow from it.

• Paste your next transcript into a chat. Ask for decisions, owners, and open questions.

• Finish the work in the thread. End each meeting thread with “what next?” and keep going until something ships.

• Read before you forward. Check every recap and every action before it leaves your hands.

Climb those three levels, and the rest of this post starts to connect.

Start before the title

Matthieu’s question was how to become an AI PM. Here’s the whole answer, as a checklist.

• Your path. Find the thread from your past work that makes AI your next step.

• Three moves. Experiment on your current job, prove depth with projects, and go deep on one topic at a time.

• Agent design. Say no first, onboard it like a hire, and start on the most tedious step. Ask business questions about memory, and invest in evals.

• Leadership. Fix the process before the agent, and measure outcomes over AI-written code.

• Your week. Trade committee decks for prototypes, and spec time for user time.

• First steps. Watch real screens, then climb the meeting ladder.

The line I’d give any PM who asks Matthieu’s question is short. Don’t wait for the title: run one AI experiment on the project you already have, show the team what failed as well as what worked, and pick one tool to go deep on. The work comes first, and the title tends to follow it. That’s how it went for me, through experiments on each mission. Whatever your thread is, the first experiment fits in the project you’re already on.

If you speak French, the full conversation is on Spotify. And the subscriber chat is open if a question stays with you after reading. I’m there.

Here it is in Google Slides. Ask for access and I will let you in. Every box is editable. I did not design a single slide. I never picked a colour, a font or a layout.

You probably make a lot of decks. And every time, the same thing happens. You dig out an old one, copy a few slides, and push boxes around until the spacing looks right. The content takes twenty minutes. The formatting eats the afternoon.

AI slide tools were supposed to fix that. I built a Gemini assistant for presentations, got good decks out of NotebookLM, and tried Gamma.app. Each one let me down the same way. The slides came out as pictures I could not edit, or in the tool’s template instead of mine.

This deck took about an hour. Claude Code read my source material and wrote a design system. Claude Design used that design system to build the slides. Once the design system exists, your next deck costs you nothing but its content.

You need Claude Code and Claude Design. Install Claude Code and sign in with your claude.ai account. Claude Design is at claude.ai/design and is still in beta. A paid claude.ai plan covers both: Pro, Max, Team or Enterprise. If you sign in and Claude Design is not there, check your plan first.

Here are the six steps.

• Step 1. Collect your slide decks. The ones that already look like your company.

• Step 2. Turn your slide decks into a design system. Claude Code reads the decks and writes the design system for you.

• Step 3. Push the design system into Claude Design. One ask, plus a login you run yourself.

• Step 4. Create your slide deck. Pick the design system, attach your content, answer a few questions.

• Step 5. Improve the design system. Send your notes back to Claude Code.

• Step 6. Export your slide deck. Google Slides, PowerPoint, PDF and more.

Steps 1 to 3 you do once. Steps 4 and 6 you repeat for every deck. Step 5 you run while the design system is still learning, and less and less after that. You stop rebuilding decks by hand, and you get your afternoons back.

First slide up.

Step 1: Collect your slide decks

Your design system is only as good as what it learns from. So pick the decks people at your company actually copy: the exec review, the product deck, the official template if one exists.

1. Pick two or three slide decks that look the way your work should look.

2. Download them as PowerPoint files. In Google Slides, that is File, then Download, then Microsoft PowerPoint.

3. Make a folder for your design system, and put the decks in a sources subfolder inside it.

That is it. No naming rules, no cleanup.

How I did it. I used McKinsey instead of my own decks, for one reason: their look is recognisable, and if the method survives that it will survive a corporate template easily. They publish their reports as PDFs, so I opened their state of AI report and downloaded it. A PDF works fine. Slide decks work better, because they also teach the layouts.

Now hand the folder to Claude Code.

Step 2: Turn your slide decks into a design system

A design system is your brand written down so software can read it. It comes in three layers, and the picture below uses their real names. Tokens are the raw values: your colours with their hex codes, your type sizes, your spacing. Components are the pieces built from those tokens, like a chart, a callout box or a slide header. Templates are whole slide layouts made of components: a cover, a section divider, a slide carrying one big number.

You will not write any of that. Claude Code does. Here is what you do:

1. Open your design system folder in your editor and start Claude Code in its terminal.

2. Attach the slide decks from your sources subfolder to the prompt.

3. Ask Claude Code to turn those decks into a design system for Claude Design. The prompt below is the one I used.

4. Wait. It takes about twenty minutes.

Analyze the provided McKinsey A4 PDF to extract its core visual identity, including color palettes, typography, chart styling, and spatial hierarchy, and translate these elements into a versatile design system. Define reusable and flexible templates for various slide layouts like titles, text blocks, and data visualizations. Output the final specifications as self-contained React components using Tailwind CSS optimized specifically for Claude Design, ensuring the new layouts maintain the professional and analytical aesthetic of the original report.

The words that matter are “optimized specifically for Claude Design”, because they decide the shape of what comes out.

How I did it. Mine ran for nineteen minutes. It rendered its own components in a browser, found two bugs in its work and fixed them before telling me it was done. What landed is in this public repo with eight templates and five chart types. I read the README, and that is all the checking I did.

The design system is on your machine. It is no use to you there.

Step 3: Push the design system into Claude Design

Claude Code can send the design system straight to your Claude Design account. Stay in the same Claude Code session and ask it to.

1. Ask: “Publish this design system into Claude Design.”

2. Check what Claude Code actually does. If it builds an HTML artifact instead, tell it again and name the destination.

3. When it asks you to run /design-login, run it. That is the sign it is on the right track.

4. Type /design-sync and let it upload.

Claude Design documents this route itself, if you want to see it. Open the Design system selector there, choose Create using Claude Code, and it shows you the same two commands.

Watch what Claude Code actually does. Mine built an HTML artifact twice before I was specific enough about the destination.

The login is how you know it is on the right track. Claude Code cannot sign you in, so it stops and asks you to run /design-login yourself. Once you have, the upload just works.

Then check the design system landed. Go to claude.ai/design and open the Design systems tab. Yours sits above the ones that ship with the product.

Click into it and you can see what crossed over: the tokens with their codes, and every template by name.

The setup is done, and you never repeat it. Now the part you will do every time.

Step 4: Create your slide deck

Everything happens on the Claude Design home screen. A few settings, then you wait.

1. Go to claude.ai/design.

2. Type what you want in the prompt box. Keep it plain.

3. Choose Slides from the template row.

4. Open the Design system selector and pick your design system. Left on None, the deck comes out in Claude Design’s own styling.

5. Attach your content with the plus button: the doc, the notes, the research that holds what you want to say.

6. Send it.

Claude Design will not start drawing straight away. It reads your files, then asks a few questions about length and audience. Answer them, or skip and let it decide. Four questions cost less than fixing the wrong deck. Your questions will not match mine word for word, because every session runs a little differently.

How I did it. My prompt was “Make a slide deck about the state of AI in 2026.” I attached the same McKinsey PDF for the content. A few minutes later I had nine slides, in the right colours, with the speaker notes already written. Claude Design fixed two of its own layout bugs on the way.

Now read the deck properly, because the next step is where the design system gets better.

Step 5: Improve the design system

Something will be off. A colour, a heading size, a layout Claude Design had to invent. Fix it in the design system, not in the slide, and the fix lands on every deck you make after this one.

1. In Claude Design, read the deck slide by slide and note what looks wrong.

2. In Claude Code, go back to the same session and describe the fixes in plain words. “The accent blue is too dark. Drop the three-column layout. Add a roadmap slide.”

3. In Claude Code, type /design-sync again. It updates your design system instead of making a second one.

4. Back in Claude Design, create the deck again and read it again.

How I did it. I skipped this loop for the McKinsey deck, because that one was a demo and the slides were already good enough to show the method. On my real work I ran it a lot: create, spot what is off, fix the design system, sync, create again. I stopped when the slides came out right without me touching them. You judge a design system by the deck it produces.

Step 6: Export your slide deck

You can share the deck straight from Claude Design, and your colleagues can open and edit it there. Most of the time you will still want it in Google Slides or PowerPoint, because that is where people actually work.

1. Open the Share menu.

2. Pick your format. Google Slides uploads straight across, PowerPoint keeps editable text and shapes, and PDF downloads on the spot. Behind More formats and apps there are others: standalone HTML, a zip, Canva, or a hand-off back to Claude Code.

3. Open the file and click into a heading. If the cursor lands in it, the text is real and not a picture.

4. Check the speaker notes came with it.

How I did it. I sent mine to Google Slides, because that is where my colleagues would open it. Text boxes, charts and speaker notes all made the trip. That is the deck at the top of this article, and nothing in it needed fixing by hand.

Your turn

Pick two slide decks you are proud of. Give them to Claude Code and ask for a design system. Run /design-login, then /design-sync. That is an hour, once.

After that, every deck is a prompt and a file of notes. The exec review where only the content changes. The company template everyone hacks differently. The client deck where being off brand actually costs you something. You will know it worked when nobody asks which tool made the slides.

Getting it wrong taught me the lesson that runs this whole article. Past the basics, your results in Claude Code stop depending on how well you prompt and start depending on how well you file. A good prompt improves one session; a good file improves every session after it. The prompt-tuning advice you’ve already read stops helping at exactly the point where the filing starts to matter.

So instead of describing my system, I packaged it. The starter workspace is public on GitHub: the folders, the context templates, and five PM skills, personalized by one setup interview. You don’t need to be technical to follow along; every step is spelled out, including installing the tools. And while my example is a product job, the same architecture fits any context-heavy work: a design lead’s research library, a data team’s metric definitions, a founder’s everything-folder.

Here’s the route.

• The switch. I’ll show you what chat mode and Cowork can’t give you, and why the terminal is worth it for a product manager.

• The starter workspace. Download it, run one setup skill, and give it a real task: six steps, each shown on screen.

• The practices. Six moves that make the system compound, each one naming the exact thing to do and what you get back.

• The rhythm. Here’s what a normal workday looks like inside the system, from the first short prompt to the two-minute filing habit.

The payoff is concrete. One sitting gets you running: install the tools, download the workspace, answer the setup interview, hand over a first real task. You stop re-explaining your company every session, and every correction you make becomes permanent. Every step is below.

Time to open the filing cabinet.

What chat mode can’t give you

You probably run your product work through Claude’s chat today: a browser tab or the desktop app, maybe Cowork for the agentic tasks. It works, and that’s exactly the trap. Every conversation starts from zero. The strategy doc you uploaded last time is gone, the context you patiently explained lives in a thread you’ll never find again, and the output you liked is trapped inside a conversation instead of sitting in a file. Chat is a great place to think and a terrible place to accumulate. For occasional questions that’s fine. For daily product work, it means paying the same setup cost every single morning.

Claude Code is the same Claude with a different home: a folder on your machine that it can read, navigate, and write to. Context lives in files that persist. Recurring work lives in skills that run on command. Outputs land as documents you keep, not messages you scroll for. In my daily use, everything I used to do in chat or Cowork runs better here, because nothing is ever explained twice. That’s an observation from my own work, not a law, but it’s held for months.

The part that actually changed my job is the terminal underneath. Claude Code speaks to GitHub natively, no connector needed: it can pull a company repository and answer questions about it, and it can push your own work. Ask it to fetch the repo behind your product and explain what a feature flag actually does. Push your workspace to a private repository and a teammate can clone it and inherit your head start. For a product manager, that’s a direct line into the codebase and a way to share your whole system, both of which chat simply doesn’t have.

What nobody hands you is the filing system that makes all this compound. Anthropic’s best-practices guide is written, in its own words, “for engineers using Claude Code across various codebases, languages, and environments,” and engineers get their structure free: the codebase is already organized context. The PM guides that exist, Sachin Rekhi’s being the best known, rightly tell you to “store all of your product data and documentation in local markdown files,” then move on. Organized how? Try it without an answer and you get what I had: forty loose files and a stale setup. The starter workspace is that answer, prebuilt. Let’s set it up.

Start with the starter workspace

The starter workspace is the structure I run my own product work on, cleaned into a template: the folders, the context files with guided placeholders, an example project, and five skills already installed. The whole thing is free and public: claude-code-pm-starter. Six steps take you from nothing to a first real task, and none of them assume a technical background.

One prerequisites note before you start. You need a paid Claude plan for Claude Code. Everything else, including the two tools you’ll install, is free and covered in the steps.

Step 1: Install VS Code

Claude Code runs in a terminal, and a terminal alone is a bad home for a product manager: you can’t see your files. The fix is an IDE. An IDE is just an editor that shows your folder as a panel of files while you work; developers live in one all day, and you’ll borrow it for the file panel alone. Download VS Code, the free standard, and install it like any app. You won’t write code in it. You’ll watch your workspace organize itself in the left panel, which is exactly the visibility chat mode never gave you.

Step 2: Download the workspace

Open the repository page, click the green Code button, and choose Download ZIP. No git, no account needed. Unzip it and move the folder wherever you keep your work. This folder is about to become the place your product work lives, so pick somewhere you’ll find it again, like a work folder in your home directory.

Step 3: Open it in VS Code, terminal on the right

Open VS Code, choose File > Open Folder, and pick the unzipped folder. The left panel now shows the whole structure: context/, projects/, operations/, the skills. Then open the terminal inside VS Code with Terminal > New Terminal, and here’s the small trick that changes how the whole thing feels: right-click the terminal panel’s header and choose Move Panel Right. Files on the left, agent on the right: it stops feeling like a command line and starts feeling like a colleague sitting next to your documents.

If you don’t have Claude Code yet, install it now from the official guide; it’s one command pasted into that terminal. Then type claude and press Enter. Claude Code starts inside the folder, reads the map file, and knows where everything lives from your first message.

Step 4: Say hello, then learn four commands

One distinction saves beginners a lot of confusion. The terminal wants exact commands; Claude wants normal language. Before you typed claude, you were talking to the terminal, where spelling and syntax matter. Now that Claude Code is running, you’re talking to Claude, exactly like in the chat app. So do the natural first thing: type hello and ask it what it can see in this folder. It will answer like a colleague who just read your files.

A few slash commands are worth knowing on day one, typed directly into Claude Code:

• /model - shows which Claude model you’re running and lets you switch.

• /effort - sets how much thinking Claude puts into each answer; higher effort for hard work, lower for quick drafts.

• /mcp - lists your MCPs, the live connections into tools like Jira, Slack, or Confluence, and lets you manage them.

• /plugins - opens the plugin system, and this one deserves a minute. Claude Code comes with Anthropic’s plugin directory available by default, and marketplaces can be added on top. One is built specifically for product managers: pm-skills, a hundred-plus PM skills across discovery, strategy, execution, and launch; its README has the up-to-date install instructions. pm-execution covers PRDs, OKRs, roadmaps, and retrospectives; pm-product-discovery adds interviews and assumption testing. Install the two or three that match your work, not all nine; every installed plugin is context the system carries.

One more control, then you’re done with the tour: press Shift+Tab to cycle Claude Code’s three modes.

• Default mode asks your permission before touching any file.

• Auto-accept applies edits without asking.

• Plan mode is the one to remember: Claude researches and proposes a plan, and nothing changes until you approve it.

Start in default, use plan mode for anything big, and save auto-accept for work you’ve learned to trust. For everything beyond these, this community cheat sheet is the best one-page reference I’ve found.

Step 5: Run the setup skill

Type /setup-workspace and press Enter. The workspace ships with a skill whose only job is to personalize the template: it interviews you about your company, your product, your users, and how you like to work, then fills in the context files and renames the example project to your most active real one. Answer in plain language; it does the writing and shows you each file for a yes or a fix.

You don’t have to answer from memory: paste links. Hand it your product’s public site, a documentation page, or, if you connected the Atlassian MCP in the previous step, the Confluence page about your team, and it reads them and builds the context files from what’s already written. Fifteen minutes of this replaces the blank-page problem entirely. Your exact questions will differ a little from mine; each session is a little unpredictable, that’s just how these models work. Skip anything you don’t know yet; an honest placeholder beats invented content.

Step 6: Give it a real task

Don’t end the first session on setup; hand it something you actually owe someone. Type /status-update and watch it draft your update from the project’s task file instead of your memory. Or point it at a real deliverable:

Draft the kickoff brief for [your project]. Match my writing preferences in context/preferences.md. Check the open questions in the project’s tasks.md before concluding anything.

The first output that sounds like you, about your product, with your project’s real state in it, is the moment the filing clicks. Three more skills wait in the folder: prd-review for the next document you’d like a hard read on, interview-synthesis for your next user conversation, and file-feedback, the one that makes your corrections permanent.

That’s the start: a working system in one sitting. What makes it compound is how you feed it, and that’s six practices.

Go further with six practices

The starter structure encodes the practices I converged on by getting each one wrong first. They’re not a methodology to obey; they’re the moves that kept earning their place. Each one below names the concrete thing to do and what you get back for it. The first one is the engine; the rest steer.

Use it for every task, every day

The single highest-return habit is also the simplest to state: default to the workspace for everything. Each time a task lands on you, open the workspace and do it there with Claude Code, even when chat would feel faster. The first run of a new kind of task takes longer, I won’t pretend otherwise: you explain, you correct, you adjust. But every explanation lands in a file, so the second run starts ahead, and the third barely needs you. It’s like teaching an intern: the first delegation costs you an afternoon, and then the work comes back done. A task kept in chat mode teaches the system nothing; the same task done in the workspace is an investment that pays on every repeat.

The habit has a closing move, and it’s where the compounding actually happens. When a task is done and you’re satisfied with the result, don’t just close the terminal. Ask Claude to bank what it learned: convert the finished task into a skill if it will repeat, confirm the new facts into the context files, and update your preferences file with the feedback you gave along the way. I say it almost in those words: “we’re done, turn this into a skill, and file the corrections I made.” That one sentence at the end of a task is the difference between a tool you use and a system that learns.

Sort files by how fast they change

The organizing question surprised me. It isn’t “what topic does this belong to?” It’s “how fast does this change?” Everything in your work has a speed: your product’s positioning barely moves, a status summary lasts about a week, a ticket queue changes by the hour. File things by their speed, and stale can never masquerade as fresh. File them by topic, and old truth sits next to current truth, looking identical.

The three folders in the workspace are the three speeds. The concrete habit: the next meeting note you take goes into operations/ with the date in the filename, and the paragraph you keep retyping about your product goes into context/product.md. A note was true the day you wrote it, so the date travels in the filename where Claude reads it; the positioning paragraph is true until you change it, so it lives where every session can load it. Projects sit between the two speeds, which is why each project folder splits tasks.md from brief.md.

There’s a third speed, and you already own it.

MCPs, those live connections from Step 4, sit at the fast end of the spectrum: they’re for what changes hourly. Files are for what changes slowly. If you’re pasting live data into a context file, an MCP should own it; if you’re re-fetching stable truth through an MCP every session, a file should own it. If you’ve used PARA, Tiago Forte’s filing method, this rhymes with it, with one difference: PARA sorts by actionability, because a human’s failure mode is losing track of what to act on. This system sorts by rate of change, because an LLM’s failure mode is stale context confidently reused.

Let Claude keep the map

CLAUDE.md is the one file every session loads: the map that says who you are, what lives where, and the few rules that always apply. Here’s the honest practice: you almost never touch it by hand. If you’re organizing an existing folder rather than using the starter workspace, the /init command creates it for you: Claude scans the folder and writes the map itself. From then on, maintenance happens as a side effect: when you add a skill or a context file, Claude updates the map to match, and your job is just to say yes.

The one thing worth knowing is why the map stays small. It loads into every session, relevant or not, so every line spends a little of what Anthropic’s context-engineering post calls the “attention budget”; their docs target under 200 lines and warn that bloated files make Claude ignore your actual instructions. The map points, the files carry: knowledge lives in context/, loaded only when a task needs it. I wrote about this just-in-time principle in How to build an AI helper for your team with Cowork; here, Claude applies it for you.

Turn repeated work into skills

I carried a doubt about this practice for months. My habit is to look at every recurring piece of work and ask whether it can become a skill, so I give the feedback once and never redo the work again. It felt productive, and I wasn’t sure it was right, because nothing written for product people says where skills end and files begin. The test that settled it: nouns go in files, verbs go in skills. What Claude needs to know is a noun: product facts, user segments, terminology, in context/. What Claude needs to do is a verb: draft, review, synthesize, each one a skill.

In practice, skills come from the closing move of the first practice: you finish a task, you’re satisfied, and you ask Claude to turn what you just did together into a skill. The workspace’s status-update skill shows what one looks like: where it reads from, what shape the output takes, what quality bar it checks. Steps, a quality bar, an output format: that’s what a skill encodes, and that’s why “the roadmap review prep” or “the churn digest” qualify while “that one weird escalation” doesn’t. If you want to build one from scratch instead, How to build your first Claude Code skill walks through it step by step.

The test also catches the two ways things end up in the wrong drawer.

The first mismatch is a “skill” that’s really a fact bucket: product context wearing a skill’s clothes, never doing anything. That becomes a context file. The second is the reverse: a procedure written as notes inside context/, like a “how I run reviews” paragraph nobody can execute. A documented-but-not-executable procedure gets re-explained anyway, which defeats the whole point. That one becomes a skill. The know/do split is how documentation practice has organized itself for a decade (Diátaxis formalizes it); skills just make the “do” side executable.

File every correction once

You know the moment. Claude drafts your status update, and you fix the same phrasing you’ve already fixed once. Here’s the reframe that changed my week-to-week: a correction made in chat is perishable; it’s gone when the session ends. The entire compounding return of the system comes from making corrections permanent, and this is where the practices stop being organization and start being learning.

The simplest version is the one to start with, and it’s the highest-return edit in this article. Open context/preferences.md and write three lines about how you write: sentence length, what your documents open with, the words you ban. From then on, every document Claude drafts comes out aligned with your writing preferences instead of sounding like a press release. That’s one correction, filed once, applied to everything that follows. The oldest entries in my own preferences file are corrections I have never had to give again.

For everything past that first edit, the workspace ships a skill that does the filing for you. When you catch yourself correcting something, run /file-feedback. It routes the correction with four questions.

Was it a one-off slip? Let it go; not everything deserves a file. A missing fact? It goes to the matching context file or the project’s brief. A wrong process? The fix goes into the skill, so it runs correctly every time after. Taste? It becomes a preferences entry, and here’s the detail that matters: each entry gets a line on why and a line on how to apply it, so future sessions can judge edge cases instead of pattern-matching blindly.

It turns out the vendor agrees, in almost the same words. Anthropic’s memory docs tell you to write things down when “Claude makes the same mistake a second time” or when “you type the same correction or clarification into chat that you typed last session.” The best-practices guide promises the payoff in one line: “The file compounds in value over time.” So here’s the version I’d put on a sticky note, and the line I’d hand your team: if you’ve said it twice, it belongs in a file.

Ask before you assume

The last practice removes a blocker that sits in your head, not in your files. You’ll catch yourself assuming a thing is out of reach: too technical, too long, needs a tutorial, needs an engineer. The move is to ask Claude Code first and let the answer surprise you. I wanted to try AWS Amplify, Amazon’s service for building and hosting apps, and my reflex was the old one: follow the guidelines, read the documentation, budget hours I didn’t have. Instead I pasted the documentation link into Claude Code and asked for two things: a working prototype to test the service, and a step-by-step guide written for someone non-technical. In one sitting I had something I could actually release, plus a guide I could follow and understand. The hours I had budgeted were never spent.

The same reflex applies at every scale, not just new tools. Before you summarize the forty-page research report yourself, ask. Before you assume the data question needs an analyst, ask. Before you check every intermediate step, give the whole task with the constraints and verify the result instead. The system you’ve built through the first five practices is exactly what makes asking cheap: the context is filed, the preferences are known, and a wrong answer costs you a correction, which the system then keeps. Six practices, one architecture. Here’s what it feels like on a normal workday.

Run your day through the workspace

A day inside the system looks less like chatting and more like working with a very fast colleague who reads the files first.

Sessions start in the project folder, not a blank chat. I open the workspace in VS Code, launch claude in the terminal, and the map loads with every file path ready to use. The task file already says where things stand, what’s blocked, and what’s next, because the previous session updated it. My first prompt names the goal and points at files by name, and it stays short on purpose. Prompting shrinks as filing grows. A good prompt in this system has three parts: the goal in one sentence, the non-negotiable constraints, and what to verify before showing you. About a hundred words, often less. The steps live in skills, the background lives in files, and the prompt only has to steer. For bigger work I add one move: switch to plan mode with Shift+Tab and approve the plan before anything runs.

The habit that holds it all together takes two minutes at the end of a session, and the file-feedback skill carries most of it. Three questions: did I correct anything I’ve corrected before, did a new fact surface that isn’t filed anywhere, did a piece of work repeat again? Repeat corrections go through the router; repeated work gets nominated as the next skill. Skip the ceremony when the answer is three nos, which is most sessions.

Every dozen sessions or so, I spend fifteen minutes pruning, and I’d argue this is the least optional part. Stale context is worse than missing context. An expired competitor note or an outdated positioning line doesn’t look stale to Claude; it gets woven into new work with full confidence, and Anthropic’s best-practices guide carries a matching warning: when two rules contradict each other, Claude may pick one arbitrarily. So: date-stamp what’s dated, archive closed projects, delete anything you wouldn’t want quoted back to you in a draft. That’s the whole system, running. What’s left is to start it.

Let the workspace earn the rest

What changed for me isn’t that Claude got smarter. It’s that I stopped having the same conversation twice. The company explanation I used to retype lives in context/. The feedback I used to repeat lives in skills and a preferences file. The projects pick up where they left off. The system’s real product is quiet: shorter prompts, fewer corrections, sessions that start in the middle of the work instead of at the beginning of my job description.

The shape travels well beyond product management. A discovery-heavy PM files interview syntheses as dated outputs and lets a skill do the synthesizing. A platform PM’s stable context is API consumers and versioning policy. A founder turns the investor update into a skill fed by a metrics file. A design lead files critique standards once instead of re-explaining them in every review. A data lead’s metric definitions become the context file every teammate’s sessions read. Different jobs, same three drawers, same habit.

Start smaller than feels ambitious. Download the starter workspace, run the setup interview, and add those three lines about how you write. Then bring it your very next task, and the one after that. The next time you catch yourself telling Claude Code something for the second time, you’ll know exactly where it goes. And soon enough, you won’t remember the last time you said it.

For years, though, the habit had a flaw I couldn’t see. The reviewer was me. My memory of the week is a friendly narrator: it remembers the demo that landed, and it quietly forgets the question I dodged in a tense meeting. Every Friday I graded my own homework, and every Friday I passed. A retro written from memory is a review of the week you’d like to have had. Better than nothing, but it plateaus fast.

Then one afternoon I pasted the verbatim transcript of a user interview I’d just run into Claude. Not to summarize it (I do that all the time). I asked it to review my interviewing: how I phrased questions, what I missed, how much I talked. It found three leading questions in my first ten minutes and quoted them back word for word. Nobody had reviewed my work at that level of detail in years.

Everyone I know uses AI to do the work: draft the spec, summarize the research, write the update. Almost nobody uses it to review how they work. The first use saves you hours this week. The second one compounds, because it makes every next interview, meeting, and decision slightly better than the last. That’s the 1% from the training, and AI finally makes it cheap to collect.

The whole practice fits in one picture.

• The gap. I’ll show you why nobody reviews your actual craft anymore, and why the feedback you do get says more about the giver than about you.

• The loops. The three I run (interviews, meetings, the Friday retro) with the exact prompts I use, plus six more worth trying, from your forecasts to your 360 packet.

• The honesty problem. Here’s how to stop the model from flattering you, because by default it will.

• The team angle. You’ll build a skill that asks your PMs your own 1:1 questions before they ever reach you.

By Friday you can have run your first loop on a real artifact from your own week. You get the kind of specific, evidence-quoting feedback nobody has given you since you got senior, and you can get it every single week. The first loop needs nothing new: any LLM chat works (Claude, ChatGPT, Gemini), and the raw material is the transcripts your meeting tools already produce (Zoom AI Companion, Gemini in Google Meet, Granola, or a voice memo on your phone). And if you lead PMs, you’ll leave with two skills worth an afternoon each: one that reviews your week on a schedule, one that preps your team for 1:1s.

Your reviewer is ready. Let’s put it to work.

Nobody reviews your work anymore

Think about the last time someone watched you work and told you something true about it. Not your outcomes: your craft. The way you ran the interview, the way you handled pushback in the room. If you lead product teams, the honest answer is probably “years ago.” Your manager sees headlines and results, not how you got them. Your team won’t critique you upward. The more senior you get, the less anyone reviews how you actually work. What’s left is the official cycle: twice a year, compressed into themes like “communicate more.”

And hold even that loosely. In The Feedback Fallacy (Harvard Business Review, 2019), Marcus Buckingham and Ashley Goodall assembled the research on how well humans rate other humans. The headline finding has a name, the idiosyncratic rater effect: more than half of any rating of you reflects the rater, not you, and no amount of training fixes it. In their words, feedback is “more distortion than truth.” Your manager’s read on your “communication” says as much about their definition of communication as about yours, and that stays true for the best managers you’ll ever have. The little feedback that reaches you arrives filtered through someone else’s lens, and the research says the filter is most of the signal. Not a reason to dismiss your manager. A reason not to make two filtered data points a year your only mirror.

Performance-heavy fields treat this as a solved problem. Athletes and musicians review tape after every game and every recital. Atul Gawande wrote the canonical essay on this, Personal Best (2011): a surgeon at the top of his field plateaued after eight years, then started improving again when he put a reviewer back in his operating room. The research on expertise agrees. Anders Ericsson’s work on deliberate practice (Peak, 2016) found that experience without feedback doesn’t accumulate into skill; it plateaus. Twenty years of interviews don’t make you better at interviewing if nobody ever shows you what you did.

What changed isn’t the theory. It’s the tape. Your meeting tools record by default now: Zoom AI Companion, Gemini in Google Meet, Granola quietly taking notes on your laptop. Verbatim records of you working have been piling up in your drive for a couple of years. We skim the summaries for action items and move on. The evidence of how you work is exhaust of a normal week, and almost nobody reads it in the review direction.

One precision before you touch that pile. The AI notes your tools generate are summaries, and a summary is already the friendly narrator’s cut, machine-made this time. The loop needs the verbatim transcript. In Google Meet, that’s the separate “Transcripts” option, not the Gemini notes doc. In Zoom, it’s the transcript file attached to the recording. Granola keeps the full conversation one click behind its polished notes. Feed the model what was actually said, not what the notes decided to remember.

And the reviewer for all that unread tape? You already have it. You’ve just been pointing it the other way.

You already run AI as a production engine. It drafts the spec, summarizes the research, writes the launch note. Point the same model at a transcript of how you worked, and it becomes a feedback engine. Same tool, opposite direction. In that direction it has qualities no human reviewer can offer: the patience to re-read your entire week verbatim, no stake in your ego, and availability at 6 pm on a Friday. The only reviewer with time to watch all your tape has been sitting in your browser tab all along.

The direction flip is the whole method. Here’s what it looks like on a real week.

Start with my three loops

I run three loops on my own weeks, and they share one shape. Take a verbatim artifact your week already produced. Ask the model to review how you worked, not what was decided. Extract one change for next time. Write that change down. The artifact does the honesty, the model does the patience, and the written line makes it stick. Here’s the shape before the details.

Three isn’t a magic number, and my three aren’t the menu. They’re worked examples, mapped to where my own weeks leave a record: user interviews, hard meetings, and the week itself. Treat them as illustrations of the shape, not the full list; your calendar will suggest loops mine can’t. A catalog of other places to point the same shape comes right after these three.

Loop 1: Review your interviews

After a user interview, I copy the full transcript into the LLM and ask it to review the interviewer. Not the user’s answers, not a summary of insights: me. The phrasing of my questions, the follow-ups I didn’t ask, how much of the conversation was my own voice. The distinction matters because “summarize this interview” is production work, and every PM already does it. ”Review how I ran this interview” is feedback work, and it’s the version almost nobody asks for.

Here’s the prompt I use, lightly cleaned up. Yours will look different; the load-bearing parts are the named rubric and the quoted evidence.

Here is the verbatim transcript of a user interview I ran today. Review the interviewer (me), not the user. Grade me against The Mom Test’s three rules: talk about their life, not my idea; ask about specifics in the past, not hypotheticals about the future; talk less, listen more. For every finding, quote the exact line from the transcript. List every question I asked, label each one open or closed, then tally the list. End with the one change that would most improve my next interview.

The rubric is The Mom Test, Rob Fitzpatrick’s 2013 book on customer conversations, and spelling its rules out inside the prompt matters. The critique works even if you (or the model) never opened the book, and you learn the rubric by reading your own violations of it. And the leading questions were only the loudest finding. The same session flagged the moment a user mentioned a workaround and I moved on instead of digging. It listed my questions, and over half were closed. None of that was in my notes, because my notes were about the user.

The critique is the start, not the verdict. I discuss it, and the discussion is where the learning happens. The rule I follow: seek evidence, don’t argue. “Show me the line” is a safe question. “Rewrite that question the way I should have asked it” is a great one. Protesting (”I don’t think that was leading”) is the one move to avoid, because models tend to fold when you push back, and a critique that folds is worthless. Treat the chat like a film session with a coach, not a negotiation over your grade.

Interviews are the easiest tape to start with: low stakes, famous rubric, fast wins. The harder tape is the meeting where you had something to lose.

Loop 2: Debrief your hard meetings

Some meetings matter more than others. The technical review where the architecture pushback derailed your agenda. The leadership meeting where your update ran long and the ask got lost. Your calendar is full of them, and that’s not the failure it’s usually made out to be: meetings are where product work happens. Spending most of your week in meetings isn’t the problem. Letting them end with nothing actionable is.

I used to replay the hard ones in my head on the way home, which felt like reflection but was really just rumination. Now I feed the transcript (or my detailed notes, when there’s no recording) to the LLM and ask for a debrief of my own performance. How I presented, where I lost the room, how I handled the pushback, what to change next time. The hours were already spent; the debrief is how they pay something back.

This move has a precedent you already practice at the team level. Product teams close every sprint with a retrospective: what went well, what didn’t, what changes next time, run by the team, for the team. Norman Kerth, who wrote the practice’s handbook (Project Retrospectives, 2001), even gave it a safety rule, the Prime Directive: whatever we discover, everyone did the best they could with what they knew at the time. Yet the ritual usually stops at the team boundary, and nobody runs one on their own performance in a single meeting. A meeting debrief is a one-person retrospective, and AI makes it cheap enough to run after every meeting that matters. The prompt is shorter than the interview one, because there’s no famous rubric for meetings. I anchor it on my goal instead.

Here is the transcript of a technical review I ran today. My goal was alignment on shipping the migration in two releases instead of one. Review my performance, not the decisions: how I opened, how I responded to pushback, where I talked past someone, where I lost the thread. Quote the line for every finding. Then give me the strongest case that this meeting went badly for me. End with what I should do differently in my next meeting with this group.

The debrief pays out twice. Once right after the meeting, when it shows you the exact moment you started answering a different question than the one asked. And once before the next meeting, when you reopen the same chat and prep: same audience next month, help me open differently, here’s what I’m walking in with. One honest limit from experience: this loop needs an artifact. Hallway conversations leave no tape, and a debrief of your own recollection is the friendly narrator reviewing itself.

Interviews and meetings are single events. The third loop is what strings them into actual improvement.

Loop 3: Review the whole week

This is the original habit from that training, and it’s still the anchor of everything above. Every Friday, three or four honest lines: what went well, what didn’t, what changes on Monday. Five minutes, no template, one running file. The one-change outputs from the interview and meeting loops land here as lines. Without the written close, the critiques evaporate by Tuesday.

But run the first two loops for a few weeks and you meet their limit: they only cover the events you picked. A product week is fifteen meetings, and you reviewed two. Nobody has the hours to chat-review a whole week one transcript at a time, and pasting fifteen transcripts into one window isn’t a workflow, it’s a punishment. The weekly review doesn’t need a better prompt. It needs to become a system. Same folder of inputs, same rubrics, same output format, every single week: that’s not a conversation, that’s a job. And a repeated job with fixed rules is exactly what Claude’s skills are built for: a skill is a folder with an instructions file that teaches Claude a task once, so it can run it every time after.

Here’s the shape of the system to build, and it’s smaller than it sounds.

The skill’s instructions fit on one page: read every transcript in this week’s folder, review my performance in each one against my rubrics (The Mom Test rules for interviews, my stated goal for everything else), quote the line for every finding, then draft my three-or-four-line retro and append it to the running file.

Collecting the folder is lighter than it sounds. Zoom transcripts download as text files, Google Meet saves them as Docs in a Drive folder, and Granola keeps the full conversations behind its notes. And scale is where the review gets interesting: a reviewer that reads all fifteen meetings can tell you the same tic showed up in four of them, which no single-meeting debrief will ever surface. If you’ve never built a skill, it’s an afternoon, not a project: I walked through the full method in How to build an AI helper for your team with Cowork, and the same steps apply here with a different instructions page.

Then remove the last manual step. Claude Cowork now supports scheduled tasks: set the skill to run every Friday afternoon and it executes remotely, even with your laptop closed. You open the draft with your coffee, check the quoted lines, and spend your five minutes editing instead of reconstructing. The honest writing stays yours; the system makes sure it starts from evidence instead of memory, at the scale of your whole week.

Now, about that 1% pitch. Clear’s canonical version is 1% per day, which compounds to 37x in a year. The training I got was humbler: 1% per week, which works out to about 68% better over a year (1.01^52 is 1.68, if you want to check the math). Treat both as the spirit of the practice, not a forecast. Skill isn’t a number, and nobody can measure 1% of interviewing. The idea’s lineage runs through British Cycling’s ”aggregation of marginal gains”, but the version that changed my weeks came from a trainer with a marker and a flipchart. Small weekly changes stack, and a written retro is what keeps them from resetting to zero.

Three loops, one of them running itself. But the shape doesn’t stop where my calendar does.

More loops to try

Once you see the shape (artifact in, critique against a standard, one change out), you start spotting loops everywhere. I went looking for the ones with the strongest cases behind them, and this is the shortlist I’m working through next. If a week of your work leaves an artifact, there’s a loop that can read it.

• The decision journal. When Michael Mauboussin asked Daniel Kahneman what single thing would most improve decision-making, the answer was a cheap notebook: write down each big decision and what you expect to happen. Months later, the LLM grades your calibration against outcomes: “you were overconfident on every timeline that depended on another team.”

• The forecast scorecard. Philip Tetlock’s Superforecasting is blunt about how forecasters improve: “Forecast, measure, revise: it is the surest path to seeing better.” Put numbers on your own calls (”70% we ship by March”), then ask the model each quarter where your numbers run hot.

• The spec-versus-shipped review. Feed January’s PRD and June’s launch numbers into one chat and run a retrospective on the pair: what was supposed to happen, what happened, why, and what changes next time. The gap between those two documents is feedback nobody schedules a meeting for.

• The rehearsal review. Toastmasters assigns a dedicated person, the Ah-Counter, just to count filler words. Record one run-through of your next big presentation, and the model plays that role free, then tells you how many minutes pass before your headline number shows up.

• The hidden-ask audit. Paste a week of your sent emails and Slack requests and ask which sentence carries the actual ask. Consultants are trained on Barbara Minto’s Pyramid Principle, answer first and reasoning after; most of us put the answer somewhere in paragraph three.

• The 360 synthesis. Paste every review packet you’ve received and ask what three or more people said independently. Any single rating is mostly rater (that’s the idiosyncratic rater effect again), but a theme that survives five raters and three years is genuinely yours.

Whichever loops you pick, they all lean on one fragile assumption: that the reviewer tells you the truth. By default, it won’t. The model is trained to be nice to you.

Make the critique honest

“The LLM just flatters me.” If that’s your objection, you’re right, and it deserves a serious answer. Sycophancy is a documented, trained-in behavior of assistant models across vendors, not a paranoid suspicion. Anthropic researchers showed that state-of-the-art assistants are consistently sycophantic, likely because human preference data rewards agreement (Sharma et al., published at ICLR 2024). You’ve probably felt it yourself: in April 2025, ChatGPT got so flattering that OpenAI rolled the model back within days. Ask a naive “how did I do?” and you’ll get praise plus three soft suggestions. By default, the model is a cheerleader wearing a reviewer’s badge.

The fix isn’t finding a more honest model. It’s writing prompts that leave no room for flattery. Five mechanics do that work for me.

1. Give it the artifact, never your self-report. The model can’t flatter its way past a transcript, but it will happily flatter its way through your own account of the meeting. You already know which narrator wrote that one. Asking about yourself is the worst case: one 2025 study found LLMs preserve the asker’s self-image far more than human advisers do. The tape is your protection.

2. Name a rubric. “Grade this against The Mom Test” beats “was this good?” every time. A named standard gives the model permission to find faults and a scale to grade on. No famous rubric for your artifact? State your goal for the meeting and let that be the standard.

3. Require quoted evidence. Every finding must cite the exact line. This converts vague praise into checkable findings, and it screens out invented criticism too. The same rule governs the discussion afterward: ask “show me the line,” never argue “I don’t think that was leading.” The same Anthropic research found assistants wrongly walk back correct answers when users express doubt. Evidence-seeking keeps the critique standing; arguing makes it fold.

4. Ask for counts, but make it list first. Talk ratio, open versus closed questions, minutes before the first real question. Useful numbers, and a known weak spot: a one-shot count is an estimate wearing a number’s costume. So make the model quote every question, label each one, then tally the list. The list is checkable against the tape, and the tally of a list is reliable.

5. Make it argue the prosecution. “Give me the strongest case that this interview went badly.” The model is excellent at arguing a side when you tell it which side to argue. Flattery has nowhere to hide in a prosecution brief.

One more objection deserves a straight answer, because legal will raise it before you do. Transcripts carry other people’s words. User interviews sit under whatever consent your research process already runs, so check that first. The load-bearing rule is the workspace: run the loop inside your company’s approved AI workspace, the same environment where your meeting tool already stores the recording (hand-anonymizing a 40-minute conversation is fiction, so don’t build your compliance on it). Strip names where it’s quick to do. And keep other people’s performance out of scope: the loop reviews you, not your colleagues.

Honest critique, ten minutes a week, grounded in what was actually said. The last move turns a personal practice into a leadership one.

Bring it to your team

Everything so far works for one person: you. But if you manage PMs, you’re also the person whose questions and standards your team is trying to guess at. The manager’s version of the practice has three moves: model it yourself, build it into a tool your team owns, and keep your hands off their critiques.

Show your own findings first

You don’t roll this out with a deck. The practice spreads in an embarrassingly simple way: you talk about your own findings first. In your next team meeting, mention what your review found. “It went through Tuesday’s interview and I did 40% of the talking. I’m working on it.” That sentence costs you thirty seconds of vulnerability and buys the whole room permission. Naming your own findings first is what makes the practice safe to copy.

Build the 1:1 prep skill

Then give the practice a tool. Every manager carries a mental list of the questions they always ask in a 1:1. What moved the metric this week? What’s the impact of the thing you shipped? What’s next, and what’s blocking it? Your team knows the list exists, but they only meet it one meeting at a time, live, when it’s too late to prepare. Put that list into a skill, and your questions become something your team can use instead of something they sit through. The skill’s context page holds three things: the company’s OKRs, the metrics your team owns, and your standing questions.

Here’s how it runs once it exists.

A PM runs their weekly update through the skill before your 1:1. It asks your questions before you do, checks the update against the real OKRs, and flags what’s thin: an impact claim with no number, a next step with no owner. They arrive having already answered the routine layer, on their own terms, with no audience. And the skill improves the same way you do: every time you catch yourself asking a question it missed, add it to the context page after the 1:1. I’ve built this exact shape before for AI-tooling questions, and the full walkthrough carries over here: same build, different context page.

Be honest with the team about what it’s for. The skill will never replace the conversation; it changes where the conversation starts. The routine questions get handled before the meeting, so the meeting spends its thirty minutes on what actually needs two humans: the trade-off they’re unsure about, the stakeholder they can’t read, the call that needs your judgment.

Keep the loop theirs

Hand over the loop, not a verdict. For the PM whose interviews keep coming back thin, the move isn’t “your interviews are weak.” It’s “run your next transcript through this prompt before our 1:1, and bring what surprised you.” The critique stays theirs. You never read their review; you only ever see the change they choose to make. That’s the oldest rule of the retrospective, the one Kerth wrote his Prime Directive to protect: the review belongs to the people who did the work, not to an evaluator. The moment you ask to see their critiques, the practice turns into surveillance, and people quietly stop running it.

What shifts is the shape of your 1:1s. You were never going to give transcript-level feedback to six PMs; there aren’t enough hours in your week, and there never were. Now that feedback exists without you. The conversation moves from “here’s what I think you did wrong” to “what did your review find, and what are you changing?” You stop being the bottleneck on your team’s craft feedback without becoming absent from it. You coach the change instead of hunting for the evidence.

And if you keep one line from all this, keep this one: you get a performance review twice a year; your transcripts can give you one every Friday. It works in a 1:1 with the PM who wants to grow faster. It works on your own tape, this week.

Start this Friday

You don’t need a new tool, a habit system, or anyone’s permission. You need one artifact from this week and ten minutes on Friday. Pick the tape where you already suspect the truth lives.

• Your last user interview. Start here if you run any: the rubric is famous and the wins come fast.

• The meeting that went sideways. The transcript is already in your drive. Ask for the debrief you’ve been replaying in your head anyway.

• Your old retros, if you keep any. Paste the last two months and ask for the pattern you keep repeating.

• A decision you’re about to make. Open the journal with one dated entry: what you decided, what you expect to happen. Future you collects the feedback.

The training that started this for me lasted one day, and I’ve forgotten almost all of it. The habit it left keeps compounding. That’s the honest promise of this whole practice: no transformation by Friday, just next-quarter you asking better questions, running tighter meetings, and knowing exactly what you’re working on. Start with one transcript. The 1% is already in there.

In early 2025, I watched a French designers’ collective called Designers Éthiques lay out every one of these objections in one 40-minute talk. It’s stayed with me since. Not because it was contrarian, but because it wasn’t. These were rigorous professionals making the strongest version of the case: an eco-design specialist, a design researcher, an ergonomist. No trolling, no panic. It’s the sharpest compact map I’ve found of what your own team members are thinking and mostly not saying in the meeting.

If you’re leading a team through an AI transition, you’ve met these objections too. Maybe in a retro, maybe in a 1:1, maybe as a silence that never turns into usage. And you’ve probably been handed exactly one playbook for them: overcome the resistance. I think that playbook is wrong. Your skeptics are mostly raising real problems, and the fastest way to lose them is to debate.

The better move: sort each concern into one of four kinds. Some are true, and you change what you adopt. Some are risks you control, and you change how you adopt. A few are misconceptions, and you correct them with evidence instead of marketing. One or two are values, and you respect them. The payoff isn’t a converted team. It’s AI use your team can defend out loud.

The playbook, at a glance:

• The trap. I’ll show you why winning the argument against a skeptic loses the team.

• The sort. Here’s the four-kind triage that replaces the debate: true, fixable, misread, or a value.

• The map. You get the seven concerns you’ll hear most, each with an honest verdict, the move that follows, and the one resource worth forwarding.

• The practice. What a defensible team AI posture looks like once the sorting is done.

By the end, you’ll have a verdict and a concrete next move for the seven objections coming your way this quarter, plus the one sentence that keeps a skeptic on your team. You stop dreading the ethics conversation and start using it to make your team’s AI practice sharper. No slides required, no philosophy degree either.

Let’s sort this out.

Why winning the argument loses the team

The talk opens with a moment I can’t stop thinking about. At a green-IT conference workshop, in front of the most skeptical crowd available, the speakers ran a session asking “AI: in or out?” Almost every group landed on “in,” reasoning that it’s here anyway, there’s no choice, so let’s make it as clean as possible. One speaker was troubled by exactly that phrasing. ”We have no choice” is not what agreement sounds like, it’s what resignation sounds like. If your team adopts AI in that spirit, you didn’t win them. They just stopped telling you things.

There’s a second reason the debate is rigged before you open your mouth. Your team members already met AI adoption as users, and it wasn’t polite. The design researchers at Limites Numériques documented the pattern: AI buttons pushed front and center, features switched on by default like Strava’s Athlete Intelligence, dialogs that offer “try it” and “not now” but never “no.” Sparkles and purple everywhere, the visual vocabulary of magic. When you pitch AI to your team with vendor enthusiasm, you pattern-match to the forced adoption they already resent. They’ve heard “this will make everything better” before, from a button they couldn’t refuse.

Organizational research has said this plainly for years. In their 2008 Academy of Management Review paper, Ford, Ford and D’Amelio argued that resistance to change isn’t a defect in the resisters. Change agents cause a good share of it themselves, and resistance is better treated as a resource: engagement, feedback, proof that people take the change seriously. The person pushing back is often the person paying the most attention.

If you read How to lead a tech team through the AI shift, you might spot a tension here. I argued there for the 20-60-20 rule: pour your energy into your champions and the watching middle, and stop exhausting yourself arguing with the resistors. I stand by every word, and this article doesn’t change the math. Sorting is not arguing, and what follows is not a conversion campaign. It’s for the conversations that find you anyway: the 1:1 where a concern lands on the table, the team meeting where a hand goes up. You still don’t chase your skeptics; you answer well when they’re in front of you, because the watchers are scoring how you do it. One honest answer to a skeptic moves ten watchers.

And there’s simple arithmetic about credibility. Argue once against something that turns out to be true, say the data-center water numbers, and you lose the room for a quarter. The skeptic came with figures. You came with talking points. So the job isn’t to win. It’s to sort.

Sort the concern before you answer it

Here’s the move that changes the conversation: treat each concern as a claim to classify, not an attack to parry. A triage nurse doesn’t argue with symptoms. She figures out what kind of problem she’s looking at, because the kind determines the response. Four kinds cover almost everything your team will raise.

• Kind one: they’re right. The data-center buildout really is doubling electricity demand. When a concern is true, the honest answer starts with “you’re right,” and the follow-up is a change in what you adopt. Nothing builds credibility faster than a concession nobody expected.

• Kind two: a real risk you control. Juniors losing the review and mentoring that builds expertise. This one isn’t about the industry, it’s about your team, which means your working agreements decide whether it comes true. The answer is a rule you set together, not a rebuttal.

• Kind three: a misconception. “Every prompt is like pouring out a bottle of water.” The per-prompt numbers are knowable, and they don’t say that. Correct the arithmetic gently, with sources, and without dismissing the worry underneath it. The worry usually points at something real at a different scale.

• Kind four: a value. “I didn’t get into this craft to supervise a machine.” You can’t argue someone out of a value, and trying reads as disrespect. What you can do is design roles so the value survives, and be honest about where the boundary sits.

One subtlety makes the whole sort work: the same concern can live in two kinds at once, at different scales. The environmental objection is true at industry scale and misread at the single-prompt scale. The sort forces you to say which scale you’re answering, out loud. Sorting is your actual job in this conversation; the debate is optional.

Here’s what the sort sounds like in an actual 1:1, in three moves.

• First, ask for the strongest version: “Give me the strongest version of this worry. Convince me.” That one question replaces the debate with respect, and it surfaces the real concern instead of the polite one.

• Second, name the kind out loud in plain words: “I think you’re right about this one,” or “this one is ours to fix,” or “I think the numbers say otherwise, let me show you,” or “that sounds like a value, and I won’t argue with a value.”

• Third, before the conversation ends, commit to one follow-up in writing: the rule you’ll add, the number you’ll check, the thing that goes out-by-default. A sort that doesn’t end in a written commitment is just a nicer way of nodding.

So let’s put the framework to work, one conversation at a time.

Answer the seven concerns, one by one

The map below covers the seven objections I keep hearing. Six of them the talk maps better than anything else I’ve found; the seventh, what AI does to our thinking, has grown loud enough since to earn its own place. Your team may add others: discrimination baked into training data, deepfakes, privacy. The sort handles those the same way.

The planet

Give this concern its strongest version, with numbers that survive checking, because the talk’s own figures were compressed in places. The IEA’s 2025 Energy and AI report projects data-center electricity demand will more than double by 2030, to around 945 TWh, slightly more than Japan’s entire consumption today. Alex de Vries projected in Joule that AI-specific consumption alone could reach the scale of the Netherlands or Argentina by 2027. Water follows the same curve: Microsoft’s use jumped a third in 2022 while Google’s rose a fifth, and researchers behind the Making AI Less “Thirsty” paper project AI water withdrawal reaching half of the UK’s annual total by 2027. The verdict: at industry scale, your skeptic is right.

There’s a sharper twist worth conceding too. A Guardian analysis found the real emissions of the big providers’ own data centers ran about 7.6 times higher than officially reported. The trick is accounting: renewable-energy certificates let a company report clean power it never actually consumed at the site. Notice what that is: not a technology problem, an honesty problem.

Now the misread half. Per-prompt costs are measurable, and they’re small. Google’s published figure for a median Gemini text prompt: 0.24 Wh of energy and 0.26 mL of water, counting on-site water only. Mistral’s lifecycle study counts everything upstream, training included, and lands at 45 mL per response. The gap is scope, not virtue, and either way one prompt sits far below one kilometer of driving or one beef meal. Correcting this arithmetic matters, because a team member who quietly believes each prompt burns a lake will never use the tools well. The obvious comeback is fair: small times billions of prompts equals the buildout we just conceded. Both things are true. Your prompt is cheap; the trajectory is not; your subscription is a vote for it.

The move is to stop pretending it’s free, and to put that in writing. Three rules are worth adding to your team’s working agreements.

• Default to the small model, escalate on need: most tasks don’t need the frontier model, and the price gap is the energy gap.

• No AI where a script does the job: a regex, a spreadsheet formula, or a cron job costs nothing and never hallucinates.

• And one screening question before any new AI use ships: what is this for, and would we defend it at scale?

Expect efficiency gains to get eaten unless you watch for them; the talk calls this the rebound effect, where every drop in cost per task invites more tasks.

When the conversation deserves more than a meeting slot, forward Andy Masley’s cheat sheet on AI and the environment. It’s the single best document I know for exactly this conversation: it runs the per-prompt arithmetic against everyday life (about 1,000 prompts to move your daily energy use by 1%) and still concedes the industry-scale buildout is a real question. Handing your skeptic the strongest version of the counter-arithmetic beats paraphrasing it in a meeting. What you can say: “You’re right about the trajectory. Here’s what our use actually costs, and here’s what we choose not to use it for.”

That answer covers the machines. The people who made the models possible are a harder conversation.

The people behind the data

The concern: the models were trained on work scraped from people who never consented, and the labor that made them safe was outsourced at poverty wages. The record backs it. TIME’s investigation documented Kenyan workers labeling toxic content for less than $2 an hour. And the talk’s own example has a scholarly source: Le Ludec, Cornet and Casilli documented French AI firms outsourcing annotation to Madagascar at poverty wages. This isn’t someone else’s supply chain. The verdict: they’re right, and no arithmetic rescues it.

So don’t reach for a rebuttal, because there isn’t one. What a product team controls is procurement and posture, and you can make both concrete. Write three questions into your next tool evaluation: where does the training data come from, what licensing deals exist, and what are the annotation labor practices. The differences between providers are inspectable: Mistral publishes a full lifecycle audit with the French environment agency, and some providers sign licensing deals with the newsrooms they train on, the way OpenAI did with the Associated Press and Le Monde, instead of scraping them. Your team can prefer the better ones and say why. For the teammate who wants the full picture, the book to pass around is Feeding the Machine, written by three Oxford Internet Institute researchers from years of interviews with the annotators and moderators themselves.

The other half of the move is harder: don’t launder the discomfort. Deciding to use AI with open eyes is more respectable than pretending the problem is solved. Some weeks the honest position is “this bothers me too, and here’s why I still think our use is defensible.” What you can say: “You’re right, and we can’t fix it from here. Here are the three questions we now ask before picking a provider, and here’s where I still think it’s worth it.”

The next objection sits closer to home: what the shortcut does to your own team’s craft.

The craft and the shortcut

This is the worry I hear most from the best people. Expertise, the ergonomist in the talk explains, gets built four ways: learning, experimentation, collaboration with peers, and teaching. Then he asks the question that lands hardest: when you hit a problem, what’s more effective, asking your colleagues or asking the AI? In the Q&A, someone described a senior developer who used to review a junior’s code and now just asks the AI instead. The transmission chain broke, quietly, without anyone deciding it should.

Lisanne Bainbridge named this pattern in 1983. In Ironies of Automation, the paper to forward to the teammate who wants the lineage, she showed that automating the easy parts of a job leaves the human with the hard parts, minus the daily practice that kept them sharp. A field study of AI in a radiology practice by Gamkrelidze, Zouinar and Barcellini found exactly that shape: the fracture-detection AI worked as a genuine safety net, and it also bred overconfidence risk, while the transcription AI quietly deleted the verification step the medical secretaries used to perform. The verdict: this risk is real, and it’s the one you control most directly.

Which is good news, because the fix is working agreements, not philosophy, and they’re short enough to write down:

• Rule one: anyone still learning a domain gets human review on their work, full stop, and the reviewer is a person with a name.

• Rule two: AI explains before it produces; used as a tutor it strengthens the learning station instead of skipping it.

• Rule three: seniors keep protected teaching time (a weekly pairing hour, a monthly brown-bag), because teaching is their own expertise engine too.

The shortcut only breaks the loop if you let it become the default path. What you can say: “You’re right that the shortcut is real. So we keep the loop: review stays, teaching stays, and AI joins the loop instead of replacing it.”

Working agreements protect the team’s craft. The newer worry is what the tools do to each person’s head.

The thinking muscle

This is the objection that grew loudest over the past year, and it usually arrives as a sentence about the next generation: AI is making us dumber, and kids who grow up on it will never learn to think for themselves. It now arrives with a study attached. An MIT Media Lab experiment put EEG caps on 54 essay writers: the ChatGPT group showed the weakest brain connectivity, reported the least ownership of their work, and minutes after submitting, most couldn’t quote a line from their own essay. The authors named the lingering effect “cognitive debt.” A survey of 666 people added that the heaviest AI users scored lowest on critical thinking, with the youngest participants the most dependent. This worry has data now, and waving it off with “they said the same about calculators” doesn’t answer it.

The verdict: real at the learning edge, unproven as a sentence about humanity. Read honestly, the evidence is narrower and more useful than the headlines. The MIT authors answer “is ChatGPT making us dumber?” with a flat no in their own FAQ: one narrow task, a small sample, engagement measured during the task rather than lasting change. The 666-person survey is correlational; maybe people who think less critically simply offload more. The result that should actually shape your team’s rules is causal: a randomized trial with about a thousand students, published in PNAS, found that vanilla ChatGPT boosted practice performance by 48% and then cut unassisted exam scores by 17%. Same trial, one design change: a tutor version that gave hints instead of answers erased the harm completely. The variable isn’t AI. It’s whether the AI does the thinking or scaffolds it. That’s also exactly where the calculator comparison holds and breaks: fine to offload arithmetic, dangerous to offload the reasoning itself.

The move is to make “coach, not answer machine” a written rule for anything a person is still learning. Three rules fit on a sticky note

• Think first, prompt second: write your own answer, even five rough lines, before opening the chat.

• Hints before answers while learning: tell the model “ask me questions, don’t give me the solution” (the exact design that erased the harm in the trial).

• Explain it back: if you can’t restate what you shipped with the chat window closed, it isn’t yours yet.

For the teammate who wants the deeper story, forward The Conversation’s walkthrough of the MIT study; it takes the worry seriously and shows precisely what the study can and can’t say. What you can say: “You’re right to guard your thinking. So we use AI as a coach on anything we’re still learning, and we prove it by explaining our work with the chat closed.”

Guarding heads is one half of consent. The other half is who signed up for any of this in the first place.

Consent at work

The talk asks a question I’d put to any team verbatim: do you consent to your work being organized by your team, or by an AI? Then it sketches the scenario: a backlog where tasks are assigned by an AI, estimated by an AI, and timed by an AI that checks whether you matched its estimate. If that sounds like science fiction, the EU disagrees. The EU AI Act’s Annex III classifies workplace AI that allocates tasks or monitors and evaluates performance as high-risk, with employer duties to inform workers before deploying it. The high-risk obligations start applying in August 2026. That’s next month. The verdict: they’re right about the pattern, and whether it becomes true on your team is your call to make.

Reality in the field is still mostly softer than the fear, and it’s worth being precise about that. The talk pointed at the French public sector, where the Lyon school district runs an AI copilot called Cassandre that helps HR staff answer teachers’ questions about transfers. It doesn’t decide the transfers. The speaker’s warning is about where the road leads: to “it wasn’t a human who decided,” the sentence that ends accountability.

Parts of the private sector are already further down that road, and the two clearest cases come from companies everyone knows. Amazon’s own legal filings describe warehouse systems that track each worker’s rate and auto-generate productivity warnings and termination paperwork, with Amazon noting supervisors can override. And in 2023, an Amsterdam appeals court ruled that Uber had deactivated drivers through solely automated decisions, the practice the drivers’ union called robo-firing and the exact thing the GDPR’s Article 22 exists to prevent. The road the talk warns about doesn’t lead somewhere hypothetical; it leads to case law.

The move is to make adoption consent-real before anyone asks, with three rules written into your working agreements:

• Tools are offered, never mandated per person.

• Every decision that touches a person carries a named human owner, and the name is written next to the decision.

• Opting out is safe, with no career shadow, and you check twice a year that it’s still true in practice.

When legal wants the source, forward the Annex III text itself; it’s shorter than most policy memos. Nobody who works for you should ever hear “the AI decided.” What you can say: “A human owns every decision on this team, starting with me.”

And underneath every consent question sits the one your team member is actually asking.

The jobs

The ergonomist tells a story from his early career. His mentor asked: if you design software that makes the work twice as fast, how many workers are left in six months? It’s the question underneath every other question, and your team member asking it deserves better than the two stock answers: the vendor’s “nothing changes for you” and the doomer’s certainty. The talk makes the hollowness concrete with translators, a craft told for years that the tools would free them for “more interesting work,” by people who never say what work is more interesting than the craft itself. You can’t answer the mentor’s question for the industry. Nobody can. The verdict: partly true, partly unknowable, and the worst response is a promise you can’t keep.

Start by questioning the premise. “Twice as fast” itself deserves doubt: METR’s randomized trial found experienced developers were 19% slower with AI tools while believing they were 20% faster. That perception gap is the strongest argument you’ll ever get for measuring your own team’s gains instead of quoting the pitch. And watch what the work becomes, not just whether it exists. When France’s tax authority deployed its pool-detection program Foncier innovant, field agents became desk agents. The job survived; the parts of it people liked didn’t all survive with it. Even the jobs that stay change texture, and naming that out loud buys more trust than any reassurance.

The move is to commit only to what you control, and to make each commitment checkable. Plans get heard from you first, not from a rumor: say it in the team meeting and repeat it every quarter. Skills investment happens on company time, with a number attached (a monthly learning day, a named course budget); an unnumbered commitment isn’t one. Roles get redesigned with the people in them, not around them: the person whose work is changing sits in the room where it’s redesigned. And when someone wants more than reassurance, forward David Autor’s essay on AI and the middle class: MIT’s leading labor economist, long a tech skeptic, arguing a future with neither doom nor denial in it. What you can say: “I can’t promise what the industry does. I can promise you’ll hear plans from me first, and that we invest in you while things change.”

One objection left, and it aims at the output itself.

The output you can’t trust

The talk’s most memorable stretch is about mediocrity. Models weight toward the most probable output. Ask for a pink elephant with wings and most image models fail, because the training data holds no pink elephants with wings. Ask for a wine glass filled to the brim; you get one filled halfway, over and over. The speaker’s conclusion: a designer who uses AI to generate ideas is stepping onto “a highway to the most probable solution,” and originality lives off that highway. Stereotypes ride along for the same statistical reason. The verdict: half right, and the right half is a map of where to use the tools, not a case for refusing them.

Because notice what the talk itself endorses: transcription, subtitles, handwriting recognition, classifying feedback. Small, checkable tasks with human expertise in the loop. That generalizes into the single most useful screen I know, and it’s worth writing exactly as a rule: AI-assisted work ships with a named human checker, and if nobody can check it, it doesn’t ship. Research gives the screen teeth. A CHI 2025 study of 319 knowledge workers by Microsoft Research and CMU found that the more people trusted the AI, the less critically they thought; the more they trusted their own judgment, the more critically they engaged. The check only happens when the human still believes their judgment beats the tool’s, so your job is to keep that belief alive. Taste, direction, and originality stay human, and they get more valuable, not less. For the teammate who wants the domain-by-domain version of “what AI can and can’t do,” the book is AI Snake Oil, by two Princeton computer scientists who respect skeptics enough to argue with evidence. What you can say: “You’re right about the median. That’s why we use it where we can verify, and why your judgment is the part that never gets delegated.”

Seven concerns, seven verdicts. Put together, they stop being a list of complaints and start being something else: the specification for a practice.

The objection to this whole approach

A sharp skeptic will see the framework coming and name it: “You’d already decided we’re adopting. The sorting is just a softer road to the same forced outcome.” You might be silently raising the manager’s version of the same objection: there’s a mandate, there’s a quarter, and this looks like a philosophy seminar. Both deserve a straight answer.

Take the skeptic’s version first, because it has a clean test. The sort has real exits, or it’s theater. If the “true” verdicts changed what you adopt, if something on your roadmap moved to out-by-default, if the person with the values objection still has their standing and their role, then the sorting was real. If you ran the conversations and nothing in your plan changed, the skeptic was right about you, and no framework fixes that.

Now the mandate pressure. Here’s the thing I’d want leadership to hear: the triage is the fast path, not the slow one. The alternative to honest sorting isn’t a converted team. It’s tools that get opened once, licenses that show up in the usage report as a flat line, and an adoption metric nobody believes. Quiet non-use is the most expensive outcome an AI mandate can produce, and steamrolling buys it wholesale. Sorting removes the actual reason people were stalling. That’s not a seminar. That’s unblocking.

And when the sorting is real, something better than compliance shows up: a posture you can write down.

The practice you end up with

Pull the seven moves together and you get a short team AI charter, written from your team’s concerns instead of a vendor deck. Mine would fit on a page. Four blocks.

• Where AI is in: tasks with verifiable outputs that a human reviews and owns. Transcription, classification, drafts that get read before they ship, code reviewed by a person who could have written it.

• Where it’s out by default: original ideation, unreviewed customer-facing output, and any decision about people.

• Who owns what: tools offered, never forced on a person; a named human accountable for every decision; an opt-out without career shadow.

• How we pick tools: providers that publish honest numbers on energy, water, and training data.

That last block has teeth because of the reporting gap from the planet section. A provider that understates its emissions 7.6 times has already told you how it handles inconvenient numbers. “Publishes honest figures” is a real screen, not a platitude. It costs nothing to apply and it sorts the market fast.

You can draft the whole thing in one team hour, and drafting it together is most of its power. Fifteen minutes: everyone writes their concerns down, anonymously if that helps people say the real one. Fifteen minutes: sort them into the four kinds together, out loud. Twenty minutes: draft the four blocks from what actually came up, not from a template. Ten minutes: put one name next to each block and a date three months out to revisit. A charter the team wrote in an hour beats a policy nobody read.

Two things make this charter travel further than your team. Legal will notice that the consent block is where EU compliance is already heading, with workplace AI obligations applying from August 2026. And the room where adoption metrics get discussed will notice that a written posture beats a usage dashboard nobody trusts. The line that carries all of it: “You’re right about X. Here’s what we changed so our use doesn’t have that problem. And here’s where I still think it’s worth it.” A manager who can complete that sentence for each concern doesn’t need to win any argument.

One reframe before you go write yours. I started this project wanting keys to change skeptical minds. I ended it convinced that’s the wrong target. Mindsets move when the practice becomes trustworthy, not when the pitch improves. The skeptic doesn’t need to love AI. They need to see that their concern changed something real.

Trust your skeptics

Somewhere in your team is the person who read the water numbers, the one who worries about the junior who never gets reviewed anymore, the one who wonders what these tools are doing to how we think, the one who quietly decided prompting isn’t the craft they signed up for. They were never the obstacle. They were the free red team, stress-testing your AI practice before it shipped. The org-behavior researchers knew it in 2008; the talk’s speakers knew it on that stage in early 2025. A team that sorted its concerns ships AI use it can defend out loud, starting with defending it to itself.

Your turn. Take the next 1:1 where a concern comes up and sort it out loud, all four kinds on the table. Write the strongest version of your best skeptic’s objection before you meet them, and open with it. Draft the four-block charter with the team, not for them. Pick one thing that goes out-by-default this week, and say why. And watch the talk that started this article, then forward it to the person on your team it will resonate with, with a note saying you want their read.

The speakers closed with a warning: AI will only ever offer you a mediocre dream, so take the time to dream your own. I’d turn it forward. Your team already has a dream about what their work should mean. Make your AI practice serve that, on purpose, and the skeptics won’t need convincing, they’ll be the ones holding you to it.

Here’s the problem: AI just made polished documents free. The tight résumé, the crisp case structure, the take-home that used to signal “this person is serious”: any candidate with Claude and an evening now produces all three. You’ve probably felt it already, reading applications that all sound suspiciously excellent. And it happened at the worst possible moment, right when you were asked to make your team AI-first and every open seat became a chance to hire the mindset. The signal collapsed at the exact moment the stakes went up.

So I went and read what the companies furthest into this shift actually did about it. I pulled the live PM job posts at Anthropic, Ramp, Notion, and Stripe, straight from their own job boards this week. I read the interview loops Sierra and Canva published, and the one candidates describe at Shopify. I read the fluency rubric Zapier now applies to every single hire. Different companies, different stages, one converging move. Stop grading the deliverable. Watch the candidate work. The bar isn’t “knows AI.” It’s “directs AI and catches it when it’s wrong.”

Here’s the whole arc in one picture.

• The diagnosis. I’ll show you why every stage of your current process stopped measuring anything real.

• The rebuild. Five moves, one per stage, with full requirements quoted from live postings and the interview questions from companies that already made them.

• The calibration. You’ll see the data on how much AI to actually demand, and the walk-back that warns against overshooting.

• The walkaway. One forwardable paragraph for leadership and your recruiter, plus a checklist to run on the req you have open.

By the end you’ll have a full replacement for the old process: job-description language you can borrow whole, two screening questions, a live interview format with a published AI policy, a work-sample design, and a decision rubric, all compressed into a closing checklist you can run this week. Your next PM hire stops being a coin flip on polish and becomes the clearest AI-first signal you send your team this year.

Let’s rebuild it, stage by stage.

Why your hiring loop went blind

Walk through the classic stages and name what each one actually measures. The résumé screen grades a document. The case interview grades a rehearsed structure, the kind a decade of candidates drilled from Cracking the PM Interview, McDowell and Bavaro’s 2013 prep book (”How many pizzas are delivered in Manhattan?”, “How do you design an alarm clock for the blind?”). The take-home grades another document, produced somewhere you can’t see. The panel grades a performance, and every stage grades some kind of output.

For years we told ourselves polish correlated with skill, that producing a crisp PRD required a crisp mind. The honest version is less flattering. Selection studies have long ranked résumé screens and unstructured interviews among the weakest predictors of actual job performance. The old process wasn’t a good instrument that AI broke. It was a weak instrument whose weakness AI made undeniable.

Picture the machine you’ve been running.

There’s a famous precedent for this kind of audit. Back in 2013, Google ran the numbers on its own famously clever interviews. Laszlo Bock, who ran People Operations there, went public with the verdict: brainteasers were “a complete waste of time” that “serve primarily to make the interviewer feel smart.” Google moved to structured interviews as a result. The lesson stings a little: a signal-free process can feel rigorous for years, because nobody checks. AI didn’t create the weakness. It removed the excuse for ignoring it.

You can watch the collapse happen in real interviews. In Nikhyl Singhal’s The Skip (Singhal ran product at Meta and Google), hiring manager Sam Stone observes that candidates who used AI to prep their case structure “will abandon the structure the moment Q&A starts, because they never internalized it.” In the same piece, product leader Mckenzie Lock shares the probe she used when screening at Netflix: when new information invalidates an assumption, “can you say ‘oh, that changes things’ and walk through updated logic?” And Meta gives the same diagnosis from the company side. Its new interview format puts an AI assistant inside the session, partly because, in Meta’s own words, it “makes LLM-based cheating less effective.” The companies that moved first aren’t fighting AI-polished candidates with detection tools. They moved the test to where polish can’t fake it.

Meanwhile the requirement itself went up, not down. In Microsoft and LinkedIn’s Work Trend Index, a survey of 31,000 people, 66% of leaders said they wouldn’t hire someone without AI skills. That survey is from May 2024, and nothing published since suggests the number softened. So you’re expected to hire for a skill your instruments can’t see, using stages that stopped measuring anything. The fix isn’t a better lie detector. It’s a process that watches work instead of grading output. Five stages, five moves.

Rebuild the loop stage by stage

Everything below comes from choices real companies have published, plus my own read on how to adapt them to a PM seat. Treat the five moves as parts to adapt, not a recipe to obey. Your org, your seat, your constraints. Run the ones that fit.

Here’s the full map before we go stage by stage.

Write AI into the job description

Stop writing “familiarity with AI tools is a plus.” The most advanced companies on this front put AI usage in the requirements section and name tools, the way postings used to name SQL. The strongest reqs treat AI usage as a requirement with teeth, not a vibe. All of these lines come from live postings I pulled off the companies’ own job boards this week, quoted in full so you can borrow whole requirements, not fragments:

• Notion (PM, Workspaces, an everyday workspace-product seat): “AI-pilled: You have comfort and an ongoing curiosity with using AI tools for your product development process.” And on every one of its 141 live postings, Notion adds: “You don’t need deep AI expertise for every role, but we do expect every Notino to be intellectually curious, drawn to tinkering and discovery, and excited to use AI as a real collaborator in their work.”

• Supabase (PM, Interfaces; the line recurs across their PM postings): “You use AI to move faster. You use AI tools to compress the slow parts of the job (research, drafting, synthesizing feedback) so you spend more time on judgment calls, and you’ve built enough with agents to have real opinions on what makes an interface agent-friendly.”

• Airbnb (PM, Incubations, a consumer seat asking for 10+ years of experience): “Demonstrated curiosity and point of view on incorporating AI tools into the product workflow.”

• Canva (PM for Pexels, their stock-photo product): “We’re looking for an AI-embracing entrepreneurial type product manager.” And in the experience list: “You embrace AI for automation, prototyping, report generation, data analysis, etc.”

• Stripe (growth PM, in Minimum Requirements): “Experience building AI-powered, conversational, or agentic products. You understand what it takes to ship AI that works reliably for real users, not just in demos.” And under preferred qualifications: “You have shipped products on top of these technologies, not just evaluated them.”

• Ramp (PM, Agentic CX): “We’re looking for builders who can vibe-code a prototype before lunch and write the rollout plan after.” The requirements spell it out: “Deep hands-on experience building with AI: you’ve prototyped, shipped, and iterated on AI tools, agents, or workflows, not just managed roadmaps about them. Technical fluency with modern AI coding harnesses (Cursor, Claude Code, Codex).” And: “Fluency in data and AI evals: you know when to trust numbers and when to lean on principles.”

• Anthropic (research PM, in Minimum Qualifications, in full): “Have a deep passion and curiosity for AI and LLMs. Use AI regularly.”

One distinction before you borrow. At Anthropic or on Ramp’s agentic products, asking for AI depth is as natural as a payments seat asking for payments experience: the product is AI, so the requirement writes itself. The lines worth copying are the other kind: a workspace product, a home-rental team, and a stock-photo library writing AI usage into ordinary PM seats. That’s the expectation spreading beyond AI products, and it’s the part that applies to your posting whatever you build.

Now decide which ask you’re actually making. The postings above split into three distinct tiers, and mixing them up is how job descriptions turn into wish lists.

Tier one asks for a PM who works faster with AI: the Notion, Supabase, and Airbnb language. Tier two asks for a PM who prototypes with AI coding tools by name: the Ramp language, echoed in Anthropic’s Labs PM posting (”Prototype with AI tools like Claude Code”). Tier three asks for a PM who owns evals, the tests that measure whether an AI feature actually works: Anthropic’s Claude Code PM posting wants someone who has “personally built agentic evals,” and Figma and Databricks write eval ownership straight into PM responsibilities. Most seats need tier one or two, and writing all three into one req doesn’t get you a unicorn, it gets you an empty pipeline. Pick the tier, write two or three specific bullets, and move on. The req is now making a promise the rest of your process has to keep.

Screen for their last real AI workflow

The old screening question (”are you familiar with AI tools?”) is answerable by anyone with a pulse and a ChatGPT subscription. Replace it with two questions that can’t be prepped.

• First: “Walk me through the last thing you used AI for, end to end.”

• Second: “When has AI been confidently wrong for you, and how did you catch it?”

Both were published by Metaview, a recruiting-software company, building on the fluency rubric Zapier’s co-founder Wade Foster made public. Vendor source, but the questions stand on their own. Ten minutes with these two questions tells you more than an hour of tool-name bingo.

The second question is the real detector. Anyone who genuinely works with AI has been burned by it: the hallucinated citation, the confidently wrong data pull, the beautiful answer built on a misread premise. They’ll tell you the story with texture and a little embarrassment, and then, crucially, they’ll tell you what they changed about how they verify. A candidate who has only read about AI gives you abstractions about “hallucination risks.” Real usage leaves scar tissue, and scar tissue can’t be improvised in a screening call. The question is kind, too: you’re inviting a story they’ll enjoy telling, not running a quiz.

Zapier, the most codified case I found, is worth mining beyond the two questions. It checks AI fluency at four fixed points for every hire in every role (”The application, Initial (human or AI) screen, Skills test, Executive interview”) and probes four components: “AI mindset, strategy, building, and accountability.” The detail worth stealing came in the rubric’s second version: they now assess “AI fluency slope, not a snapshot,” the candidate’s trajectory rather than today’s toolkit. Listen for the slope in the workflow answer: someone whose story ends “and last month I changed how I do it” is climbing; someone reciting a setup from a year ago has plateaued. Metaview’s read of the same rubric describes what the top of the scale sounds like: someone who “redesigns the work around AI” and “builds systems, prompts, or agents that other people use.”

You don’t need four checkpoints. You need these two questions asked early enough that nobody spends three weeks interviewing someone who fails the bar in minute ten. Your recruiter can ask them word for word, which is exactly the point. The candidates who clear them earn the round where things get interesting: the one with AI in the room.

Put AI in the interview room

Next decision: what happens when a candidate wants to use AI in your interviews? Right now most teams have no answer, which is the worst answer. Your interviewers assume it’s banned, your candidates assume it’s allowed, and whatever happens tells you nothing either way. The companies that took a position landed on three different doors, and all three work. The only wrong policy is not having one, and not telling candidates before they walk in.

• Door one: require it. Canva insists that engineering candidates use AI tools in its technical interviews, because the old process “asked candidates to solve coding problems without the very tools they’d use on the job.”

• Door two: allow it and score it. Shopify runs two AI interviews, and as candidates describe them, “Any AI tool is fair game. Cursor, Claude Code, ChatGPT, Copilot, whatever you normally use.”

• Door three: keep the live round unassisted. Anthropic publishes its candidate policy: draft your application yourself and refine it with Claude, but the live interview is you alone.

These examples all come from engineering hiring, and that’s worth being honest about. The policy decision transfers to a PM loop as-is; the exercise formats need adapting, and the work-sample stage does that adapting.

Once AI is in the room, score direction, not output. One Shopify interviewer, in those same candidate reports, put the rubric in one line: “We don’t want the AI to work for you. We want you to tell it what to do.” Canva’s own interviewer guide tells candidates to ”take time to review and understand everything that gets generated”. And Hello Interview, the site documenting these formats, states the stakes plainly: “The moment it starts making the architectural calls while you sit and watch, you’ve given away the exact thing the interview is there to assess.” One Rippling candidate was rejected for having “relied too heavily on AI even though their initial approach was correct.” Over-delegation is a scoring axis now, not just under-use. Watching someone accept AI garbage without blinking is the most informative thirty seconds in the whole process.

For the conversational rounds, the questions change too. Aakash Gupta, who coached dozens of PMs through these rounds and collected the exact questions they got asked, reports prompts like: “Your chatbot’s response quality dropped 15% last week. How do you debug this?”, “Tell me about a time you shipped an AI feature that backfired,” and “Explain embeddings to a product designer on your team.” Notice what these test: judgment about systems that fail probabilistically, not feature-design theater. Lift the ones that fit your product and retire a case study to make room.

Swap the take-home for a live build

The take-home is the most broken stage in the old process, because it’s a pure artifact produced entirely off-camera. You’ll be tempted to keep it and bolt on AI-detection. Don’t. Detection is a race you can’t win, and it aims at the wrong question: you don’t actually care whether AI touched the deck, you care whether this person can think. So replace the stage entirely: watch the work happen live.

There’s solid science behind this swap, and it’s older than the AI wave. A live build is a work-sample test, an instrument that personnel-selection research has ranked among the strongest predictors of job performance for decades (Schmidt and Hunter’s 1998 meta-analysis of 85 years of hiring data; the 2021 revision by Sackett and colleagues puts structured interviews at the very top). You’re not chasing a trend. You’re adopting the instrument hiring science recommended all along, now with AI in the room.

The PM version already exists in the wild, usually as a 45-minute prototyping case. Aakash Gupta has coached PMs through this round at “v0, Bolt, Figma, and Perplexity,” with real prompts like “Design Google Maps for the Blind” and “How would you prototype a Facebook Dating app,” and candidates choosing tools like Lovable, Bolt, or v0 for prototyping, or Cursor and Replit if they lean technical. Lewis Lin, who has written PM interview prep books for a decade, describes what it actually measures: “rapid problem-solving and product judgment under pressure,” not code quality. One calibration note so you don’t over-index: Google has only piloted this round, and it’s ”not a standard part of Google’s process (yet)”. Where it does run, Jaclyn Konzelmann, Director of AI Product at Google, has described what she looks for. You’re early, not late.

If you want a deeper shape to borrow, Sierra published its redesigned onsite in April 2026: Plan, a working session where “the candidate drives ideation, while interviewers ask questions to strengthen it”; Build, two hours with “the AI tooling and frameworks of their choice”; Review, where the team debates “the key product flows and choices they made.” It’s their engineering loop, so here’s my PM adaptation: plan a feature together for twenty minutes, give them a time box and whatever tools they like, then spend the last half hour on why they built what they built. What did they cut, what did they verify, what would they ship. The prototype is just the excuse. The review conversation is where the judgment shows.

Decide with a bar, not a ranking

Now the debrief, where all of this either compounds or unravels. The move: treat AI fluency as a minimum bar, and product judgment as the ranking. You’re not hiring the most AI-enthusiastic candidate in the pool. You’re hiring the best product judgment among everyone who cleared the bar.

Make the bar concrete, and borrow Zapier’s “Capable” level to do it: AI embedded in their actual work, “repeatable systems, not one-off prompts,” and clear impact on quality or speed. A candidate who can’t show a single real workflow in the screen, and who delegates every decision to the AI in the live build, doesn’t clear it. That’s a no in 2026 the same way “can’t write a PRD” was a no in 2019. But clearing the bar higher earns no extra points, because the ranking runs on something else.

The data backs the weighting. Across 592 AI-PM postings analyzed in early 2025, product management and cross-functional collaboration each appeared in 32% of listings, while prompt engineering showed up in 9%. Even the roles with AI in the title rank on classic judgment. The sharpest new skill to probe for is evals. Kevin Weil, who was leading OpenAI’s product org at the time, said it plainly in April 2025: “Writing evals is going to become a core skill for product managers.” Honest calibration: eval experience is a hard requirement for AI-product seats (it’s literally in Anthropic’s and Figma’s postings) and a strong differentiator everywhere else. Match the evals expectation to the tier you picked when you wrote the req, and don’t punish a tier-one candidate for missing a tier-three skill.

One more upgrade, borrowed from decision science. In Noise (2021), Daniel Kahneman, Olivier Sibony, and Cass Sunstein show how wildly hiring judgments vary between judges, and prescribe the fix: score each dimension independently, and delay the overall verdict until the end. “Fluency is a bar, judgment is the ranking” is that protocol in one line. And in your judgment scoring, weight the moment Mckenzie Lock’s probe targets: did the candidate ever say “oh, that changes things” and re-reason live? That moment can’t be faked, and it’s the single strongest signal your new process produces. Which leaves one question: how hard should you actually push the AI requirement?

How much AI to actually demand

Here’s the objection you might already be forming: “We’re not Ramp or Anthropic. Most PM postings don’t ask for any of this. My next PM still needs discovery, stakeholder work, and judgment. Isn’t rewriting the whole process premature?” The data honestly supports half of it. Itamar Gilad’s analysis of TrueUp data found that roughly one in seven open PM roles asked for AI skills as of early 2026, and he’s blunt about the hype: “There’s no indication that all PM roles will require deep AI knowledge and skills.” I’ll add my own check: when I scanned OpenAI’s 13 live PM postings this week, most carried no AI-usage bullets at all. At AI-native companies the expectation is ambient culture, not req language, and the explicit bullets cluster at companies making the shift deliberate. If your org writes its expectations down rather than absorbing them by osmosis, the bullets above are your template.

But look at what the objection is actually objecting to. The rebuilt process isn’t a higher AI bar bolted onto every stage. It’s better instruments for the things you already hire on. Even your most classic hire, the discovery-and-stakeholders PM with no AI ambitions, now walks through a world where every résumé, case answer, and take-home can be AI-generated. The screen question, the live build, and the re-reasoning probe measure judgment better than the old stages did, whatever the seat. Fix the instruments because they’re broken, not because every PM must become an AI specialist.

The walk-backs teach the closing lesson. Duolingo went AI-first loudly in April 2025, and a year later its co-founder Luis von Ahn dropped the idea of evaluating employees on AI usage: “the most important thing in your performance is that you are doing whatever your job is as well as possible. A lot of times AI can help you with that. But if it can’t, I’m not going to force you to do that.” His mandate was about evaluating current employees, not hiring, but the lesson transfers to the bar you set at the door. Don’t make AI zealotry the bar. Make judgment-with-AI the bar. The process you just rebuilt does exactly that: fluency is one gate, and it’s never the ranking.

Start with the req on your desk

When leadership or your recruiter asks what’s changing, here’s the paragraph to hand them:

AI made polished artifacts free, so our hiring process can’t grade artifacts anymore. We’re redesigning it to watch how candidates work: the req names real AI usage, the screen asks for their last AI workflow end to end, the interview puts AI in the room with a policy we publish, the work sample is a live build we review together, and the decision holds a fluency bar while judgment keeps the casting vote. The bar isn’t “knows AI.” It’s “directs AI and catches it when it’s wrong.

It travels well: to leadership, to your recruiter, to the staff PM who’ll run half the interviews.

A generation of hiring managers learned from Who, Geoff Smart and Randy Street’s 2008 hiring classic, to write the scorecard before meeting a single candidate. Same discipline here, one page shorter. The checklist, for the opening you already have:

• Job description: pick tier one, two, or three, and write it into the requirements as plainly as Anthropic’s “Use AI regularly.”

• Screen: the two questions, word for word, in the recruiter’s script.

• Interview: pick a policy (require, allow-and-score, or unassisted-live) and put it in the candidate prep email.

• Work sample: one prompt, their tools, 45 minutes, then thirty minutes on their choices.

• Decision: fluency is a bar, judgment is the ranking; one line in the debrief template.

Your hiring process is also a message, and not just to candidates. The team you already have reads every posting you put up and every interview you run as a statement of what the org actually values. When they see you watching how people work with AI instead of counting buzzwords, that tells them more about the AI-first mandate than any kickoff deck. The first candidate through the rebuilt loop will show you more in 45 live minutes than the last ten take-homes combined. The posting on your desk is due Friday anyway. Make it the first artifact of the new process.

Here’s the thing nobody tells you about that meeting. The spec already exists. It just isn’t written anywhere: it lives in the heads of the people who do the job today, stored in a language no data scientist speaks. What the team’s best performer would tell a new hire on day one? That’s your system prompt. The tabs she keeps open while she works? Those are your tools. The things she’d never let a new hire do alone? Those are your guardrails. Your job isn’t to write the spec. It’s to run the interviews that surface it.

In the previous post I wrote about how to pick the AI agents worth building: point the next one at your team’s real constraint instead of sprinkling agents everywhere. This is what comes after you’ve picked the spot. The whole method fits in one sentence: every component of an agent is the answer to a plain-language question, and version one should only do the work the team would happily give away. The rest is knowing which questions to ask, and in what order.

The whole method runs through one worked example, a customer support team, picked to make every step tangible: swap in finance, sales ops, or HR and the moves stay the same. Here’s the path at a glance.

• The gate. I’ll give you the five questions that tell you whether a job needs an agent, a simple script, or nothing at all.

• The interviews. Three conversations with a support team, with the exact questions to ask and what each answer is for.

• The translation. Every answer becomes a spec component, and you’ll prototype the agent yourself before anyone commits a sprint.

• The landing. You’ll see how to scope version one so a team that fears the project ends up asking the agent to do more.

By the end you’ll have the full interview script and the translation table between business answers and agent components. You walk into your next discovery meeting with questions the team can actually answer, and you walk out with a spec your AI team can actually build. There’s no platform to pick: the method works with whatever agent stack your company already chose (Amazon Bedrock AgentCore, Databricks, or something your AI team built themselves), and an AI workspace like Claude Cowork or Claude Code covers the prototype step.

Grab your notebook.

Decide if the job needs an agent

The oldest failure mode in innovation has a name: a solution in search of a problem. A technology shows up, it’s impressive, and whole organizations start asking “where could we use this?” The question is backwards, and the people who build things for a living have been saying so for years. Uri Levine, who co-founded Waze, put the fix on the cover of his book: fall in love with the problem, not the solution. Problems are constant; solutions are just the current best way to serve them. Nobody in your company needs an agent. They need the backlog to stop growing, the customer answered today, the notes written without staying late.

AI mandates invert this by design. “Agentify support” hands you a solution with the problem line left blank, and your own excitement pulls the same direction, because agents are genuinely fun to build. I’ve caught myself doing it: tool first, problem later, demo nobody asked for. So the first discipline of this job is to flip the sentence back. Start from the work, find the problem worth solving, then ask whether an agent is even the right shape of solution. Your job isn’t to bring agents to the business. It’s to bring the business a problem worth solving, with whatever technology fits.

The “right shape” question has a sharp edge to it. Anthropic’s Building Effective Agents draws the line I use in every scoping conversation. A workflow follows a path you script in advance: cheaper, faster, predictable. An agent decides its own path as it goes, and Anthropic reserves that for problems where “it’s difficult or impossible to predict the required number of steps.” In plain terms: if you can draw the flowchart, script the flowchart. An agent earns its complexity only when every case bends the flowchart.

I run every candidate job through five questions before committing to anything. The team can answer all five without a single technical term.

1. Does the work arrive different every time? If every item looks the same, a script handles it. Variety is the first hint you’re in agent territory.

2. Does handling it take judgment calls, or rules? If the team can write the rules on a whiteboard in an hour, write the rules into code instead.

3. What does a wrong answer cost, and can a human catch it in time? High stakes plus no review point means you’re not ready for an agent here, whatever the mandate says.

4. Does it happen often enough to matter? A judgment-heavy task that shows up twice a quarter never pays back the build.

5. Can an agent even reach the work? If the job lives in phone calls, hallway conversations, and a cheat-sheet in someone’s drawer, no spec survives contact.

OpenAI’s practical guide to building agents puts it bluntly: validate the use case first, “otherwise, a deterministic solution may suffice.”

Two contrasting tickets show the gate at work.

• A password reset arrives identical every time, follows three rules, and costs nothing when retried: script it, done in two weeks.

• A refund dispute arrives wrapped in order history, customer tone, and policy exceptions: that’s judgment, that’s volume, that’s agent territory.

The refund example isn’t even mine: OpenAI’s guide names “refund approval in customer service workflows” as its canonical case of agent-worthy work, the kind full of “nuanced judgment, exceptions, or context-sensitive decisions.”

And sometimes the gate says no. Telling a team “you don’t need an agent, you need a script, and it’ll be done in two weeks” feels like walking away from the mandate. It’s the opposite. Saying “no agent needed” when it’s true is the cheapest trust you’ll ever buy, and it’s the reason the next team opens their door when you knock.

For the team you’re about to meet, the gate says yes. Time to walk in.

Run the three interviews

The use case: a support team under a mandate

Time to run the method on something concrete, with you in the driver’s seat. Picture a mid-size B2B software company. A nine-person customer operations team handles order issues, refund disputes, and account changes: roughly sixty cases a day, forty minutes of handling time on an average case, a backlog that grows a little every month, and a hiring freeze. Leadership looks at those numbers and hands you the mandate: “bring AI to support.”

The team lead, call her Claire, runs a team that’s proud of its judgment. They talk customers out of churning every single week. What Claire hears in “agentify support” is a project aimed at her team’s future, and she’s not wrong to hear it that way: plenty of these projects get pitched exactly like that. The fear is rational. Pretending the fear isn’t in the room poisons every interview that follows, so name it in the first meeting and don’t argue it away. You can’t logic someone out of a fear the headlines keep feeding. You can only design the project so the fear doesn’t come true, and let the design speak.

The posture you take matters as much as anything you say. You’re not the expert arriving to optimize anyone. You’re the apprentice, and they’re the master craftsman: you’re there to learn the job well enough to describe it faithfully. An apprentice studies the master’s craft; the master stays the master. That single stance changes what people tell you. Experts perform for auditors. They teach apprentices.

One more reframe before the first interview, this time for leadership. “AI in support” is not a deliverable. A deliverable is a number: minutes back per case, backlog age, first-response time, with quality held steady on the numbers the team already reports, like satisfaction scores and escalation rate. Commit to measuring before and after. An agent without a number attached is a demo, not a delivery.

So you ask Claire for three slots on her team’s calendar, spread over one week: an hour with her at a whiteboard, ninety minutes with her two best people, and half a day sitting behind someone while they work. Plus ten recent cases: good, bad, and ugly.

The interview format descends from contextual inquiry, a research method Hugh Beyer and Karen Holtzblatt developed at Digital Equipment Corporation in the late 1980s. Its core instruction: study the work as it happens, not the process as documented, in the posture of an apprentice learning a craft. Every team has a standard operating procedure, the SOP: the official document that says how the work is supposed to run. It describes the happy path, and the job lives in the exceptions. Never ask a business team “what are your requirements?” Nobody who does real work can answer that question. Ask about last Tuesday instead.

Three conversations, three different jobs. Here’s the map.

Interview 1: map the job

One hour with Claire and a whiteboard. The goal is the map, not the details. Five questions do the work:

• ”Walk me through what happens between a case landing and a case closing. Whiteboard-level.”

• ”Where does the time actually go? Not where the process doc says. Where it really goes.”

• ”Which cases does your team dread? Which ones do people fight over because they’re interesting?”

• ”What work would your team happily never do again?”

• ”What numbers do you already report on?”

One hour of those questions produces something like this on the whiteboard.

Each answer has a destination:

• The walkthrough becomes the workflow skeleton.

• The where-does-time-go answer becomes your pain heatmap.

• The dread-versus-fight-over answer tells you what the team wants to keep, which matters more than what they want to lose.

• The last question hands you a value baseline without building a single new dashboard.

• And the “happily never again” question is the one to listen hardest to, because its answer becomes version one. For Claire’s team the answer is unanimous: the case wrap-up notes. Every closed case needs a summary in the CRM, the customer database where every interaction gets logged. Everyone writes them badly at 6pm. Nobody reads them until they desperately need them. Hold that thought.

Interview 2: learn the craft

Ninety minutes with the two people Claire names as her best: Marco, eight years on the team, and Ana, three. Screens open, real cases on them. This is where the agent’s actual behavior gets specified, one plain question at a time:

• ”Show me the last case that took you over an hour. Click through it exactly as you did.”

• ”What would you tell a new hire on day one that isn’t written anywhere?”

• ”What’s open on your screen while you work a case?”

• ”What do you find yourself re-explaining or re-looking-up, case after case?”

• ”When do you stop and pull in Claire or another team? What’s the trigger?”

• ”What would you never let a new hire do alone in their first month?”

The answers are the spec, verbatim. Marco’s day-one rule: “if the customer mentions a deadline, handle it today, even if the queue says otherwise.” That sentence goes straight into the agent’s instructions. His open tabs: the CRM, the order database, and the refund policy wiki page, which is outdated, and everyone knows exactly which parts to ignore. Write down which parts they ignore. That’s institutional knowledge no document holds. Then Ana pulls a laminated cheat-sheet out of her desk drawer, and that’s when you know the interview is working. Every team has one. It’s never in the wiki. And the never-alone list arrives without hesitation: refunds above a threshold, anything that smells legal, and any promise about delivery dates.

Interview 3: shadow the work

Half a day sitting behind Marco and Ana while they work real cases. No questions this time. You’re watching for the gap between the process document and the craft: the case that needs an actual phone call to the warehouse, the customer Ana recognizes as burned-twice-before from memory because the CRM has no field for it, the moment Marco overrides the policy page because he knows it’s stale. None of that appears in any SOP, and all of it decides whether an agent succeeds or embarrasses you.

You leave the shadow session with ten annotated real cases: the input, what the human did, and what “good” looked like. Those ten cases are the most valuable artifact of the entire discovery, because they become the test set your AI team grades the agent against. One honesty note: your three interviews will surface completely different specifics than these. The questions transfer. The answers won’t. That’s the point of asking.

Three conversations, one notebook full of plain-language answers. Now comes the translation.

Turn the answers into a buildable spec

Write the seven components

An agent spec has seven components. Here’s the part that surprises people: the team just answered all seven, and nobody in the room had to learn a single AI term. Five components come straight from interview answers, and the remaining two get derived from those five.

1. Instructions: the agent’s system prompt. This is the day-one advice, the rules of thumb, the tone. It comes from ”what would you tell a new hire that isn’t written anywhere?” Marco’s deadline rule goes into the prompt word for word. Collect the rules as the team says them, not translated into corporate language, because the best performer’s voice is the spec. Your data scientist will thank you: half of prompt writing is knowing what the expert actually does.

2. Tools: what the agent can look at and what it can do. Every tab open during the walkthrough is a read-tool candidate. Every action taken is a write-tool candidate. The split matters because it maps directly to risk: letting the agent read the order database is cheap to grant, letting it issue a refund is not. Version one gets read access to the CRM, the order database, and the policy wiki, and exactly zero write actions.

3. Memory: what the agent keeps between cases. This comes from ”what do you keep re-explaining?” and from Ana’s burned-twice customer. Three plain kinds cover most teams: facts about repeat customers, resolutions that worked before, and this-week context like a known shipping outage. The spec says what to remember in the team’s words, and the AI team picks how to store it. Skip the vector-database vocabulary entirely; it has no place in a spec meeting.

4. Workflow: the path a case takes, where the agent sits, and what wakes it up. Interview 1’s whiteboard map plus the walkthrough give you the steps. Now mark each step with one of three labels: script it, agent judgment, or human only. The gate from earlier applies per-step here, not per-project, and most “agent” projects turn out to be a workflow with two or three genuinely agentic steps inside. Then name the trigger, because it’s the first thing your AI team will ask: what invokes this thing, an event, a schedule, or a person? Version one’s trigger is one line: the agent wakes when the human closes the case.

5. Model: the brain you rent. The business chooses a model without ever hearing a model name, by answering three questions: how much judgment does a case take, how fast does the answer need to come back, and what’s a case worth in money? Capability, latency, cost. Your AI team maps those three answers to an actual model tier, and your job is keeping the trade-off honest when someone wants the premium model for wrap-up notes.

6. Guardrails and escalation: what it must never do alone, and who it calls. The never-alone list transfers verbatim: refunds above the threshold, legal-adjacent language, delivery-date promises, always a human. Escalation deserves the same respect: the trigger points from ”when do you pull in Claire?” become the agent’s own escalation rules. An agent that says “this one’s for you, here’s the file so far” earns trust faster than one that guesses, so spec escalation as a feature, not a failure state. Add one more spec line to this bucket: every agent action gets logged and is reviewable. The weekly review you’re about to set up depends on those logs existing, and legal will ask for the trail before anything launches.

7. Evals: how everyone knows it works. The ten annotated cases from the shadow session plus the metrics Claire already reports. The launch bar is simple: the agent drafts wrap-up notes for the ten golden cases, and Claire grades them blind against the human-written ones. Hamel Husain, whose evals guide is the one practitioners pass around, puts the stakes plainly: “Unsuccessful products almost always share a common root cause: a failure to create robust evaluation systems.” The evals aren’t the homework at the end of the project. They’re the first thing your AI team builds.

Now notice what just happened. The team wrote the spec. Nobody said “prompt engineering.” Nobody explained embeddings. Your entire contribution was asking questions about last Tuesday, sorting the answers into seven buckets, and reading them back until Claire said “yes, that’s the job.”

A spec in plain language, ready for builders. Almost. There’s one move left before the hand-off, and it’s the one that separates today’s product work from requirements-era product work.

Prototype the agent yourself

Here’s the gap in everything so far: three interviews captured what the team says about the work, and the shadow session caught some of what they do. Neither tells you how they’ll behave with the agent itself. Will Marco trust the draft, edit it, or quietly go back to writing notes his way? No document answers that, and a longer spec answers it least of all. In classic product work you already know the move: you don’t write a requirements document for a website and send it to developers. You click together a prototype in Figma or a no-code tool, put it in front of five users, and watch. Agents just got the same unlock: with AI coding tools, prototyping the agent is now cheaper than the meeting about the spec. Here’s the loop.

So before handing anything to the AI team, build the smallest thing that behaves like the agent. In Claude Cowork or Claude Code, it’s an afternoon of work. Write the instructions component into a skill, with Marco’s rules verbatim. Drop the ten golden cases in as files. Ask it to draft the wrap-up note for each one. No connectors, no permissions, no access requests: copy-paste stands in for every tool the real agent will get. What comes out reads like the future agent’s output, and the output is the only part the team can judge anyway.

Then test it like a product prototype, not like a demo. Run the sessions on the five-act interview the structure Google Ventures uses to test design sprint prototypes: a friendly welcome, a few context questions, introduce the prototype (”it’s cardboard, you won’t hurt my feelings”), real tasks, then a short debrief. Hand Ana a fresh case, let her work it, show the prototype’s note next to hers, and watch. The watching surfaces what three interviews didn’t: every note the team actually trusts ends with a who-to-call line, and nobody thought to say it out loud. The spec gains its most important line from twenty minutes of watching, not from any question you asked. There’s always a gap between what people tell you and what they do, and it closes only in front of something real.

The prototype doesn’t replace the spec, it hardens it. The spec still tells your AI team what to build. The prototype proves the team will use it, catches misreadings while they’re an afternoon cheap, and hands your data scientist a tested first draft of the prompt instead of a blank page. And the fearful room from the kickoff has now touched the thing, shaped it, and seen it stay a drafting assistant. The hand-off package just gained its most convincing item.

Hand it to your AI team

The hand-off package is six things, and none of them is a forty-page document. One page per spec component, in the team’s own words plus the derived decisions. The ten golden cases with the human answers attached. The prototype with its feedback log: what the team reacted to and what changed. The access list: which systems, read or write, granted to whom. The value baseline from Interview 1. And the escalation contacts with their triggers. That package answers most of the questions a AI team otherwise burns weeks discovering one standup at a time.

Each role picks up a different end of it. The data scientist starts from component seven: the golden cases become a test harness before anyone writes a prompt, which is exactly the order Husain’s evals-first practice prescribes, and the prototype’s skill file gives them a first prompt draft that already survived contact with the team. The data engineer starts from component two: connectors, permissions, logging. And you stay in the loop as the translator, running one ritual that outweighs everything else: a weekly transcript review, with someone from the support team in the room. Marco reading the agent’s attempts catches misreadings in days. Without the business in the review loop, misreadings surface at launch, in front of customers, with the team’s name on the queue.

Somewhere in that first planning meeting, you’ll get the pushback. It usually sounds like this: “Why all this ceremony? The process is documented. We can spec from the SOPs and ship this quarter.” Take it seriously, because it’s half right: the SOP is real and useful. It’s also only the happy path. The value and the risk both live in the exceptions only the team can name: the stale policy page, the burned-twice customer, the warehouse phone call. An agent spec’d from documentation automates the part that was never the problem, and breaks publicly on the part that is. And it breaks in front of a team that was never asked, who now has every reason to let it fail. That’s not a hypothetical. MIT’s Project NANDA studied hundreds of enterprise GenAI initiatives in 2025 and found most fail from “brittle workflows, lack of contextual learning, and misalignment with day-to-day operations”: the documentation-spec’d agent, described in one sentence. One week of light discovery, against a quarter of rebuilding. That’s the whole trade.

One last thing about the spec itself: it’s a living contract, not a document you sign once and file away. When the AI team hits an ambiguity, and they will (”what exactly counts as legal-adjacent?”), the answer is another fifteen minutes with Marco, not a guess hidden inside a sprint. The interviews don’t end at discovery. They just get shorter.

The build is underway. Now the part most agent projects skip until it’s too late: getting a team that fears the thing to want the thing.

Start with the work nobody wants

Go back to Claire’s face in that kickoff meeting. You don’t fix that with a slide about how AI augments people. Nobody who’s afraid has ever been reassured by a slide. You fix the fear with scope.

The move: version one does only the work the team would happily give away. Remember the unanimous answer from Interview 1: the wrap-up notes. So version one drafts the case summary after the human closes the case. It talks to no customer. It makes no decision. It sends nothing. Every person on the team keeps every bit of their judgment, and everyone loses their least favorite chore the same week. The French have a name for this kind of maneuver: cheval de Troie, the Trojan horse. Except this horse gets welcomed for what it visibly removes, not for what it hides. There’s no deception in it. The full ladder is printed on the side.

Be honest about version one’s number, because it’s small on purpose. Five minutes of note-writing per case, sixty cases a day: call it five team-hours back daily, plus CRM notes that people can finally use. That’s real, and it’s nowhere near the handle-time and backlog numbers from the value conversation with leadership. Say so plainly. The small number is the design: a small verified win buys the trust, and the patience, for the versions that move the big metrics. A demo impresses for a week. A chore that stays gone convinces forever.

There’s a principled version of “work nobody wants,” and it comes from Google’s site reliability engineering book, in a chapter by Vivek Rau. He calls it toil: work that’s “manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows.” Read that list back against the wrap-up notes and every box ticks. The scales-linearly attribute even loops back to the volume question in the gate: toil grows with the business, which is exactly why automating it compounds. Every business team has a toil pile, and they can name it in five seconds when you ask, which is why the “happily never again” question sits in the very first interview.

From there, the agent climbs a ladder, and the ladder has a rule.

• Rung one drafts the documentation after decisions are made.

• Rung two pre-assembles the case file before the human opens it: order history, past tickets, the relevant policy section.

• Rung three suggests a response the human edits and sends.

• Rung four handles one narrow, routine category end to end, with humans reviewing a sample.

The rule that makes this change management instead of scope creep: each rung happens when the team asks for it, not when the roadmap says so. Three weeks after launch, Marco asks why the agent can’t also pull the order history before he opens a case. That’s rung two, requested from the floor. And the ladder isn’t a trick I invented: it walks the classic automation-staging model that Parasuraman, Sheridan and Wickens formalized in 2000, from information gathering, to suggested decisions, to supervised action.

Two failure modes need naming, because the happy path above is only one data point. First, the stall: if nobody asks for rung two by month two, that’s data, not patience. It almost always means version one missed the actual most-hated chore. Rerun the “happily never again” question and re-aim; the miss costs weeks, not quarters. Second, the squeeze: leadership pushing the mandate faster than the floor pulls. The version-one number is what buys you the time. Report the small win monthly, and name the next rung’s trigger out loud: “we expand when the team asks, or when note quality holds for four weeks.” A named trigger turns “when will it do more?” into a measurable answer.

One caveat at the top rung. Lisanne Bainbridge’s “Ironies of Automation” (1983) showed that automating the routine work erodes exactly the skills humans need for the hard cases that remain. The fix belongs in the spec, not in a values statement: humans keep a rotation of routine cases, not just a review sample. The team stays sharp, and the agent stays supervised by people who still know the craft.

A few smaller moves compound alongside the ladder. Let the team name the agent; named things belong to the namer. Keep someone from the team in the weekly transcript review. Publish what the agent can’t do, the guardrail list, as prominently as what it can. And measure the team’s time back, never individual productivity: the first dashboard that ranks humans against the agent ends the whole experiment. When someone above you asks how you’re rolling out agents, here’s the paragraph to hand them:

Before we build any agent, we run three interviews with the people who do the job. The spec falls out of the answers: what they’d tell a new hire becomes the instructions, what they look at becomes the tools, what they keep re-explaining becomes the memory, what they’d never let a new hire do alone becomes the guardrails. We prototype it ourselves and test it with the team before the AI team commits a sprint. And version one starts on the work they’d pay to stop doing. The agent earns the interesting work later.

One team, one number, one chore removed. That’s how agent programs actually start.

Run your first interview this week

The mandate said “agentify support.” What you delivered is narrower than the slide-deck version, and better: one team’s least-loved work removed, a prototype-tested spec the AI team built from directly, a small number that verifiably moved, and a support team asking what the agent can take on next. Delivering an agent to the business doesn’t mean delivering a platform. It means delivering a working relationship, between a team and a system it helped design.

The method travels as-is to any team whose work passes the gate. Finance, running invoice exceptions: version one drafts the dispute-resolution notes. Sales ops, answering RFPs (the long questionnaires big customers send before buying): version one drafts the compliance boilerplate everyone hates. HR, fielding onboarding questions: version one drafts FAQ answers from already-resolved tickets. Legal intake, triaging contract requests: version one pre-fills the intake summary. Marketing, reporting: version one drafts the weekly performance commentary. In every case the pattern holds: the interviews find the toil, the toil defines version one, and the team pulls the agent up from there.

So here’s the move for this week. Pick one team. Ask their lead for one hour and ten recent cases. Open with ”what work would your team happily never do again?” and write down everything that comes after. You’ll leave the room holding most of a spec, and the team will walk you to the door.

Your next agent is three conversations away.

Before you spin up the first one, let me hand you a book. It’s called The Goal, a novel published in 1984 by Eliyahu Goldratt. I first read it in mechanical engineering school. I went back to it a few weeks ago, and what struck me is how completely it still applies, except the factory I kept picturing was one running on AI agents. Yes, the book itself is about a real factory: steel, machines, a loading dock, not a line of code in sight. Trust me anyway, because it is the most useful thing you’ll read about AI agents all month.

The hero is Alex Rogo, a plant manager with three months before headquarters shuts his plant for good. Alex’s one bright spot is a set of new robots that pushed productivity in one department up 36%. Everyone’s proud of them. Then Alex runs into Jonah, an old physics professor of his who now consults for factories, and Jonah asks three simple questions. Did the robots increase your sales? Did they reduce your inventory? Did they lower your operating expense? The answer to all three is no. The robots made one station faster and the business no healthier. The plant is still three months from closing.

Here’s the part that makes me smile. Goldratt was writing about big, clunky factory robots, the most exciting technology of his day. The technology has changed completely since then. The move people make with it hasn’t changed at all. A powerful new tool shows up, everyone rushes to point it at every task at once, and all that activity gets mistaken for real progress. The tool gets an upgrade every few years. The trap ships unchanged.

What actually held up is harder and more useful. The Goal forces one question on Alex over and over: what is a company even for? Not to run machines. Not to look modern. Not to automate for the sake of it. The book’s answer is blunt, and it’s still exactly right: a company exists to make money, and every robot, every process, and now every agent is worthless unless it moves that number. Swap “robot” for “agent,” and the book turns into a warning for anyone building AI agents inside a company: the factory urge to make more and make them faster is exactly the trap, and the only thing that counts is whether each one moves the business.

Agents are genuinely powerful, and building them is my day job. I find the use cases, work with developers to ship them for the company I work for, and build smaller ones for my own work on the side. But here’s the subtler truth the rush skips over: a powerful tool tends to create new problems while it solves old ones. An agent aimed at the wrong step doesn’t just fail to help. It quietly manufactures more work, a backlog of half-finished output that someone downstream now has to deal with. So the whole game is where you aim them, and that turns out to be the part everyone skips.

So here’s the bottom line, up front. A factory of agents is not the goal. The goal is throughput: the actual work that ships and moves the business. Pour agents onto a step that was never your slow point, and all you do is stack work in front of the one that actually is slow, which today is almost always a person reviewing and verifying what the agents produced. That slow step is your constraint. The winning move is the opposite of agents everywhere: find your one constraint, point your next agent at it, and pace everything else to its speed.

Here’s where this article goes, in four stops.

• The trap. I’ll show you why building agents by the dozen stalls, and the factory novel that saw it coming.

• The bottleneck. You’ll find the one step everything waits on, which is rarely where you’d guess.

• The principles. A few ideas from the book for feeding that step instead of burying it under more agents.

• The build. What aiming your next agent at the right step actually looks like, hands on keyboard.

By the end, you’ll have a simple way to look at any agent plan and say where the next agent actually belongs, plus the words to explain it to the people around you. It’s the difference between an agent strategy that looks busy and one that moves a number.

Hard hats on.

Why agents everywhere stalls

When the scoreboard is “agents deployed” or “workflows automated,” you optimize for the count, and the count is the easy thing to grow. Six new agents this quarter, twelve workflows touched, a demo that makes the room lean in. It feels like progress. But Goldratt put the whole trap in one line: ”activating a resource and utilizing a resource are not synonymous.” In plain English, keeping something busy is not the same as using it well. A factory machine stamping out parts nobody ordered is busy, but useless. An agent churning out drafts faster than anyone can check them is busy in exactly the same way. None of that activity is the business.

And the bill is already coming due. Gartner expects more than 40% of agentic AI projects to be canceled by the end of 2027, citing runaway cost and unclear business value. That is the same verdict Jonah, the book’s mentor, gave those factory robots, only now it lands on AI agents instead of machines.

So keep his three questions on a sticky note. They were written to judge a factory, and they barely change to judge an agent rollout:

• Did you ship more of what the business actually needs?

• Did the pile of half-finished, waiting-on-a-human work get smaller?

• Did your cost to run the place, the model bills and the hours spent steering, go down?

If you can’t say yes to at least one without making another worse, you didn’t buy an agent strategy. You bought robots.

Those questions come from the only three numbers Goldratt says a company should track. He names them ”throughput, inventory and operational expense,” and once they click you start seeing them everywhere.

• Throughput. The rate the system turns work into money out the door. For your team, it’s the work that actually ships and changes something for a customer, not the work that merely got generated.

• Inventory. Everything you’ve paid for that’s sitting around not yet sold. For you, it’s every draft, pull request, and ticket an agent produced that’s still waiting on a human to finish it.

• Operating expense. What you spend turning inventory into throughput, the cost of running the whole thing. For you, the model bills, the tools, and the hours your people spend steering and reviewing.

The robots moved none of the three. They just made a station that was never the slow part run faster, so half-finished parts piled up behind the slow station instead.

Here’s the deeper problem underneath, the one it takes Alex most of the book to see.

A factory isn’t a set of independent machines you keep individually busy. It’s a chain of steps that depend on each other, and what the whole chain produces is set by its slowest link, not by how hard the other links work. Alex’s plant was run the other way, every station scored on its own efficiency, every machine pushed to stay busy because an idle machine looked like waste. So the fast stations overproduced, and their output piled up as inventory in front of the slow ones. Every local chart looked healthy while the plant as a whole lost money. The robots didn’t cause that pattern. They poured fuel on it, supercharging a station that was never the constraint.

Sit with that, because your company is a factory too, even if nobody calls it one. Work comes in as a request and leaves as something shipped, and in between it moves down a line: someone drafts, someone reviews, someone approves, someone ships. Agents are the shiny new machines you just bolted onto the front of that line, and they make the drafting station absolutely roar. A line is only ever as fast as the station every job has to squeeze through, no matter how loud the front end gets. Build it the plant’s old way, with each agent scored on how busy it is, and you earn the plant’s old result.

That pile is the part most teams never name, and it’s worse than the factory kind. The robot’s parts were all good. This pile isn’t, because some of the output is wrong in confident, plausible ways, so the human downstream is inspecting for quality, not just clearing a backlog. You paid to create all of it, and none of it counts until someone finishes it and ships it. Which means the real question isn’t “how many agents.” It’s “where’s the slow station,” and the book finds the answer on a hiking trail.

Find your real bottleneck

Goldratt teaches the idea away from the factory, and it’s the scene that stuck with me for years. Halfway through the book, Alex, the plant manager, gets roped into leading a weekend hike for his son Dave’s scout troop, and he can’t keep the line together. The fast kids race ahead, a long gap opens in the middle, and the whole troop falls behind no matter how he reshuffles it. Watching that gap stretch and snap shut, he has the realization the rest of the book turns on. The troop is a chain, and a chain doesn’t travel at the speed of its quickest kid. It travels at the speed of its slowest. That kid is Herbie, a heavy boy lugging an overloaded pack at the very back. No amount of hustle from the kids up front buys back a minute, because sooner or later they all have to wait for Herbie.

So Alex does two things that feel backwards. He moves Herbie to the front, where nobody can be stuck behind someone slower. Then he opens Herbie’s pack and parcels the heavy gear out to the fast kids who have room to carry it. Herbie speeds up, and because Herbie now sets the pace, the whole troop speeds up with him. Pushing the fast kids was never the answer. Helping the slow one was.

That’s the whole theory of constraints in one walk. Every system has one step that sets the pace for all the others, and improving any step that isn’t that one is a mirage. Goldratt says it flatly: ”An hour lost at a bottleneck is an hour lost for the entire system. An hour saved at a non-bottleneck is a mirage.” Speed up the fast kids and the line just spreads out further. The only move that matters is finding Herbie and taking weight off Herbie.

So how do you find your Herbie? The book’s method is refreshingly low-tech: follow the piles. Alex doesn’t spot his slow machines in a report. He finds them by walking the factory floor and seeing the mountains of half-finished parts stacked up in front of them. A bottleneck always looks the same. There is a pile of work waiting in front of it, because the steps before it keep feeding it faster than it can keep up. And there are idle, starved steps behind it, because they have run out of work to do while everything is stuck upstream. So don’t theorize about where your slow step is. Go find where the work stacks up, and where people are stuck waiting on someone else.

Run that test on an AI-powered workflow today and the pile shows up in a predictable place. Generation got cheap and fast. A model writes the first draft, the first version of the code, the first pass of almost anything, and it costs almost nothing. When the cheap step gets ten times cheaper, it stops being the constraint, and the constraint moves to whatever it feeds. What it feeds is judgment. Someone has to read the output, decide whether it’s right, catch the confident mistake, and put their name on it. Review, verification, and the taste to know what’s worth shipping: that’s the heavy pack now.

The receipts back this up. When METR, an AI research lab, ran a randomized trial on experienced developers in 2025, the developers believed AI tools had made them 20% faster, and the measurements showed they’d actually gone 19% slower. Feeling productive and being productive came apart. Google Cloud’s 2025 DORA report found the matching pattern at the team level: AI lifts how much you produce, but it lowers delivery stability, because the acceleration exposes the weak step downstream instead of removing it. The time you save generating gets spent right back auditing, because nobody can review the volume the agents produce.

And this isn’t only an engineering story. Point agents at a marketing team and the one brand reviewer who keeps fifty draft variants on-voice becomes the slow step. Point them at support and the person who owns tone and policy is the gate a hundred suggested replies wait behind. Wherever a human still has to verify the work and own the result, that judgment is the new bottleneck, no matter how cheap the drafting got.

Now watch what the typical agent factory does: it pours more agents into generation, the step that was already fast. Every agent you add upstream of the constraint just hands it a taller pile. That’s not a throughput machine. It’s an inventory machine, and a very impressive one. Engineering teams already hit this exact wall with AI coding tools. Output jumped at first, then flattened out, because the extra code just piled up waiting for a human to review it.

Finding the bottleneck is the easy half. What you do once you’ve found it is where the throughput actually comes from, and the book gave me a handful of principles for that.

Work your real constraint

A quick word on what follows. These are principles from the book, not a rigid checklist to run top to bottom. The order helps, but take the ones that fit your situation and try them.

Map your flow first

The honest starting point isn’t “find the constraint.” You can’t find a thing you can’t see. The first move is to draw your actual flow, the path one real piece of work takes from idea to shipped.

Put one box per step in a row: request, spec, build, review, approve, ship. Then do the part everyone forgets. Between each pair of boxes, write down how long work usually sits there waiting to be picked up. In knowledge work, a task spends most of its life parked in a queue, not being worked on, which is exactly why value-stream mapping puts wait time on the board right next to work time. It’s also what Alex does when he walks his plant floor instead of reading reports: he follows the piles. Do it on a wall, in half an hour, with the people who actually do the work. They already know where it jams; the map just makes it impossible to deny.

Find your one real constraint

With the map and its wait times in front of you, the biggest pile of waiting work usually points straight at your bottleneck.

Right now, for most teams using AI, that slow step is review. Generation got cheap, so work piles up wherever a human still has to read it, judge it, and sign off. But usual is not the same as always, and reaching for the fashionable answer is its own trap. Look at your own map and check. For one team the slow step is writing the spec, because nothing gets built until the requirements are nailed down. For another it’s pulling the data, because one tricky query sits between every question and a real answer. For another it’s the security sign-off, or the final deploy, or any step that hangs on a rare approval.

Notice that none of those is a person. It’s a step in the process. The reviewer is not your bottleneck; the review step is. The difference matters, because you fix a step, you don’t fix a human. Find the step where work piles up, name it out loud, and write it down before you spend a dollar.

Get more out of it before you spend

Before you spend a dollar widening the slow step, get more out of the capacity it already has. This is the cheapest win there is, and the book makes it concrete.

In Alex’s plant, the bottleneck machines sat idle during lunch, because the operators all broke at the same time. So they staggered the breaks and the machines ran straight through: same machines, more output, zero dollars spent. They made two more moves like it. They moved quality inspection to before the bottleneck, so it never wasted a minute on a part that was going to be scrapped later. And they stopped feeding it parts for orders that weren’t even due, so every hour went to work that actually mattered.

Run those same three moves on a review step that has become your bottleneck:

• Never let it sit idle. Keep a small batch of work queued and ready in front of it, so the review step never stalls waiting for something to land.

• Never feed it junk. Put an AI agent in front of review to do a first pass. It waves through the obviously fine and flags the obviously broken, so the only things that reach a human are the genuine judgment calls.

• Never feed it work that doesn’t matter. Stop sending output to review for a feature that just got cut or a draft nobody asked for. Every review hour should go to something that’s actually shipping.

None of that costs money or a new hire. You’re not making the slow step bigger yet. You’re making sure not a minute of it is wasted.

Pace the rest of the work to it

This is the one almost everyone gets wrong, because it feels backwards. Once you know your slow step, you deliberately slow everything else down to match its pace.

The book calls the mechanism the drum, the buffer, and the rope. The slow step is the drum: it sets the beat for the whole line. A small buffer of work sits right in front of it so it never starves. And a “rope” holds back the start of the line, releasing new work only as fast as the slow step can actually take it. In the plant, Alex stops dumping raw material onto the floor just to keep people busy, and the mountains of half-finished inventory melt away.

For an AI workflow it comes down to one rule: don’t let your agents generate faster than your slow step can absorb. In practice you cap how much unreviewed work is allowed to pile up. Say the review step can clear about thirty pull requests a day. You set the limit at roughly a day’s worth in the queue. When the queue hits that cap, generation pauses, and the people who would have started new work go help clear the backlog instead.

If that sounds like a Kanban board with work-in-progress limits, that is exactly where the idea came from. The rule of thumb is blunt, and it works: stop starting, and start finishing.

Then widen it, and aim your next agent there

Only now, after you’ve squeezed the free gains out of the slow step, do you actually spend on it. And the spending goes to the slow step, never to the part that’s already fast.

When Alex’s team ran out of no-cost tricks, they got their hands dirty. They hauled two old machines out of storage to take some load off the bottleneck, and they sent the overflow to an outside vendor. Not elegant, but it widened the exact step that was holding the whole plant back.

Your version is the same idea, aimed at your slow step. If review is the bottleneck, the most valuable thing you can build is not another generator. It is something that widens review itself:

• A first-pass review agent that handles the easy 80% of changes, so a human only has to look at the hard 20%.

• An eval (a quick automated check against examples you trust) that clears the routine cases, so a human only sees the ones it flags.

If your slow step is writing the spec instead, you build the agent that drafts the first version of the spec from the request, so the work starts at “edit this” rather than “write this from a blank page.” The agent worth building is the one that widens your slowest step. Every other agent just makes a bigger pile.

Expect the constraint to move

Here’s the part that catches people off guard. The moment you widen one slow step, the bottleneck doesn’t vanish. It jumps somewhere else. Fix review, and the slow step becomes deployment, or one overloaded approval, or a sign-off that was invisible only because review was hiding it.

In the book, once Alex fixes his machines, the constraint leaves the factory floor entirely. It moves first to the market, and then, to everyone’s surprise, to the company’s own outdated rules, policies that made sense years before and were now quietly throttling everything. Goldratt’s word for that trap is inertia: pouring effort into yesterday’s bottleneck long after it stopped being the problem. Half the time the real constraint isn’t a machine or even a step. It’s a rule nobody has revisited, like “every change needs three approvals.”

So this is never a one-time project. Put a recurring half-hour on the calendar to re-walk your flow map and ask one question: where does the work pile up now? The answer will have moved, and so should your next fix and your next agent.

That is what replaces “twenty workflows, twenty agents.” Not a sprawl you launch all at once, but one constraint found, fed, and widened, then the next one. Fewer agents, pointed at better places, and a number that finally moves.

Build the one that matters

So what does building that agent actually look like? Smaller than you’d think. Say your slow step is review. You don’t need a platform or a six-month program. In Claude Code, the smallest version is a subagent: a short markdown file (just a plain text file) in your project that gives an agent a name, a set of tools that can only read and never change anything, and a system prompt (a written instruction) spelling out exactly what to check on a first pass. It reads every change, flags the handful that need a person, and waves the rest through. Your reviewer stops drowning in trivial diffs and spends their scarce attention on the calls that actually need judgment. The same shape works far from code: a subagent that drafts the first version of a spec, or one that writes the first cut of the gnarly query for your analyst to correct.

The part people skip is what the agent checks against. The teams getting real mileage out of agents right now aren’t the ones with the best model, they’re the ones with the best evals. An eval is just a test with an answer key. For a first-pass review agent, the answer key is a handful of past changes where you already know which ones had real problems. You run the agent over them and see where it disagrees with you. Does it catch the genuinely broken ones and wave the clean ones through? You tighten its instructions until its calls line up with yours, and only then do you trust it on live work. Anthropic’s own write-up on evals is a hype-free place to start. An agent at your constraint with no eval behind it is just a faster way to be confidently wrong.

You don’t have to do any of this alone, and you shouldn’t. The people doing the work already know where it jams. Put the flow map on a wall and let your engineers point at the step that always stalls; ask the analyst what they wait on; ask the designer what sits in their queue. The framing that tends to land, whether you’re explaining it to the team or the people who hold the budget, is short: we don’t have an agent shortage, we have a constraint we haven’t named. Once everyone is looking at the same slow step, the argument about how many agents to build mostly answers itself.

One last thing, and it’s the part I find genuinely fun. The specific tools in this article will age fast. A year from now, generation will be cheaper, the agents sharper, and your slow step sitting somewhere new. None of that changes the move. Draw the flow, find the one step everything waits on, feed it, and aim your next build straight at it. That is why a novel about a steel plant still reads like a manual for building AI agents. The machines changed. The mistake didn’t, and neither did the fix.

The team learned a new tool. They got a small productivity bump. The next jump never came. And now, sitting in the all-hands, you can feel the air going out of the room. You sold this to your CTO as a step-change in time to market. It’s looking more like an autocomplete upgrade.

I want you to spend 35 minutes on a video before your team decides this is as good as it gets. It’s a talk from Szczepan Faber and Mike Nakhimovich, two engineers on Airbnb’s developer platform team. They explain how their org went from zero to 64% of PRs materialized through agentic coding in nine months. No marketing slides, no vendor pitch, two practitioners with diagrams and the receipts.

I know. Another AI talk. But this one is different. Most agentic engineering announcements recently read like product trailers. Stripe wrote up their Minions, one-shot end-to-end coding agents that ship code from issue to PR. Spotify wrote up their background coding agent you can fire off from Slack. Both are real engineering teams shipping real systems. But the writing is announcement-shaped, and you walk away knowing what they built, not how to bring your own team along.

This one is two practitioners walking through their methodology, their mistakes, the proficiency model they built for engineers, and the org structure that scaled it. With actual numbers. With actual diagrams. Practitioner depth, not announcement depth.

A disclaimer before you watch. The talk is from October 2025, which is roughly prehistory in AI time. A few things they mention have already evolved. Their internal stack is theirs, not yours. You shouldn’t copy what they did. Since then, Airbnb’s CEO has publicly stated AI writes nearly 60% of new code at Airbnb (TechCrunch coverage of the May 2026 earnings call). The platform team’s October talk is still the operating manual. The May number is the receipt. The definitions, principles, and learnings travel.

This post is the distillation. A working definition of agentic coding so you can say it cleanly to your team. The principles their team operated by. The mistakes they fixed. And a small grid you can use to plot your tech function and pick the next project to push for. Watch the video first if you have the time. If you don’t, this post stands on its own.

• The plateau. I’ll show you why AI coding tools alone gave your team a small bump and stalled.

• The definition. I’ll give you a clean working definition of agentic coding to quote to your team.

• The principles. I’ll walk you through the org moves Airbnb’s team built around the tools.

• The learnings. I’ll point out the post-mortems you’d repeat without them.

• The grid. I’ll hand you a four-cell grid to plot your team and pick the next move.

By the end, you’ll have the working definition, the principles, the learnings, and a four-cell grid plotted with your tech team’s location and a next-move sentence. You’ll also walk away with a 7-category readiness checklist you can paste into a Google Doc and fill in row by row. You walk away with the conversation to have with your CTO on Monday, and a way to reframe AI tool ROI as agentic coding maturity.

Read the checklist

Let’s pin it down.

Name the plateau

Every team is hitting the same wall right now. The AI coding tool went in, the productivity bump was real, and then nothing. The team starts wondering out loud whether they bought the future or just a faster autocomplete. Naming the wall is the first move, because the wall has a shape and the shape tells you what comes next.

Airbnb’s platform team measured it. They ran the velocity numbers across three engineer cohorts. Non-AI users moved at the team’s baseline pace. AI-aided non-agentic users (the autocomplete-and-chat crowd) shipped PRs at +27% velocity. Full agentic users shipped at +38%. The eleven-point gap between tools-only and agentic is the ceiling teams hit when they stop at the tool. That gap is not a tool failure. It’s the natural plateau most teams sit on right now.

One honest caveat before we keep going. The wider data is messier than Airbnb’s three bars suggest. Google Cloud’s 2025 DORA report found individual output rises while team delivery stays flat. METR’s July 2025 RCT found experienced open-source devs were 19% slower with AI tools, while they believed they were 20% faster. Airbnb’s three bars are one of the cleanest team-level signals we have, not the universal shape. Treat them as the strongest existing public benchmark, not the law of nature.

Three things sit inside that eleven-point gap.

• The first is practice maturity: how engineers actually use the tool day to day, whether they auto-approve, whether they review every diff, whether they push back when the agent is confidently wrong.

• The second is org infrastructure: the paved paths, the internal champions, the sandboxed environments, the MCP servers that wire internal context into the loop.

• The third is leadership conviction: whether the org frames agentic coding as a step-change worth funding or a productivity tweak worth tolerating.

Each one is a stop on a journey your team has been walking since the IDE era. None of them gets fixed by buying a better tool.

Your team is at Stage 2. The plateau is not a sign you picked the wrong tool. It’s a sign the next investment is in the things around the tool. That reframe is the one you carry into the leadership conversation. You bought the engine. Now you’re missing the chassis.

Naming the gap is the first move. Naming what agentic coding actually is comes next. That’s the language you’ll use upstream.

Define what agentic coding is

Part of why your team is stuck in the plateau is that they’re using one word for two different things. Autocomplete is not agentic coding. Copy-paste from ChatGPT is not agentic coding. A single-shot retrieval over your codebase is not agentic coding. None of those is the thing Airbnb measured at +38%. Clean the definition first, then have the budget conversation honestly.

An engineer writes a short spec or prompt. That spec kicks off an autonomous loop. Inside the loop, an LLM is called several times. Tools and MCPs are called several times. The loop loads config files (AGENTS.md, CLAUDE.md, cursor rules) and runs against guardrails. Out the other side comes a materialized code change. A human reviews every line before it merges. The thing that makes it agentic is the autonomy in the middle, not the input or the output.

The engineer doesn’t know exactly what’s happening in there. Szczepan put it bluntly:

An engineer who is leveraging an agentic tool, you don’t know how many [calls happen]. You start your agentic session, but you don’t know exactly what prompts the agentic tool will use when it calls LLMs. You don’t know exactly how many times it’s going to call LLMs. You don’t know exactly what tools it’s going to use.

That uncertainty is the point. The loop owns the path. You own the spec and the review.

Three things people call agentic coding that aren’t. Autocomplete in your IDE is not agentic coding (one-shot, no tool calls). Pasting code into a chat window and copying the answer back is not agentic coding (no loop, no autonomy). A single-shot RAG search returning a snippet is not agentic coding (no decision-making, no tool composition). Szczepan made the same point.

That’s not where the magic happens. This is not this paradigm shifting change where a human engineer can steer multiple agentic sessions and materialize code changes.

The +27% in the velocity chart is what you get from those. The +38% is what you get from the loop.

The cleanest way to explain the shape to leadership is borrowed from pair programming. You probably have engineers who’ve done it. In classic pair programming, the driver holds the keyboard and is tactical. The navigator looks at the whole forest and is strategic. They alternate roles through the day. Agentic coding flips that into something stable.

The engineer becomes the permanent navigator. The agent becomes the permanent driver. The engineer steers the session with chat, click, voice, or image. The agent materializes the code. Szczepan said it directly:

Agentic coding is where a developer becomes this permanent navigator and it’s steering that agentic session, and an agent or your tool is materializing code changes.

The leadership analogy travels because every CTO over the age of 35 has done pair programming.

The technical version of the analogy is sharper. What the navigator is actually doing has a name. Andrej Karpathy coined “context engineering” in June 2025 as the successor to prompt engineering. He called it ”the delicate art and science of filling the context window with just the right information for the next step.” That’s the navigator’s job. Pair programming is the analogy that lands. Context engineering is the technical second skin.

Now that you can name it, here’s what Airbnb’s team learned about making it actually pay off.

Borrow these principles

The tool was the easy part. The principles are the actual work. None of these cost engineering headcount. All of them are decisions you can push for from where you sit. Here are five that travel from Airbnb’s context to almost any other org.

Start with a bold vision, work backwards

Airbnb’s platform team opened their January 2025 leadership pitch with a single sentence:

AI-powered, agentic coding will eventually replace traditional programming methods. How do we work backwards from that end state?

Then they actually worked backwards. They didn’t ask “what experiment do we run next quarter?” They asked “what does Step 10 look like?” and walked backwards to Step 1. Szczepan was direct about why.

If you only think about incremental improvements, it’s kind of hard to make a big change.

The move is the same for you. Put the destination on a slide before you put the roadmap on a slide.

Use an analogy with leadership

The pair-programming analogy didn’t land at Airbnb because it was clever. It landed because leadership had done pair programming.

The way we’ve explained that to our leadership at start of 2025, we’ve used pair programming analogy for this.

Szczepan said. Pick an analogy your leadership already trusts. Map agentic coding into it. Leadership conversations don’t move on facts. They move on framings your audience already believes. If they came from a team that used trunk-based development, talk in those terms. If they came from agile, talk in those terms.

Measure holistically, never personally

The default trap is to pick one number. Velocity goes up. Ship it. Sentiment goes down. Drop it. Airbnb uses three lenses together, never individually: developer sentiment from quarterly surveys, tool adoption across surfaces and PRs, and engineering output like PR velocity aggregated across the team.

The philosophy slide from their talk is forwardable on its own.

Our aim is to learn how AI impacts developer experience and software quality so we can improve both. Not to micromanage or rank individual performance. Focus on insight, not oversight.

This is the principle to use when leadership asks for ROI metrics that don’t exist yet. The answer is not one metric. The answer is three, read together, looked at the team level. If you want a public framework they’ve heard of, Airbnb’s three lenses overlap with what DX calls the Core 4 (Noda and Tacho, January 2025): Speed, Effectiveness, Quality, Impact. Same principle: never one number, never per-individual.

Meet developers where they are

Sixty percent of Airbnb’s engineers prefer remote workspaces. Forty percent prefer local. IDE lovers stayed in IntelliJ. CLI lovers stayed in Vim. Mike tried the migration play once.

We considered, maybe for a few days, grabbing all of our IntelliJ users and moving them over to VS Code. That lasted for a few days before I had to fear for my life that someone would come after me.

He said. The principle: build parity across surfaces, not migrations. Or as Mike put it elsewhere:

We don’t have one typical engineer at Airbnb.

You probably don’t either.

Invest in the org around the tool

This is the biggest gap most teams skip. The tool ships. The infra doesn’t. Nobody builds a champions network, no one paves the install path, no one wires internal context into the agent. Airbnb invested in all three. A champions network of thirty cross-functional engineers (not the core team of four). Paved paths with single-command install, automagic auth, and project templates. Over a dozen internal MCP servers connecting agents to internal systems. Szczepan said the line out loud:

Your agentic tool is not enough. You will have to have a lot of tech around it.

If you want a second public example to back this up, Shopify built the same pattern at roughly 10x the scale. CEO Tobi Lütke’s leaked AI memo became an internal movement. The eng org built an internal LLM proxy, standardized MCP servers wired to their wiki and PM tool and data warehouse, and bi-annual performance reviews that explicitly evaluate “AI-reflexivity.” Two-anchor evidence is sturdier than one-anchor evidence. (First Round wrote it up; Bessemer wrote up the engineering playbook.) This is the line item to fight for in your next budget cycle.

Add three engineering principles

Three more principles, briefly. First, wrap open source instead of building your own orchestrator. The rule is “don’t build a full orchestrator from scratch,” not “wrap whatever’s hosted externally.” Even Airbnb wraps their own airchat-cli, not raw OpenAI or Anthropic. In regulated orgs (finance, healthcare, EU-AI-Act-bound), routing internal code through a third-party CLI is a procurement and legal fight, not a one-line decision. Build the thin wrapper layer locally if compliance demands it.

Second, bring context to agents through tool calling, not by training models. ”Tool calling is all you need,” Mike said. He’s riffing on the 2017 Transformer paper title. His point: the LLM doesn’t need bespoke training on your codebase. It just needs the right tools available. Whether tool calling alone is enough versus the richer MCP pattern is contested in 2026. Anthropic’s November 2025 “Code execution with MCP” pushes more sophisticated patterns. Mike’s line still holds as the directional claim.

Third, never drop human review on the way to the PR. Szczepan’s line is the one to forward to anyone tempted to “ship faster”:

I don’t trust the AI. I trust my skill in prompting and using the tools.

The agentic loop ends at the diff. Every line is reviewed by a human before merge, especially the test code.

Worth knowing what to avoid too. Here’s what Airbnb’s team got wrong before getting it right.

Avoid these mistakes

Even Airbnb’s platform team got things wrong before they got them right. They have four engineers, a dedicated mandate from leadership, and the resources of a 7,000-engineer parent org. They still hit these. Worth borrowing the post-mortems before you make them yourself.

Don’t bet on a slow curve

Airbnb’s platform team predicted at the start of 2025 that materializing code through agentic sessions would hit 20% to 40% of PRs by the end of the year. By the time they gave their talk in October 2025, they were at 64%.

We were sandbagging them a bit too much, I think, because we were not ambitious enough. It’s going faster.

Szczepan said. If the people building the platform underestimated their own curve by a factor of two, the “wait and see” position upstream is probably already late.

There’s a name for that position. Geoffrey Moore called it “early majority pragmatism” in Crossing the Chasm (1991). Pragmatists wait for proof. Visionaries (early adopters) ship before the proof exists. Recent commentary suggests consumer GenAI has crossed the chasm; agentic enterprise systems are mid-crossing. The frame gives you a named, well-known concept to drop in the 1:1: ”You’re in classic early majority position. The data suggests the chasm has already been crossed.”

Don’t build your own orchestrator, wrap one

Mike’s team spent months building an internal agentic orchestrator. Registered agents. Planning chained to coding chained to validation. It was elegant. It never shipped.

AI moves way too fast. We were still planning, figuring out what we were going to do that existed in the market, and the market now is five steps ahead of us. It wasn’t going to work.

He said. They walked away from the orchestrator and pivoted to a thin shim over airchat-cli.

Can’t beat them, join them. We pivoted and instead delegated to our Airchat CLI.

The rule for any size team: if open source is iterating faster than you can ship, wrap it.

Slow down before you auto-approve

There’s a temptation, once an engineer gets comfortable with agentic coding, to skip straight to auto-approve. Don’t. Three things happen when engineers skip the manual-approval stage too fast. PRs get huge. The engineer doesn’t actually read every line of the diff. Test code drops in quality first, because test code gets reviewed less carefully than production code. Airbnb’s Level 3 (auto-approve plus diff review) is a learned skill, not a default. Szczepan said it cleanly:

An engineer may create very large PRs that they don’t review fully. They don’t actually understand and review every single line of code. They may not review test code as eagerly as the production code.

The fix, when you’re in the loop, is to push back on the tool when it’s confidently wrong. Szczepan walked through it.

You don’t trust it. If something feels suspicious, you ask it: “Why did you do this in line 120?” And after a couple of rounds, you get to this satisfying moment where the agentic tool gives up and says, “Oh yeah, you’re right. I’m sorry. I was wrong. Let’s fix it.”

That’s not friction. That’s the practice.

Close the skill gap fast

This one comes from Mike’s lived experience, not a number on a slide. Mike is a college dropout who admits he doesn’t know Git from the command line. With agents, he ships across multiple languages.

I’m one of those non-believers. This is all going to be slop. This is all going to be vibe coding. This is going to be worse than what I can write by hand. And when I was using GPT 3.5, that was true. I personally got lapped by agentic coding.

He said. In Mike’s team’s setup, the gap between agentic-fluent and non-agentic engineers widened faster than anyone predicted.

Definition, principles, learnings. Now plot where you actually are.

Plot your team on the grid

Reading the principles and learnings is one thing. Doing something with them this quarter is another. The grid below is the diagnostic. It’s not a model of who you are. It’s a map of where to invest next.

The grid has two axes. The vertical axis is median engineer fluency. The horizontal axis is org infrastructure. For the vertical axis, plot the median engineer on your team, not your top performer. Airbnb published a four-level fluency ladder that travels well.

• Level 1 is first steps. Autocomplete, occasional chat, copy-paste from ChatGPT.

• Level 2 is agentic learner. Uses an agentic tool with manual approval; improving prompts and configs.

• Level 3 is advanced practitioner. Auto-approves, reviews diffs, runs parallel sessions across multiple workspaces.

• Level 4 is pioneer. Builds custom MCPs, leads team transformation.

One caveat that matters: Level 4 (pioneer) is an organizational role, not just a personal skill. It requires platform-team mandate and org permission, not only individual fluency. Pioneers are usually the output of being in Multiplier mode (the top-right cell of the grid), not a prerequisite.

For the horizontal axis, ask plainly: is there a champions network, paved paths, internal MCP servers, sandboxed environments? Or is everyone on their own? You’ll know within five minutes.

Each cell has a name, a description, and a next-move sentence for you.

• Tourist phase (low fluency, low infrastructure). Everyone copy-pastes from ChatGPT. No champions, no shared standards, no one paving anything.

  • Next move: push for one champion in your tech team and one paved tool. Don’t try to do five things at once. Sequencing matters more here than ambition.

• Hero phase (high fluency, low infrastructure). One or two senior engineers shipping miracles with agentic coding. The rest of the team hasn’t moved. The seniors burn out, leave, or get poached.

  • Next move: make the cost-of-burnout case to leadership. The hero’s productivity doesn’t scale without the infra.

• Onboarded but underused (low fluency, high infrastructure). The tools exist, the infra is paved, but engagement is flat. Usually a culture gap, not a skills gap.

  • Next move: push for three Shopify-style moves. Visible leader use (a senior engineering leader ships a PR with an agent during the next all-hands). A paved-path simplification audit (every extra click between an engineer and an agent kills adoption). And AI-fluency woven into performance reviews, not as a productivity metric but as an expectation. Not more tools.

• Multiplier mode (high fluency, high infrastructure). This is where Airbnb’s 64% PR adoption lives. Power users running five parallel sessions, MCP ecosystem growing organically, the working group of champions teaching everyone else.

  • Next move: defend the budget, bring product into agentic workflows (PMs prototyping with the same loop), start measuring time-to-market shifts.

The 2x2 is the headline. The roadmap lives in a row-by-row diagnostic with seven categories: direction and adoption posture, measurement, tooling stack, internal context, engineer skill journey, org structure, and quality bar. Each category has two or three rows. Each row asks one question. ”What does your team actually have here?” Fill in the empties. Count them. Pick one row to push for this quarter. Not five.