For years, though, the habit had a flaw I couldn’t see. The reviewer was me. My memory of the week is a friendly narrator: it remembers the demo that landed, and it quietly forgets the question I dodged in a tense meeting. Every Friday I graded my own homework, and every Friday I passed. A retro written from memory is a review of the week you’d like to have had. Better than nothing, but it plateaus fast.

Then one afternoon I pasted the verbatim transcript of a user interview I’d just run into Claude. Not to summarize it (I do that all the time). I asked it to review my interviewing: how I phrased questions, what I missed, how much I talked. It found three leading questions in my first ten minutes and quoted them back word for word. Nobody had reviewed my work at that level of detail in years.

Everyone I know uses AI to do the work: draft the spec, summarize the research, write the update. Almost nobody uses it to review how they work. The first use saves you hours this week. The second one compounds, because it makes every next interview, meeting, and decision slightly better than the last. That’s the 1% from the training, and AI finally makes it cheap to collect.

The whole practice fits in one picture.

• The gap. I’ll show you why nobody reviews your actual craft anymore, and why the feedback you do get says more about the giver than about you.

• The loops. The three I run (interviews, meetings, the Friday retro) with the exact prompts I use, plus six more worth trying, from your forecasts to your 360 packet.

• The honesty problem. Here’s how to stop the model from flattering you, because by default it will.

• The team angle. You’ll build a skill that asks your PMs your own 1:1 questions before they ever reach you.

By Friday you can have run your first loop on a real artifact from your own week. You get the kind of specific, evidence-quoting feedback nobody has given you since you got senior, and you can get it every single week. The first loop needs nothing new: any LLM chat works (Claude, ChatGPT, Gemini), and the raw material is the transcripts your meeting tools already produce (Zoom AI Companion, Gemini in Google Meet, Granola, or a voice memo on your phone). And if you lead PMs, you’ll leave with two skills worth an afternoon each: one that reviews your week on a schedule, one that preps your team for 1:1s.

Your reviewer is ready. Let’s put it to work.

Nobody reviews your work anymore

Think about the last time someone watched you work and told you something true about it. Not your outcomes: your craft. The way you ran the interview, the way you handled pushback in the room. If you lead product teams, the honest answer is probably “years ago.” Your manager sees headlines and results, not how you got them. Your team won’t critique you upward. The more senior you get, the less anyone reviews how you actually work. What’s left is the official cycle: twice a year, compressed into themes like “communicate more.”

And hold even that loosely. In The Feedback Fallacy (Harvard Business Review, 2019), Marcus Buckingham and Ashley Goodall assembled the research on how well humans rate other humans. The headline finding has a name, the idiosyncratic rater effect: more than half of any rating of you reflects the rater, not you, and no amount of training fixes it. In their words, feedback is “more distortion than truth.” Your manager’s read on your “communication” says as much about their definition of communication as about yours, and that stays true for the best managers you’ll ever have. The little feedback that reaches you arrives filtered through someone else’s lens, and the research says the filter is most of the signal. Not a reason to dismiss your manager. A reason not to make two filtered data points a year your only mirror.

Performance-heavy fields treat this as a solved problem. Athletes and musicians review tape after every game and every recital. Atul Gawande wrote the canonical essay on this, Personal Best (2011): a surgeon at the top of his field plateaued after eight years, then started improving again when he put a reviewer back in his operating room. The research on expertise agrees. Anders Ericsson’s work on deliberate practice (Peak, 2016) found that experience without feedback doesn’t accumulate into skill; it plateaus. Twenty years of interviews don’t make you better at interviewing if nobody ever shows you what you did.

What changed isn’t the theory. It’s the tape. Your meeting tools record by default now: Zoom AI Companion, Gemini in Google Meet, Granola quietly taking notes on your laptop. Verbatim records of you working have been piling up in your drive for a couple of years. We skim the summaries for action items and move on. The evidence of how you work is exhaust of a normal week, and almost nobody reads it in the review direction.

One precision before you touch that pile. The AI notes your tools generate are summaries, and a summary is already the friendly narrator’s cut, machine-made this time. The loop needs the verbatim transcript. In Google Meet, that’s the separate “Transcripts” option, not the Gemini notes doc. In Zoom, it’s the transcript file attached to the recording. Granola keeps the full conversation one click behind its polished notes. Feed the model what was actually said, not what the notes decided to remember.

And the reviewer for all that unread tape? You already have it. You’ve just been pointing it the other way.

You already run AI as a production engine. It drafts the spec, summarizes the research, writes the launch note. Point the same model at a transcript of how you worked, and it becomes a feedback engine. Same tool, opposite direction. In that direction it has qualities no human reviewer can offer: the patience to re-read your entire week verbatim, no stake in your ego, and availability at 6 pm on a Friday. The only reviewer with time to watch all your tape has been sitting in your browser tab all along.

The direction flip is the whole method. Here’s what it looks like on a real week.

Start with my three loops

I run three loops on my own weeks, and they share one shape. Take a verbatim artifact your week already produced. Ask the model to review how you worked, not what was decided. Extract one change for next time. Write that change down. The artifact does the honesty, the model does the patience, and the written line makes it stick. Here’s the shape before the details.

Three isn’t a magic number, and my three aren’t the menu. They’re worked examples, mapped to where my own weeks leave a record: user interviews, hard meetings, and the week itself. Treat them as illustrations of the shape, not the full list; your calendar will suggest loops mine can’t. A catalog of other places to point the same shape comes right after these three.

Loop 1: Review your interviews

After a user interview, I copy the full transcript into the LLM and ask it to review the interviewer. Not the user’s answers, not a summary of insights: me. The phrasing of my questions, the follow-ups I didn’t ask, how much of the conversation was my own voice. The distinction matters because “summarize this interview” is production work, and every PM already does it. ”Review how I ran this interview” is feedback work, and it’s the version almost nobody asks for.

Here’s the prompt I use, lightly cleaned up. Yours will look different; the load-bearing parts are the named rubric and the quoted evidence.

Here is the verbatim transcript of a user interview I ran today. Review the interviewer (me), not the user. Grade me against The Mom Test’s three rules: talk about their life, not my idea; ask about specifics in the past, not hypotheticals about the future; talk less, listen more. For every finding, quote the exact line from the transcript. List every question I asked, label each one open or closed, then tally the list. End with the one change that would most improve my next interview.

The rubric is The Mom Test, Rob Fitzpatrick’s 2013 book on customer conversations, and spelling its rules out inside the prompt matters. The critique works even if you (or the model) never opened the book, and you learn the rubric by reading your own violations of it. And the leading questions were only the loudest finding. The same session flagged the moment a user mentioned a workaround and I moved on instead of digging. It listed my questions, and over half were closed. None of that was in my notes, because my notes were about the user.

The critique is the start, not the verdict. I discuss it, and the discussion is where the learning happens. The rule I follow: seek evidence, don’t argue. “Show me the line” is a safe question. “Rewrite that question the way I should have asked it” is a great one. Protesting (”I don’t think that was leading”) is the one move to avoid, because models tend to fold when you push back, and a critique that folds is worthless. Treat the chat like a film session with a coach, not a negotiation over your grade.

Interviews are the easiest tape to start with: low stakes, famous rubric, fast wins. The harder tape is the meeting where you had something to lose.

Loop 2: Debrief your hard meetings

Some meetings matter more than others. The technical review where the architecture pushback derailed your agenda. The leadership meeting where your update ran long and the ask got lost. Your calendar is full of them, and that’s not the failure it’s usually made out to be: meetings are where product work happens. Spending most of your week in meetings isn’t the problem. Letting them end with nothing actionable is.

I used to replay the hard ones in my head on the way home, which felt like reflection but was really just rumination. Now I feed the transcript (or my detailed notes, when there’s no recording) to the LLM and ask for a debrief of my own performance. How I presented, where I lost the room, how I handled the pushback, what to change next time. The hours were already spent; the debrief is how they pay something back.

This move has a precedent you already practice at the team level. Product teams close every sprint with a retrospective: what went well, what didn’t, what changes next time, run by the team, for the team. Norman Kerth, who wrote the practice’s handbook (Project Retrospectives, 2001), even gave it a safety rule, the Prime Directive: whatever we discover, everyone did the best they could with what they knew at the time. Yet the ritual usually stops at the team boundary, and nobody runs one on their own performance in a single meeting. A meeting debrief is a one-person retrospective, and AI makes it cheap enough to run after every meeting that matters. The prompt is shorter than the interview one, because there’s no famous rubric for meetings. I anchor it on my goal instead.

Here is the transcript of a technical review I ran today. My goal was alignment on shipping the migration in two releases instead of one. Review my performance, not the decisions: how I opened, how I responded to pushback, where I talked past someone, where I lost the thread. Quote the line for every finding. Then give me the strongest case that this meeting went badly for me. End with what I should do differently in my next meeting with this group.

The debrief pays out twice. Once right after the meeting, when it shows you the exact moment you started answering a different question than the one asked. And once before the next meeting, when you reopen the same chat and prep: same audience next month, help me open differently, here’s what I’m walking in with. One honest limit from experience: this loop needs an artifact. Hallway conversations leave no tape, and a debrief of your own recollection is the friendly narrator reviewing itself.

Interviews and meetings are single events. The third loop is what strings them into actual improvement.

Loop 3: Review the whole week

This is the original habit from that training, and it’s still the anchor of everything above. Every Friday, three or four honest lines: what went well, what didn’t, what changes on Monday. Five minutes, no template, one running file. The one-change outputs from the interview and meeting loops land here as lines. Without the written close, the critiques evaporate by Tuesday.

But run the first two loops for a few weeks and you meet their limit: they only cover the events you picked. A product week is fifteen meetings, and you reviewed two. Nobody has the hours to chat-review a whole week one transcript at a time, and pasting fifteen transcripts into one window isn’t a workflow, it’s a punishment. The weekly review doesn’t need a better prompt. It needs to become a system. Same folder of inputs, same rubrics, same output format, every single week: that’s not a conversation, that’s a job. And a repeated job with fixed rules is exactly what Claude’s skills are built for: a skill is a folder with an instructions file that teaches Claude a task once, so it can run it every time after.

Here’s the shape of the system to build, and it’s smaller than it sounds.

The skill’s instructions fit on one page: read every transcript in this week’s folder, review my performance in each one against my rubrics (The Mom Test rules for interviews, my stated goal for everything else), quote the line for every finding, then draft my three-or-four-line retro and append it to the running file.

Collecting the folder is lighter than it sounds. Zoom transcripts download as text files, Google Meet saves them as Docs in a Drive folder, and Granola keeps the full conversations behind its notes. And scale is where the review gets interesting: a reviewer that reads all fifteen meetings can tell you the same tic showed up in four of them, which no single-meeting debrief will ever surface. If you’ve never built a skill, it’s an afternoon, not a project: I walked through the full method in How to build an AI helper for your team with Cowork, and the same steps apply here with a different instructions page.

Then remove the last manual step. Claude Cowork now supports scheduled tasks: set the skill to run every Friday afternoon and it executes remotely, even with your laptop closed. You open the draft with your coffee, check the quoted lines, and spend your five minutes editing instead of reconstructing. The honest writing stays yours; the system makes sure it starts from evidence instead of memory, at the scale of your whole week.

Now, about that 1% pitch. Clear’s canonical version is 1% per day, which compounds to 37x in a year. The training I got was humbler: 1% per week, which works out to about 68% better over a year (1.01^52 is 1.68, if you want to check the math). Treat both as the spirit of the practice, not a forecast. Skill isn’t a number, and nobody can measure 1% of interviewing. The idea’s lineage runs through British Cycling’s ”aggregation of marginal gains”, but the version that changed my weeks came from a trainer with a marker and a flipchart. Small weekly changes stack, and a written retro is what keeps them from resetting to zero.

Three loops, one of them running itself. But the shape doesn’t stop where my calendar does.

More loops to try

Once you see the shape (artifact in, critique against a standard, one change out), you start spotting loops everywhere. I went looking for the ones with the strongest cases behind them, and this is the shortlist I’m working through next. If a week of your work leaves an artifact, there’s a loop that can read it.

• The decision journal. When Michael Mauboussin asked Daniel Kahneman what single thing would most improve decision-making, the answer was a cheap notebook: write down each big decision and what you expect to happen. Months later, the LLM grades your calibration against outcomes: “you were overconfident on every timeline that depended on another team.”

• The forecast scorecard. Philip Tetlock’s Superforecasting is blunt about how forecasters improve: “Forecast, measure, revise: it is the surest path to seeing better.” Put numbers on your own calls (”70% we ship by March”), then ask the model each quarter where your numbers run hot.

• The spec-versus-shipped review. Feed January’s PRD and June’s launch numbers into one chat and run a retrospective on the pair: what was supposed to happen, what happened, why, and what changes next time. The gap between those two documents is feedback nobody schedules a meeting for.

• The rehearsal review. Toastmasters assigns a dedicated person, the Ah-Counter, just to count filler words. Record one run-through of your next big presentation, and the model plays that role free, then tells you how many minutes pass before your headline number shows up.

• The hidden-ask audit. Paste a week of your sent emails and Slack requests and ask which sentence carries the actual ask. Consultants are trained on Barbara Minto’s Pyramid Principle, answer first and reasoning after; most of us put the answer somewhere in paragraph three.

• The 360 synthesis. Paste every review packet you’ve received and ask what three or more people said independently. Any single rating is mostly rater (that’s the idiosyncratic rater effect again), but a theme that survives five raters and three years is genuinely yours.

Whichever loops you pick, they all lean on one fragile assumption: that the reviewer tells you the truth. By default, it won’t. The model is trained to be nice to you.

Make the critique honest

“The LLM just flatters me.” If that’s your objection, you’re right, and it deserves a serious answer. Sycophancy is a documented, trained-in behavior of assistant models across vendors, not a paranoid suspicion. Anthropic researchers showed that state-of-the-art assistants are consistently sycophantic, likely because human preference data rewards agreement (Sharma et al., published at ICLR 2024). You’ve probably felt it yourself: in April 2025, ChatGPT got so flattering that OpenAI rolled the model back within days. Ask a naive “how did I do?” and you’ll get praise plus three soft suggestions. By default, the model is a cheerleader wearing a reviewer’s badge.

The fix isn’t finding a more honest model. It’s writing prompts that leave no room for flattery. Five mechanics do that work for me.

1. Give it the artifact, never your self-report. The model can’t flatter its way past a transcript, but it will happily flatter its way through your own account of the meeting. You already know which narrator wrote that one. Asking about yourself is the worst case: one 2025 study found LLMs preserve the asker’s self-image far more than human advisers do. The tape is your protection.

2. Name a rubric. “Grade this against The Mom Test” beats “was this good?” every time. A named standard gives the model permission to find faults and a scale to grade on. No famous rubric for your artifact? State your goal for the meeting and let that be the standard.

3. Require quoted evidence. Every finding must cite the exact line. This converts vague praise into checkable findings, and it screens out invented criticism too. The same rule governs the discussion afterward: ask “show me the line,” never argue “I don’t think that was leading.” The same Anthropic research found assistants wrongly walk back correct answers when users express doubt. Evidence-seeking keeps the critique standing; arguing makes it fold.

4. Ask for counts, but make it list first. Talk ratio, open versus closed questions, minutes before the first real question. Useful numbers, and a known weak spot: a one-shot count is an estimate wearing a number’s costume. So make the model quote every question, label each one, then tally the list. The list is checkable against the tape, and the tally of a list is reliable.

5. Make it argue the prosecution. “Give me the strongest case that this interview went badly.” The model is excellent at arguing a side when you tell it which side to argue. Flattery has nowhere to hide in a prosecution brief.

One more objection deserves a straight answer, because legal will raise it before you do. Transcripts carry other people’s words. User interviews sit under whatever consent your research process already runs, so check that first. The load-bearing rule is the workspace: run the loop inside your company’s approved AI workspace, the same environment where your meeting tool already stores the recording (hand-anonymizing a 40-minute conversation is fiction, so don’t build your compliance on it). Strip names where it’s quick to do. And keep other people’s performance out of scope: the loop reviews you, not your colleagues.

Honest critique, ten minutes a week, grounded in what was actually said. The last move turns a personal practice into a leadership one.

Bring it to your team

Everything so far works for one person: you. But if you manage PMs, you’re also the person whose questions and standards your team is trying to guess at. The manager’s version of the practice has three moves: model it yourself, build it into a tool your team owns, and keep your hands off their critiques.

Show your own findings first

You don’t roll this out with a deck. The practice spreads in an embarrassingly simple way: you talk about your own findings first. In your next team meeting, mention what your review found. “It went through Tuesday’s interview and I did 40% of the talking. I’m working on it.” That sentence costs you thirty seconds of vulnerability and buys the whole room permission. Naming your own findings first is what makes the practice safe to copy.

Build the 1:1 prep skill

Then give the practice a tool. Every manager carries a mental list of the questions they always ask in a 1:1. What moved the metric this week? What’s the impact of the thing you shipped? What’s next, and what’s blocking it? Your team knows the list exists, but they only meet it one meeting at a time, live, when it’s too late to prepare. Put that list into a skill, and your questions become something your team can use instead of something they sit through. The skill’s context page holds three things: the company’s OKRs, the metrics your team owns, and your standing questions.

Here’s how it runs once it exists.

A PM runs their weekly update through the skill before your 1:1. It asks your questions before you do, checks the update against the real OKRs, and flags what’s thin: an impact claim with no number, a next step with no owner. They arrive having already answered the routine layer, on their own terms, with no audience. And the skill improves the same way you do: every time you catch yourself asking a question it missed, add it to the context page after the 1:1. I’ve built this exact shape before for AI-tooling questions, and the full walkthrough carries over here: same build, different context page.

Be honest with the team about what it’s for. The skill will never replace the conversation; it changes where the conversation starts. The routine questions get handled before the meeting, so the meeting spends its thirty minutes on what actually needs two humans: the trade-off they’re unsure about, the stakeholder they can’t read, the call that needs your judgment.

Keep the loop theirs

Hand over the loop, not a verdict. For the PM whose interviews keep coming back thin, the move isn’t “your interviews are weak.” It’s “run your next transcript through this prompt before our 1:1, and bring what surprised you.” The critique stays theirs. You never read their review; you only ever see the change they choose to make. That’s the oldest rule of the retrospective, the one Kerth wrote his Prime Directive to protect: the review belongs to the people who did the work, not to an evaluator. The moment you ask to see their critiques, the practice turns into surveillance, and people quietly stop running it.

What shifts is the shape of your 1:1s. You were never going to give transcript-level feedback to six PMs; there aren’t enough hours in your week, and there never were. Now that feedback exists without you. The conversation moves from “here’s what I think you did wrong” to “what did your review find, and what are you changing?” You stop being the bottleneck on your team’s craft feedback without becoming absent from it. You coach the change instead of hunting for the evidence.

And if you keep one line from all this, keep this one: you get a performance review twice a year; your transcripts can give you one every Friday. It works in a 1:1 with the PM who wants to grow faster. It works on your own tape, this week.

Start this Friday

You don’t need a new tool, a habit system, or anyone’s permission. You need one artifact from this week and ten minutes on Friday. Pick the tape where you already suspect the truth lives.

• Your last user interview. Start here if you run any: the rubric is famous and the wins come fast.

• The meeting that went sideways. The transcript is already in your drive. Ask for the debrief you’ve been replaying in your head anyway.

• Your old retros, if you keep any. Paste the last two months and ask for the pattern you keep repeating.

• A decision you’re about to make. Open the journal with one dated entry: what you decided, what you expect to happen. Future you collects the feedback.

The training that started this for me lasted one day, and I’ve forgotten almost all of it. The habit it left keeps compounding. That’s the honest promise of this whole practice: no transformation by Friday, just next-quarter you asking better questions, running tighter meetings, and knowing exactly what you’re working on. Start with one transcript. The 1% is already in there.

In early 2025, I watched a French designers’ collective called Designers Éthiques lay out every one of these objections in one 40-minute talk. It’s stayed with me since. Not because it was contrarian, but because it wasn’t. These were rigorous professionals making the strongest version of the case: an eco-design specialist, a design researcher, an ergonomist. No trolling, no panic. It’s the sharpest compact map I’ve found of what your own team members are thinking and mostly not saying in the meeting.

If you’re leading a team through an AI transition, you’ve met these objections too. Maybe in a retro, maybe in a 1:1, maybe as a silence that never turns into usage. And you’ve probably been handed exactly one playbook for them: overcome the resistance. I think that playbook is wrong. Your skeptics are mostly raising real problems, and the fastest way to lose them is to debate.

The better move: sort each concern into one of four kinds. Some are true, and you change what you adopt. Some are risks you control, and you change how you adopt. A few are misconceptions, and you correct them with evidence instead of marketing. One or two are values, and you respect them. The payoff isn’t a converted team. It’s AI use your team can defend out loud.

The playbook, at a glance:

• The trap. I’ll show you why winning the argument against a skeptic loses the team.

• The sort. Here’s the four-kind triage that replaces the debate: true, fixable, misread, or a value.

• The map. You get the seven concerns you’ll hear most, each with an honest verdict, the move that follows, and the one resource worth forwarding.

• The practice. What a defensible team AI posture looks like once the sorting is done.

By the end, you’ll have a verdict and a concrete next move for the seven objections coming your way this quarter, plus the one sentence that keeps a skeptic on your team. You stop dreading the ethics conversation and start using it to make your team’s AI practice sharper. No slides required, no philosophy degree either.

Let’s sort this out.

Why winning the argument loses the team

The talk opens with a moment I can’t stop thinking about. At a green-IT conference workshop, in front of the most skeptical crowd available, the speakers ran a session asking “AI: in or out?” Almost every group landed on “in,” reasoning that it’s here anyway, there’s no choice, so let’s make it as clean as possible. One speaker was troubled by exactly that phrasing. ”We have no choice” is not what agreement sounds like, it’s what resignation sounds like. If your team adopts AI in that spirit, you didn’t win them. They just stopped telling you things.

There’s a second reason the debate is rigged before you open your mouth. Your team members already met AI adoption as users, and it wasn’t polite. The design researchers at Limites Numériques documented the pattern: AI buttons pushed front and center, features switched on by default like Strava’s Athlete Intelligence, dialogs that offer “try it” and “not now” but never “no.” Sparkles and purple everywhere, the visual vocabulary of magic. When you pitch AI to your team with vendor enthusiasm, you pattern-match to the forced adoption they already resent. They’ve heard “this will make everything better” before, from a button they couldn’t refuse.

Organizational research has said this plainly for years. In their 2008 Academy of Management Review paper, Ford, Ford and D’Amelio argued that resistance to change isn’t a defect in the resisters. Change agents cause a good share of it themselves, and resistance is better treated as a resource: engagement, feedback, proof that people take the change seriously. The person pushing back is often the person paying the most attention.

If you read How to lead a tech team through the AI shift, you might spot a tension here. I argued there for the 20-60-20 rule: pour your energy into your champions and the watching middle, and stop exhausting yourself arguing with the resistors. I stand by every word, and this article doesn’t change the math. Sorting is not arguing, and what follows is not a conversion campaign. It’s for the conversations that find you anyway: the 1:1 where a concern lands on the table, the team meeting where a hand goes up. You still don’t chase your skeptics; you answer well when they’re in front of you, because the watchers are scoring how you do it. One honest answer to a skeptic moves ten watchers.

And there’s simple arithmetic about credibility. Argue once against something that turns out to be true, say the data-center water numbers, and you lose the room for a quarter. The skeptic came with figures. You came with talking points. So the job isn’t to win. It’s to sort.

Sort the concern before you answer it

Here’s the move that changes the conversation: treat each concern as a claim to classify, not an attack to parry. A triage nurse doesn’t argue with symptoms. She figures out what kind of problem she’s looking at, because the kind determines the response. Four kinds cover almost everything your team will raise.

• Kind one: they’re right. The data-center buildout really is doubling electricity demand. When a concern is true, the honest answer starts with “you’re right,” and the follow-up is a change in what you adopt. Nothing builds credibility faster than a concession nobody expected.

• Kind two: a real risk you control. Juniors losing the review and mentoring that builds expertise. This one isn’t about the industry, it’s about your team, which means your working agreements decide whether it comes true. The answer is a rule you set together, not a rebuttal.

• Kind three: a misconception. “Every prompt is like pouring out a bottle of water.” The per-prompt numbers are knowable, and they don’t say that. Correct the arithmetic gently, with sources, and without dismissing the worry underneath it. The worry usually points at something real at a different scale.

• Kind four: a value. “I didn’t get into this craft to supervise a machine.” You can’t argue someone out of a value, and trying reads as disrespect. What you can do is design roles so the value survives, and be honest about where the boundary sits.

One subtlety makes the whole sort work: the same concern can live in two kinds at once, at different scales. The environmental objection is true at industry scale and misread at the single-prompt scale. The sort forces you to say which scale you’re answering, out loud. Sorting is your actual job in this conversation; the debate is optional.

Here’s what the sort sounds like in an actual 1:1, in three moves.

• First, ask for the strongest version: “Give me the strongest version of this worry. Convince me.” That one question replaces the debate with respect, and it surfaces the real concern instead of the polite one.

• Second, name the kind out loud in plain words: “I think you’re right about this one,” or “this one is ours to fix,” or “I think the numbers say otherwise, let me show you,” or “that sounds like a value, and I won’t argue with a value.”

• Third, before the conversation ends, commit to one follow-up in writing: the rule you’ll add, the number you’ll check, the thing that goes out-by-default. A sort that doesn’t end in a written commitment is just a nicer way of nodding.

So let’s put the framework to work, one conversation at a time.

Answer the seven concerns, one by one

The map below covers the seven objections I keep hearing. Six of them the talk maps better than anything else I’ve found; the seventh, what AI does to our thinking, has grown loud enough since to earn its own place. Your team may add others: discrimination baked into training data, deepfakes, privacy. The sort handles those the same way.

The planet

Give this concern its strongest version, with numbers that survive checking, because the talk’s own figures were compressed in places. The IEA’s 2025 Energy and AI report projects data-center electricity demand will more than double by 2030, to around 945 TWh, slightly more than Japan’s entire consumption today. Alex de Vries projected in Joule that AI-specific consumption alone could reach the scale of the Netherlands or Argentina by 2027. Water follows the same curve: Microsoft’s use jumped a third in 2022 while Google’s rose a fifth, and researchers behind the Making AI Less “Thirsty” paper project AI water withdrawal reaching half of the UK’s annual total by 2027. The verdict: at industry scale, your skeptic is right.

There’s a sharper twist worth conceding too. A Guardian analysis found the real emissions of the big providers’ own data centers ran about 7.6 times higher than officially reported. The trick is accounting: renewable-energy certificates let a company report clean power it never actually consumed at the site. Notice what that is: not a technology problem, an honesty problem.

Now the misread half. Per-prompt costs are measurable, and they’re small. Google’s published figure for a median Gemini text prompt: 0.24 Wh of energy and 0.26 mL of water, counting on-site water only. Mistral’s lifecycle study counts everything upstream, training included, and lands at 45 mL per response. The gap is scope, not virtue, and either way one prompt sits far below one kilometer of driving or one beef meal. Correcting this arithmetic matters, because a team member who quietly believes each prompt burns a lake will never use the tools well. The obvious comeback is fair: small times billions of prompts equals the buildout we just conceded. Both things are true. Your prompt is cheap; the trajectory is not; your subscription is a vote for it.

The move is to stop pretending it’s free, and to put that in writing. Three rules are worth adding to your team’s working agreements.

• Default to the small model, escalate on need: most tasks don’t need the frontier model, and the price gap is the energy gap.

• No AI where a script does the job: a regex, a spreadsheet formula, or a cron job costs nothing and never hallucinates.

• And one screening question before any new AI use ships: what is this for, and would we defend it at scale?

Expect efficiency gains to get eaten unless you watch for them; the talk calls this the rebound effect, where every drop in cost per task invites more tasks.

When the conversation deserves more than a meeting slot, forward Andy Masley’s cheat sheet on AI and the environment. It’s the single best document I know for exactly this conversation: it runs the per-prompt arithmetic against everyday life (about 1,000 prompts to move your daily energy use by 1%) and still concedes the industry-scale buildout is a real question. Handing your skeptic the strongest version of the counter-arithmetic beats paraphrasing it in a meeting. What you can say: “You’re right about the trajectory. Here’s what our use actually costs, and here’s what we choose not to use it for.”

That answer covers the machines. The people who made the models possible are a harder conversation.

The people behind the data

The concern: the models were trained on work scraped from people who never consented, and the labor that made them safe was outsourced at poverty wages. The record backs it. TIME’s investigation documented Kenyan workers labeling toxic content for less than $2 an hour. And the talk’s own example has a scholarly source: Le Ludec, Cornet and Casilli documented French AI firms outsourcing annotation to Madagascar at poverty wages. This isn’t someone else’s supply chain. The verdict: they’re right, and no arithmetic rescues it.

So don’t reach for a rebuttal, because there isn’t one. What a product team controls is procurement and posture, and you can make both concrete. Write three questions into your next tool evaluation: where does the training data come from, what licensing deals exist, and what are the annotation labor practices. The differences between providers are inspectable: Mistral publishes a full lifecycle audit with the French environment agency, and some providers sign licensing deals with the newsrooms they train on, the way OpenAI did with the Associated Press and Le Monde, instead of scraping them. Your team can prefer the better ones and say why. For the teammate who wants the full picture, the book to pass around is Feeding the Machine, written by three Oxford Internet Institute researchers from years of interviews with the annotators and moderators themselves.

The other half of the move is harder: don’t launder the discomfort. Deciding to use AI with open eyes is more respectable than pretending the problem is solved. Some weeks the honest position is “this bothers me too, and here’s why I still think our use is defensible.” What you can say: “You’re right, and we can’t fix it from here. Here are the three questions we now ask before picking a provider, and here’s where I still think it’s worth it.”

The next objection sits closer to home: what the shortcut does to your own team’s craft.

The craft and the shortcut

This is the worry I hear most from the best people. Expertise, the ergonomist in the talk explains, gets built four ways: learning, experimentation, collaboration with peers, and teaching. Then he asks the question that lands hardest: when you hit a problem, what’s more effective, asking your colleagues or asking the AI? In the Q&A, someone described a senior developer who used to review a junior’s code and now just asks the AI instead. The transmission chain broke, quietly, without anyone deciding it should.

Lisanne Bainbridge named this pattern in 1983. In Ironies of Automation, the paper to forward to the teammate who wants the lineage, she showed that automating the easy parts of a job leaves the human with the hard parts, minus the daily practice that kept them sharp. A field study of AI in a radiology practice by Gamkrelidze, Zouinar and Barcellini found exactly that shape: the fracture-detection AI worked as a genuine safety net, and it also bred overconfidence risk, while the transcription AI quietly deleted the verification step the medical secretaries used to perform. The verdict: this risk is real, and it’s the one you control most directly.

Which is good news, because the fix is working agreements, not philosophy, and they’re short enough to write down:

• Rule one: anyone still learning a domain gets human review on their work, full stop, and the reviewer is a person with a name.

• Rule two: AI explains before it produces; used as a tutor it strengthens the learning station instead of skipping it.

• Rule three: seniors keep protected teaching time (a weekly pairing hour, a monthly brown-bag), because teaching is their own expertise engine too.

The shortcut only breaks the loop if you let it become the default path. What you can say: “You’re right that the shortcut is real. So we keep the loop: review stays, teaching stays, and AI joins the loop instead of replacing it.”

Working agreements protect the team’s craft. The newer worry is what the tools do to each person’s head.

The thinking muscle

This is the objection that grew loudest over the past year, and it usually arrives as a sentence about the next generation: AI is making us dumber, and kids who grow up on it will never learn to think for themselves. It now arrives with a study attached. An MIT Media Lab experiment put EEG caps on 54 essay writers: the ChatGPT group showed the weakest brain connectivity, reported the least ownership of their work, and minutes after submitting, most couldn’t quote a line from their own essay. The authors named the lingering effect “cognitive debt.” A survey of 666 people added that the heaviest AI users scored lowest on critical thinking, with the youngest participants the most dependent. This worry has data now, and waving it off with “they said the same about calculators” doesn’t answer it.

The verdict: real at the learning edge, unproven as a sentence about humanity. Read honestly, the evidence is narrower and more useful than the headlines. The MIT authors answer “is ChatGPT making us dumber?” with a flat no in their own FAQ: one narrow task, a small sample, engagement measured during the task rather than lasting change. The 666-person survey is correlational; maybe people who think less critically simply offload more. The result that should actually shape your team’s rules is causal: a randomized trial with about a thousand students, published in PNAS, found that vanilla ChatGPT boosted practice performance by 48% and then cut unassisted exam scores by 17%. Same trial, one design change: a tutor version that gave hints instead of answers erased the harm completely. The variable isn’t AI. It’s whether the AI does the thinking or scaffolds it. That’s also exactly where the calculator comparison holds and breaks: fine to offload arithmetic, dangerous to offload the reasoning itself.

The move is to make “coach, not answer machine” a written rule for anything a person is still learning. Three rules fit on a sticky note

• Think first, prompt second: write your own answer, even five rough lines, before opening the chat.

• Hints before answers while learning: tell the model “ask me questions, don’t give me the solution” (the exact design that erased the harm in the trial).

• Explain it back: if you can’t restate what you shipped with the chat window closed, it isn’t yours yet.

For the teammate who wants the deeper story, forward The Conversation’s walkthrough of the MIT study; it takes the worry seriously and shows precisely what the study can and can’t say. What you can say: “You’re right to guard your thinking. So we use AI as a coach on anything we’re still learning, and we prove it by explaining our work with the chat closed.”

Guarding heads is one half of consent. The other half is who signed up for any of this in the first place.

Consent at work

The talk asks a question I’d put to any team verbatim: do you consent to your work being organized by your team, or by an AI? Then it sketches the scenario: a backlog where tasks are assigned by an AI, estimated by an AI, and timed by an AI that checks whether you matched its estimate. If that sounds like science fiction, the EU disagrees. The EU AI Act’s Annex III classifies workplace AI that allocates tasks or monitors and evaluates performance as high-risk, with employer duties to inform workers before deploying it. The high-risk obligations start applying in August 2026. That’s next month. The verdict: they’re right about the pattern, and whether it becomes true on your team is your call to make.

Reality in the field is still mostly softer than the fear, and it’s worth being precise about that. The talk pointed at the French public sector, where the Lyon school district runs an AI copilot called Cassandre that helps HR staff answer teachers’ questions about transfers. It doesn’t decide the transfers. The speaker’s warning is about where the road leads: to “it wasn’t a human who decided,” the sentence that ends accountability.

Parts of the private sector are already further down that road, and the two clearest cases come from companies everyone knows. Amazon’s own legal filings describe warehouse systems that track each worker’s rate and auto-generate productivity warnings and termination paperwork, with Amazon noting supervisors can override. And in 2023, an Amsterdam appeals court ruled that Uber had deactivated drivers through solely automated decisions, the practice the drivers’ union called robo-firing and the exact thing the GDPR’s Article 22 exists to prevent. The road the talk warns about doesn’t lead somewhere hypothetical; it leads to case law.

The move is to make adoption consent-real before anyone asks, with three rules written into your working agreements:

• Tools are offered, never mandated per person.

• Every decision that touches a person carries a named human owner, and the name is written next to the decision.

• Opting out is safe, with no career shadow, and you check twice a year that it’s still true in practice.

When legal wants the source, forward the Annex III text itself; it’s shorter than most policy memos. Nobody who works for you should ever hear “the AI decided.” What you can say: “A human owns every decision on this team, starting with me.”

And underneath every consent question sits the one your team member is actually asking.

The jobs

The ergonomist tells a story from his early career. His mentor asked: if you design software that makes the work twice as fast, how many workers are left in six months? It’s the question underneath every other question, and your team member asking it deserves better than the two stock answers: the vendor’s “nothing changes for you” and the doomer’s certainty. The talk makes the hollowness concrete with translators, a craft told for years that the tools would free them for “more interesting work,” by people who never say what work is more interesting than the craft itself. You can’t answer the mentor’s question for the industry. Nobody can. The verdict: partly true, partly unknowable, and the worst response is a promise you can’t keep.

Start by questioning the premise. “Twice as fast” itself deserves doubt: METR’s randomized trial found experienced developers were 19% slower with AI tools while believing they were 20% faster. That perception gap is the strongest argument you’ll ever get for measuring your own team’s gains instead of quoting the pitch. And watch what the work becomes, not just whether it exists. When France’s tax authority deployed its pool-detection program Foncier innovant, field agents became desk agents. The job survived; the parts of it people liked didn’t all survive with it. Even the jobs that stay change texture, and naming that out loud buys more trust than any reassurance.

The move is to commit only to what you control, and to make each commitment checkable. Plans get heard from you first, not from a rumor: say it in the team meeting and repeat it every quarter. Skills investment happens on company time, with a number attached (a monthly learning day, a named course budget); an unnumbered commitment isn’t one. Roles get redesigned with the people in them, not around them: the person whose work is changing sits in the room where it’s redesigned. And when someone wants more than reassurance, forward David Autor’s essay on AI and the middle class: MIT’s leading labor economist, long a tech skeptic, arguing a future with neither doom nor denial in it. What you can say: “I can’t promise what the industry does. I can promise you’ll hear plans from me first, and that we invest in you while things change.”

One objection left, and it aims at the output itself.

The output you can’t trust

The talk’s most memorable stretch is about mediocrity. Models weight toward the most probable output. Ask for a pink elephant with wings and most image models fail, because the training data holds no pink elephants with wings. Ask for a wine glass filled to the brim; you get one filled halfway, over and over. The speaker’s conclusion: a designer who uses AI to generate ideas is stepping onto “a highway to the most probable solution,” and originality lives off that highway. Stereotypes ride along for the same statistical reason. The verdict: half right, and the right half is a map of where to use the tools, not a case for refusing them.

Because notice what the talk itself endorses: transcription, subtitles, handwriting recognition, classifying feedback. Small, checkable tasks with human expertise in the loop. That generalizes into the single most useful screen I know, and it’s worth writing exactly as a rule: AI-assisted work ships with a named human checker, and if nobody can check it, it doesn’t ship. Research gives the screen teeth. A CHI 2025 study of 319 knowledge workers by Microsoft Research and CMU found that the more people trusted the AI, the less critically they thought; the more they trusted their own judgment, the more critically they engaged. The check only happens when the human still believes their judgment beats the tool’s, so your job is to keep that belief alive. Taste, direction, and originality stay human, and they get more valuable, not less. For the teammate who wants the domain-by-domain version of “what AI can and can’t do,” the book is AI Snake Oil, by two Princeton computer scientists who respect skeptics enough to argue with evidence. What you can say: “You’re right about the median. That’s why we use it where we can verify, and why your judgment is the part that never gets delegated.”

Seven concerns, seven verdicts. Put together, they stop being a list of complaints and start being something else: the specification for a practice.

The objection to this whole approach

A sharp skeptic will see the framework coming and name it: “You’d already decided we’re adopting. The sorting is just a softer road to the same forced outcome.” You might be silently raising the manager’s version of the same objection: there’s a mandate, there’s a quarter, and this looks like a philosophy seminar. Both deserve a straight answer.

Take the skeptic’s version first, because it has a clean test. The sort has real exits, or it’s theater. If the “true” verdicts changed what you adopt, if something on your roadmap moved to out-by-default, if the person with the values objection still has their standing and their role, then the sorting was real. If you ran the conversations and nothing in your plan changed, the skeptic was right about you, and no framework fixes that.

Now the mandate pressure. Here’s the thing I’d want leadership to hear: the triage is the fast path, not the slow one. The alternative to honest sorting isn’t a converted team. It’s tools that get opened once, licenses that show up in the usage report as a flat line, and an adoption metric nobody believes. Quiet non-use is the most expensive outcome an AI mandate can produce, and steamrolling buys it wholesale. Sorting removes the actual reason people were stalling. That’s not a seminar. That’s unblocking.

And when the sorting is real, something better than compliance shows up: a posture you can write down.

The practice you end up with

Pull the seven moves together and you get a short team AI charter, written from your team’s concerns instead of a vendor deck. Mine would fit on a page. Four blocks.

• Where AI is in: tasks with verifiable outputs that a human reviews and owns. Transcription, classification, drafts that get read before they ship, code reviewed by a person who could have written it.

• Where it’s out by default: original ideation, unreviewed customer-facing output, and any decision about people.

• Who owns what: tools offered, never forced on a person; a named human accountable for every decision; an opt-out without career shadow.

• How we pick tools: providers that publish honest numbers on energy, water, and training data.

That last block has teeth because of the reporting gap from the planet section. A provider that understates its emissions 7.6 times has already told you how it handles inconvenient numbers. “Publishes honest figures” is a real screen, not a platitude. It costs nothing to apply and it sorts the market fast.

You can draft the whole thing in one team hour, and drafting it together is most of its power. Fifteen minutes: everyone writes their concerns down, anonymously if that helps people say the real one. Fifteen minutes: sort them into the four kinds together, out loud. Twenty minutes: draft the four blocks from what actually came up, not from a template. Ten minutes: put one name next to each block and a date three months out to revisit. A charter the team wrote in an hour beats a policy nobody read.

Two things make this charter travel further than your team. Legal will notice that the consent block is where EU compliance is already heading, with workplace AI obligations applying from August 2026. And the room where adoption metrics get discussed will notice that a written posture beats a usage dashboard nobody trusts. The line that carries all of it: “You’re right about X. Here’s what we changed so our use doesn’t have that problem. And here’s where I still think it’s worth it.” A manager who can complete that sentence for each concern doesn’t need to win any argument.

One reframe before you go write yours. I started this project wanting keys to change skeptical minds. I ended it convinced that’s the wrong target. Mindsets move when the practice becomes trustworthy, not when the pitch improves. The skeptic doesn’t need to love AI. They need to see that their concern changed something real.

Trust your skeptics

Somewhere in your team is the person who read the water numbers, the one who worries about the junior who never gets reviewed anymore, the one who wonders what these tools are doing to how we think, the one who quietly decided prompting isn’t the craft they signed up for. They were never the obstacle. They were the free red team, stress-testing your AI practice before it shipped. The org-behavior researchers knew it in 2008; the talk’s speakers knew it on that stage in early 2025. A team that sorted its concerns ships AI use it can defend out loud, starting with defending it to itself.

Your turn. Take the next 1:1 where a concern comes up and sort it out loud, all four kinds on the table. Write the strongest version of your best skeptic’s objection before you meet them, and open with it. Draft the four-block charter with the team, not for them. Pick one thing that goes out-by-default this week, and say why. And watch the talk that started this article, then forward it to the person on your team it will resonate with, with a note saying you want their read.

The speakers closed with a warning: AI will only ever offer you a mediocre dream, so take the time to dream your own. I’d turn it forward. Your team already has a dream about what their work should mean. Make your AI practice serve that, on purpose, and the skeptics won’t need convincing, they’ll be the ones holding you to it.

Here’s the problem: AI just made polished documents free. The tight résumé, the crisp case structure, the take-home that used to signal “this person is serious”: any candidate with Claude and an evening now produces all three. You’ve probably felt it already, reading applications that all sound suspiciously excellent. And it happened at the worst possible moment, right when you were asked to make your team AI-first and every open seat became a chance to hire the mindset. The signal collapsed at the exact moment the stakes went up.

So I went and read what the companies furthest into this shift actually did about it. I pulled the live PM job posts at Anthropic, Ramp, Notion, and Stripe, straight from their own job boards this week. I read the interview loops Sierra and Canva published, and the one candidates describe at Shopify. I read the fluency rubric Zapier now applies to every single hire. Different companies, different stages, one converging move. Stop grading the deliverable. Watch the candidate work. The bar isn’t “knows AI.” It’s “directs AI and catches it when it’s wrong.”

Here’s the whole arc in one picture.

• The diagnosis. I’ll show you why every stage of your current process stopped measuring anything real.

• The rebuild. Five moves, one per stage, with full requirements quoted from live postings and the interview questions from companies that already made them.

• The calibration. You’ll see the data on how much AI to actually demand, and the walk-back that warns against overshooting.

• The walkaway. One forwardable paragraph for leadership and your recruiter, plus a checklist to run on the req you have open.

By the end you’ll have a full replacement for the old process: job-description language you can borrow whole, two screening questions, a live interview format with a published AI policy, a work-sample design, and a decision rubric, all compressed into a closing checklist you can run this week. Your next PM hire stops being a coin flip on polish and becomes the clearest AI-first signal you send your team this year.

Let’s rebuild it, stage by stage.

Why your hiring loop went blind

Walk through the classic stages and name what each one actually measures. The résumé screen grades a document. The case interview grades a rehearsed structure, the kind a decade of candidates drilled from Cracking the PM Interview, McDowell and Bavaro’s 2013 prep book (”How many pizzas are delivered in Manhattan?”, “How do you design an alarm clock for the blind?”). The take-home grades another document, produced somewhere you can’t see. The panel grades a performance, and every stage grades some kind of output.

For years we told ourselves polish correlated with skill, that producing a crisp PRD required a crisp mind. The honest version is less flattering. Selection studies have long ranked résumé screens and unstructured interviews among the weakest predictors of actual job performance. The old process wasn’t a good instrument that AI broke. It was a weak instrument whose weakness AI made undeniable.

Picture the machine you’ve been running.

There’s a famous precedent for this kind of audit. Back in 2013, Google ran the numbers on its own famously clever interviews. Laszlo Bock, who ran People Operations there, went public with the verdict: brainteasers were “a complete waste of time” that “serve primarily to make the interviewer feel smart.” Google moved to structured interviews as a result. The lesson stings a little: a signal-free process can feel rigorous for years, because nobody checks. AI didn’t create the weakness. It removed the excuse for ignoring it.

You can watch the collapse happen in real interviews. In Nikhyl Singhal’s The Skip (Singhal ran product at Meta and Google), hiring manager Sam Stone observes that candidates who used AI to prep their case structure “will abandon the structure the moment Q&A starts, because they never internalized it.” In the same piece, product leader Mckenzie Lock shares the probe she used when screening at Netflix: when new information invalidates an assumption, “can you say ‘oh, that changes things’ and walk through updated logic?” And Meta gives the same diagnosis from the company side. Its new interview format puts an AI assistant inside the session, partly because, in Meta’s own words, it “makes LLM-based cheating less effective.” The companies that moved first aren’t fighting AI-polished candidates with detection tools. They moved the test to where polish can’t fake it.

Meanwhile the requirement itself went up, not down. In Microsoft and LinkedIn’s Work Trend Index, a survey of 31,000 people, 66% of leaders said they wouldn’t hire someone without AI skills. That survey is from May 2024, and nothing published since suggests the number softened. So you’re expected to hire for a skill your instruments can’t see, using stages that stopped measuring anything. The fix isn’t a better lie detector. It’s a process that watches work instead of grading output. Five stages, five moves.

Rebuild the loop stage by stage

Everything below comes from choices real companies have published, plus my own read on how to adapt them to a PM seat. Treat the five moves as parts to adapt, not a recipe to obey. Your org, your seat, your constraints. Run the ones that fit.

Here’s the full map before we go stage by stage.

Write AI into the job description

Stop writing “familiarity with AI tools is a plus.” The most advanced companies on this front put AI usage in the requirements section and name tools, the way postings used to name SQL. The strongest reqs treat AI usage as a requirement with teeth, not a vibe. All of these lines come from live postings I pulled off the companies’ own job boards this week, quoted in full so you can borrow whole requirements, not fragments:

• Notion (PM, Workspaces, an everyday workspace-product seat): “AI-pilled: You have comfort and an ongoing curiosity with using AI tools for your product development process.” And on every one of its 141 live postings, Notion adds: “You don’t need deep AI expertise for every role, but we do expect every Notino to be intellectually curious, drawn to tinkering and discovery, and excited to use AI as a real collaborator in their work.”

• Supabase (PM, Interfaces; the line recurs across their PM postings): “You use AI to move faster. You use AI tools to compress the slow parts of the job (research, drafting, synthesizing feedback) so you spend more time on judgment calls, and you’ve built enough with agents to have real opinions on what makes an interface agent-friendly.”

• Airbnb (PM, Incubations, a consumer seat asking for 10+ years of experience): “Demonstrated curiosity and point of view on incorporating AI tools into the product workflow.”

• Canva (PM for Pexels, their stock-photo product): “We’re looking for an AI-embracing entrepreneurial type product manager.” And in the experience list: “You embrace AI for automation, prototyping, report generation, data analysis, etc.”

• Stripe (growth PM, in Minimum Requirements): “Experience building AI-powered, conversational, or agentic products. You understand what it takes to ship AI that works reliably for real users, not just in demos.” And under preferred qualifications: “You have shipped products on top of these technologies, not just evaluated them.”

• Ramp (PM, Agentic CX): “We’re looking for builders who can vibe-code a prototype before lunch and write the rollout plan after.” The requirements spell it out: “Deep hands-on experience building with AI: you’ve prototyped, shipped, and iterated on AI tools, agents, or workflows, not just managed roadmaps about them. Technical fluency with modern AI coding harnesses (Cursor, Claude Code, Codex).” And: “Fluency in data and AI evals: you know when to trust numbers and when to lean on principles.”

• Anthropic (research PM, in Minimum Qualifications, in full): “Have a deep passion and curiosity for AI and LLMs. Use AI regularly.”

One distinction before you borrow. At Anthropic or on Ramp’s agentic products, asking for AI depth is as natural as a payments seat asking for payments experience: the product is AI, so the requirement writes itself. The lines worth copying are the other kind: a workspace product, a home-rental team, and a stock-photo library writing AI usage into ordinary PM seats. That’s the expectation spreading beyond AI products, and it’s the part that applies to your posting whatever you build.

Now decide which ask you’re actually making. The postings above split into three distinct tiers, and mixing them up is how job descriptions turn into wish lists.

Tier one asks for a PM who works faster with AI: the Notion, Supabase, and Airbnb language. Tier two asks for a PM who prototypes with AI coding tools by name: the Ramp language, echoed in Anthropic’s Labs PM posting (”Prototype with AI tools like Claude Code”). Tier three asks for a PM who owns evals, the tests that measure whether an AI feature actually works: Anthropic’s Claude Code PM posting wants someone who has “personally built agentic evals,” and Figma and Databricks write eval ownership straight into PM responsibilities. Most seats need tier one or two, and writing all three into one req doesn’t get you a unicorn, it gets you an empty pipeline. Pick the tier, write two or three specific bullets, and move on. The req is now making a promise the rest of your process has to keep.

Screen for their last real AI workflow

The old screening question (”are you familiar with AI tools?”) is answerable by anyone with a pulse and a ChatGPT subscription. Replace it with two questions that can’t be prepped.

• First: “Walk me through the last thing you used AI for, end to end.”

• Second: “When has AI been confidently wrong for you, and how did you catch it?”

Both were published by Metaview, a recruiting-software company, building on the fluency rubric Zapier’s co-founder Wade Foster made public. Vendor source, but the questions stand on their own. Ten minutes with these two questions tells you more than an hour of tool-name bingo.

The second question is the real detector. Anyone who genuinely works with AI has been burned by it: the hallucinated citation, the confidently wrong data pull, the beautiful answer built on a misread premise. They’ll tell you the story with texture and a little embarrassment, and then, crucially, they’ll tell you what they changed about how they verify. A candidate who has only read about AI gives you abstractions about “hallucination risks.” Real usage leaves scar tissue, and scar tissue can’t be improvised in a screening call. The question is kind, too: you’re inviting a story they’ll enjoy telling, not running a quiz.

Zapier, the most codified case I found, is worth mining beyond the two questions. It checks AI fluency at four fixed points for every hire in every role (”The application, Initial (human or AI) screen, Skills test, Executive interview”) and probes four components: “AI mindset, strategy, building, and accountability.” The detail worth stealing came in the rubric’s second version: they now assess “AI fluency slope, not a snapshot,” the candidate’s trajectory rather than today’s toolkit. Listen for the slope in the workflow answer: someone whose story ends “and last month I changed how I do it” is climbing; someone reciting a setup from a year ago has plateaued. Metaview’s read of the same rubric describes what the top of the scale sounds like: someone who “redesigns the work around AI” and “builds systems, prompts, or agents that other people use.”

You don’t need four checkpoints. You need these two questions asked early enough that nobody spends three weeks interviewing someone who fails the bar in minute ten. Your recruiter can ask them word for word, which is exactly the point. The candidates who clear them earn the round where things get interesting: the one with AI in the room.

Put AI in the interview room

Next decision: what happens when a candidate wants to use AI in your interviews? Right now most teams have no answer, which is the worst answer. Your interviewers assume it’s banned, your candidates assume it’s allowed, and whatever happens tells you nothing either way. The companies that took a position landed on three different doors, and all three work. The only wrong policy is not having one, and not telling candidates before they walk in.

• Door one: require it. Canva insists that engineering candidates use AI tools in its technical interviews, because the old process “asked candidates to solve coding problems without the very tools they’d use on the job.”

• Door two: allow it and score it. Shopify runs two AI interviews, and as candidates describe them, “Any AI tool is fair game. Cursor, Claude Code, ChatGPT, Copilot, whatever you normally use.”

• Door three: keep the live round unassisted. Anthropic publishes its candidate policy: draft your application yourself and refine it with Claude, but the live interview is you alone.

These examples all come from engineering hiring, and that’s worth being honest about. The policy decision transfers to a PM loop as-is; the exercise formats need adapting, and the work-sample stage does that adapting.

Once AI is in the room, score direction, not output. One Shopify interviewer, in those same candidate reports, put the rubric in one line: “We don’t want the AI to work for you. We want you to tell it what to do.” Canva’s own interviewer guide tells candidates to ”take time to review and understand everything that gets generated”. And Hello Interview, the site documenting these formats, states the stakes plainly: “The moment it starts making the architectural calls while you sit and watch, you’ve given away the exact thing the interview is there to assess.” One Rippling candidate was rejected for having “relied too heavily on AI even though their initial approach was correct.” Over-delegation is a scoring axis now, not just under-use. Watching someone accept AI garbage without blinking is the most informative thirty seconds in the whole process.

For the conversational rounds, the questions change too. Aakash Gupta, who coached dozens of PMs through these rounds and collected the exact questions they got asked, reports prompts like: “Your chatbot’s response quality dropped 15% last week. How do you debug this?”, “Tell me about a time you shipped an AI feature that backfired,” and “Explain embeddings to a product designer on your team.” Notice what these test: judgment about systems that fail probabilistically, not feature-design theater. Lift the ones that fit your product and retire a case study to make room.

Swap the take-home for a live build

The take-home is the most broken stage in the old process, because it’s a pure artifact produced entirely off-camera. You’ll be tempted to keep it and bolt on AI-detection. Don’t. Detection is a race you can’t win, and it aims at the wrong question: you don’t actually care whether AI touched the deck, you care whether this person can think. So replace the stage entirely: watch the work happen live.

There’s solid science behind this swap, and it’s older than the AI wave. A live build is a work-sample test, an instrument that personnel-selection research has ranked among the strongest predictors of job performance for decades (Schmidt and Hunter’s 1998 meta-analysis of 85 years of hiring data; the 2021 revision by Sackett and colleagues puts structured interviews at the very top). You’re not chasing a trend. You’re adopting the instrument hiring science recommended all along, now with AI in the room.

The PM version already exists in the wild, usually as a 45-minute prototyping case. Aakash Gupta has coached PMs through this round at “v0, Bolt, Figma, and Perplexity,” with real prompts like “Design Google Maps for the Blind” and “How would you prototype a Facebook Dating app,” and candidates choosing tools like Lovable, Bolt, or v0 for prototyping, or Cursor and Replit if they lean technical. Lewis Lin, who has written PM interview prep books for a decade, describes what it actually measures: “rapid problem-solving and product judgment under pressure,” not code quality. One calibration note so you don’t over-index: Google has only piloted this round, and it’s ”not a standard part of Google’s process (yet)”. Where it does run, Jaclyn Konzelmann, Director of AI Product at Google, has described what she looks for. You’re early, not late.

If you want a deeper shape to borrow, Sierra published its redesigned onsite in April 2026: Plan, a working session where “the candidate drives ideation, while interviewers ask questions to strengthen it”; Build, two hours with “the AI tooling and frameworks of their choice”; Review, where the team debates “the key product flows and choices they made.” It’s their engineering loop, so here’s my PM adaptation: plan a feature together for twenty minutes, give them a time box and whatever tools they like, then spend the last half hour on why they built what they built. What did they cut, what did they verify, what would they ship. The prototype is just the excuse. The review conversation is where the judgment shows.

Decide with a bar, not a ranking

Now the debrief, where all of this either compounds or unravels. The move: treat AI fluency as a minimum bar, and product judgment as the ranking. You’re not hiring the most AI-enthusiastic candidate in the pool. You’re hiring the best product judgment among everyone who cleared the bar.

Make the bar concrete, and borrow Zapier’s “Capable” level to do it: AI embedded in their actual work, “repeatable systems, not one-off prompts,” and clear impact on quality or speed. A candidate who can’t show a single real workflow in the screen, and who delegates every decision to the AI in the live build, doesn’t clear it. That’s a no in 2026 the same way “can’t write a PRD” was a no in 2019. But clearing the bar higher earns no extra points, because the ranking runs on something else.

The data backs the weighting. Across 592 AI-PM postings analyzed in early 2025, product management and cross-functional collaboration each appeared in 32% of listings, while prompt engineering showed up in 9%. Even the roles with AI in the title rank on classic judgment. The sharpest new skill to probe for is evals. Kevin Weil, who was leading OpenAI’s product org at the time, said it plainly in April 2025: “Writing evals is going to become a core skill for product managers.” Honest calibration: eval experience is a hard requirement for AI-product seats (it’s literally in Anthropic’s and Figma’s postings) and a strong differentiator everywhere else. Match the evals expectation to the tier you picked when you wrote the req, and don’t punish a tier-one candidate for missing a tier-three skill.

One more upgrade, borrowed from decision science. In Noise (2021), Daniel Kahneman, Olivier Sibony, and Cass Sunstein show how wildly hiring judgments vary between judges, and prescribe the fix: score each dimension independently, and delay the overall verdict until the end. “Fluency is a bar, judgment is the ranking” is that protocol in one line. And in your judgment scoring, weight the moment Mckenzie Lock’s probe targets: did the candidate ever say “oh, that changes things” and re-reason live? That moment can’t be faked, and it’s the single strongest signal your new process produces. Which leaves one question: how hard should you actually push the AI requirement?

How much AI to actually demand

Here’s the objection you might already be forming: “We’re not Ramp or Anthropic. Most PM postings don’t ask for any of this. My next PM still needs discovery, stakeholder work, and judgment. Isn’t rewriting the whole process premature?” The data honestly supports half of it. Itamar Gilad’s analysis of TrueUp data found that roughly one in seven open PM roles asked for AI skills as of early 2026, and he’s blunt about the hype: “There’s no indication that all PM roles will require deep AI knowledge and skills.” I’ll add my own check: when I scanned OpenAI’s 13 live PM postings this week, most carried no AI-usage bullets at all. At AI-native companies the expectation is ambient culture, not req language, and the explicit bullets cluster at companies making the shift deliberate. If your org writes its expectations down rather than absorbing them by osmosis, the bullets above are your template.

But look at what the objection is actually objecting to. The rebuilt process isn’t a higher AI bar bolted onto every stage. It’s better instruments for the things you already hire on. Even your most classic hire, the discovery-and-stakeholders PM with no AI ambitions, now walks through a world where every résumé, case answer, and take-home can be AI-generated. The screen question, the live build, and the re-reasoning probe measure judgment better than the old stages did, whatever the seat. Fix the instruments because they’re broken, not because every PM must become an AI specialist.

The walk-backs teach the closing lesson. Duolingo went AI-first loudly in April 2025, and a year later its co-founder Luis von Ahn dropped the idea of evaluating employees on AI usage: “the most important thing in your performance is that you are doing whatever your job is as well as possible. A lot of times AI can help you with that. But if it can’t, I’m not going to force you to do that.” His mandate was about evaluating current employees, not hiring, but the lesson transfers to the bar you set at the door. Don’t make AI zealotry the bar. Make judgment-with-AI the bar. The process you just rebuilt does exactly that: fluency is one gate, and it’s never the ranking.

Start with the req on your desk

When leadership or your recruiter asks what’s changing, here’s the paragraph to hand them:

AI made polished artifacts free, so our hiring process can’t grade artifacts anymore. We’re redesigning it to watch how candidates work: the req names real AI usage, the screen asks for their last AI workflow end to end, the interview puts AI in the room with a policy we publish, the work sample is a live build we review together, and the decision holds a fluency bar while judgment keeps the casting vote. The bar isn’t “knows AI.” It’s “directs AI and catches it when it’s wrong.

It travels well: to leadership, to your recruiter, to the staff PM who’ll run half the interviews.

A generation of hiring managers learned from Who, Geoff Smart and Randy Street’s 2008 hiring classic, to write the scorecard before meeting a single candidate. Same discipline here, one page shorter. The checklist, for the opening you already have:

• Job description: pick tier one, two, or three, and write it into the requirements as plainly as Anthropic’s “Use AI regularly.”

• Screen: the two questions, word for word, in the recruiter’s script.

• Interview: pick a policy (require, allow-and-score, or unassisted-live) and put it in the candidate prep email.

• Work sample: one prompt, their tools, 45 minutes, then thirty minutes on their choices.

• Decision: fluency is a bar, judgment is the ranking; one line in the debrief template.

Your hiring process is also a message, and not just to candidates. The team you already have reads every posting you put up and every interview you run as a statement of what the org actually values. When they see you watching how people work with AI instead of counting buzzwords, that tells them more about the AI-first mandate than any kickoff deck. The first candidate through the rebuilt loop will show you more in 45 live minutes than the last ten take-homes combined. The posting on your desk is due Friday anyway. Make it the first artifact of the new process.

Here’s the thing nobody tells you about that meeting. The spec already exists. It just isn’t written anywhere: it lives in the heads of the people who do the job today, stored in a language no data scientist speaks. What the team’s best performer would tell a new hire on day one? That’s your system prompt. The tabs she keeps open while she works? Those are your tools. The things she’d never let a new hire do alone? Those are your guardrails. Your job isn’t to write the spec. It’s to run the interviews that surface it.

In the previous post I wrote about how to pick the AI agents worth building: point the next one at your team’s real constraint instead of sprinkling agents everywhere. This is what comes after you’ve picked the spot. The whole method fits in one sentence: every component of an agent is the answer to a plain-language question, and version one should only do the work the team would happily give away. The rest is knowing which questions to ask, and in what order.

The whole method runs through one worked example, a customer support team, picked to make every step tangible: swap in finance, sales ops, or HR and the moves stay the same. Here’s the path at a glance.

• The gate. I’ll give you the five questions that tell you whether a job needs an agent, a simple script, or nothing at all.

• The interviews. Three conversations with a support team, with the exact questions to ask and what each answer is for.

• The translation. Every answer becomes a spec component, and you’ll prototype the agent yourself before anyone commits a sprint.

• The landing. You’ll see how to scope version one so a team that fears the project ends up asking the agent to do more.

By the end you’ll have the full interview script and the translation table between business answers and agent components. You walk into your next discovery meeting with questions the team can actually answer, and you walk out with a spec your AI team can actually build. There’s no platform to pick: the method works with whatever agent stack your company already chose (Amazon Bedrock AgentCore, Databricks, or something your AI team built themselves), and an AI workspace like Claude Cowork or Claude Code covers the prototype step.

Grab your notebook.

Decide if the job needs an agent

The oldest failure mode in innovation has a name: a solution in search of a problem. A technology shows up, it’s impressive, and whole organizations start asking “where could we use this?” The question is backwards, and the people who build things for a living have been saying so for years. Uri Levine, who co-founded Waze, put the fix on the cover of his book: fall in love with the problem, not the solution. Problems are constant; solutions are just the current best way to serve them. Nobody in your company needs an agent. They need the backlog to stop growing, the customer answered today, the notes written without staying late.

AI mandates invert this by design. “Agentify support” hands you a solution with the problem line left blank, and your own excitement pulls the same direction, because agents are genuinely fun to build. I’ve caught myself doing it: tool first, problem later, demo nobody asked for. So the first discipline of this job is to flip the sentence back. Start from the work, find the problem worth solving, then ask whether an agent is even the right shape of solution. Your job isn’t to bring agents to the business. It’s to bring the business a problem worth solving, with whatever technology fits.

The “right shape” question has a sharp edge to it. Anthropic’s Building Effective Agents draws the line I use in every scoping conversation. A workflow follows a path you script in advance: cheaper, faster, predictable. An agent decides its own path as it goes, and Anthropic reserves that for problems where “it’s difficult or impossible to predict the required number of steps.” In plain terms: if you can draw the flowchart, script the flowchart. An agent earns its complexity only when every case bends the flowchart.

I run every candidate job through five questions before committing to anything. The team can answer all five without a single technical term.

1. Does the work arrive different every time? If every item looks the same, a script handles it. Variety is the first hint you’re in agent territory.

2. Does handling it take judgment calls, or rules? If the team can write the rules on a whiteboard in an hour, write the rules into code instead.

3. What does a wrong answer cost, and can a human catch it in time? High stakes plus no review point means you’re not ready for an agent here, whatever the mandate says.

4. Does it happen often enough to matter? A judgment-heavy task that shows up twice a quarter never pays back the build.

5. Can an agent even reach the work? If the job lives in phone calls, hallway conversations, and a cheat-sheet in someone’s drawer, no spec survives contact.

OpenAI’s practical guide to building agents puts it bluntly: validate the use case first, “otherwise, a deterministic solution may suffice.”

Two contrasting tickets show the gate at work.

• A password reset arrives identical every time, follows three rules, and costs nothing when retried: script it, done in two weeks.

• A refund dispute arrives wrapped in order history, customer tone, and policy exceptions: that’s judgment, that’s volume, that’s agent territory.

The refund example isn’t even mine: OpenAI’s guide names “refund approval in customer service workflows” as its canonical case of agent-worthy work, the kind full of “nuanced judgment, exceptions, or context-sensitive decisions.”

And sometimes the gate says no. Telling a team “you don’t need an agent, you need a script, and it’ll be done in two weeks” feels like walking away from the mandate. It’s the opposite. Saying “no agent needed” when it’s true is the cheapest trust you’ll ever buy, and it’s the reason the next team opens their door when you knock.

For the team you’re about to meet, the gate says yes. Time to walk in.

Run the three interviews

The use case: a support team under a mandate

Time to run the method on something concrete, with you in the driver’s seat. Picture a mid-size B2B software company. A nine-person customer operations team handles order issues, refund disputes, and account changes: roughly sixty cases a day, forty minutes of handling time on an average case, a backlog that grows a little every month, and a hiring freeze. Leadership looks at those numbers and hands you the mandate: “bring AI to support.”

The team lead, call her Claire, runs a team that’s proud of its judgment. They talk customers out of churning every single week. What Claire hears in “agentify support” is a project aimed at her team’s future, and she’s not wrong to hear it that way: plenty of these projects get pitched exactly like that. The fear is rational. Pretending the fear isn’t in the room poisons every interview that follows, so name it in the first meeting and don’t argue it away. You can’t logic someone out of a fear the headlines keep feeding. You can only design the project so the fear doesn’t come true, and let the design speak.

The posture you take matters as much as anything you say. You’re not the expert arriving to optimize anyone. You’re the apprentice, and they’re the master craftsman: you’re there to learn the job well enough to describe it faithfully. An apprentice studies the master’s craft; the master stays the master. That single stance changes what people tell you. Experts perform for auditors. They teach apprentices.

One more reframe before the first interview, this time for leadership. “AI in support” is not a deliverable. A deliverable is a number: minutes back per case, backlog age, first-response time, with quality held steady on the numbers the team already reports, like satisfaction scores and escalation rate. Commit to measuring before and after. An agent without a number attached is a demo, not a delivery.

So you ask Claire for three slots on her team’s calendar, spread over one week: an hour with her at a whiteboard, ninety minutes with her two best people, and half a day sitting behind someone while they work. Plus ten recent cases: good, bad, and ugly.

The interview format descends from contextual inquiry, a research method Hugh Beyer and Karen Holtzblatt developed at Digital Equipment Corporation in the late 1980s. Its core instruction: study the work as it happens, not the process as documented, in the posture of an apprentice learning a craft. Every team has a standard operating procedure, the SOP: the official document that says how the work is supposed to run. It describes the happy path, and the job lives in the exceptions. Never ask a business team “what are your requirements?” Nobody who does real work can answer that question. Ask about last Tuesday instead.

Three conversations, three different jobs. Here’s the map.

Interview 1: map the job

One hour with Claire and a whiteboard. The goal is the map, not the details. Five questions do the work:

• ”Walk me through what happens between a case landing and a case closing. Whiteboard-level.”

• ”Where does the time actually go? Not where the process doc says. Where it really goes.”

• ”Which cases does your team dread? Which ones do people fight over because they’re interesting?”

• ”What work would your team happily never do again?”

• ”What numbers do you already report on?”

One hour of those questions produces something like this on the whiteboard.

Each answer has a destination:

• The walkthrough becomes the workflow skeleton.

• The where-does-time-go answer becomes your pain heatmap.

• The dread-versus-fight-over answer tells you what the team wants to keep, which matters more than what they want to lose.

• The last question hands you a value baseline without building a single new dashboard.

• And the “happily never again” question is the one to listen hardest to, because its answer becomes version one. For Claire’s team the answer is unanimous: the case wrap-up notes. Every closed case needs a summary in the CRM, the customer database where every interaction gets logged. Everyone writes them badly at 6pm. Nobody reads them until they desperately need them. Hold that thought.

Interview 2: learn the craft

Ninety minutes with the two people Claire names as her best: Marco, eight years on the team, and Ana, three. Screens open, real cases on them. This is where the agent’s actual behavior gets specified, one plain question at a time:

• ”Show me the last case that took you over an hour. Click through it exactly as you did.”

• ”What would you tell a new hire on day one that isn’t written anywhere?”

• ”What’s open on your screen while you work a case?”

• ”What do you find yourself re-explaining or re-looking-up, case after case?”

• ”When do you stop and pull in Claire or another team? What’s the trigger?”

• ”What would you never let a new hire do alone in their first month?”

The answers are the spec, verbatim. Marco’s day-one rule: “if the customer mentions a deadline, handle it today, even if the queue says otherwise.” That sentence goes straight into the agent’s instructions. His open tabs: the CRM, the order database, and the refund policy wiki page, which is outdated, and everyone knows exactly which parts to ignore. Write down which parts they ignore. That’s institutional knowledge no document holds. Then Ana pulls a laminated cheat-sheet out of her desk drawer, and that’s when you know the interview is working. Every team has one. It’s never in the wiki. And the never-alone list arrives without hesitation: refunds above a threshold, anything that smells legal, and any promise about delivery dates.

Interview 3: shadow the work

Half a day sitting behind Marco and Ana while they work real cases. No questions this time. You’re watching for the gap between the process document and the craft: the case that needs an actual phone call to the warehouse, the customer Ana recognizes as burned-twice-before from memory because the CRM has no field for it, the moment Marco overrides the policy page because he knows it’s stale. None of that appears in any SOP, and all of it decides whether an agent succeeds or embarrasses you.

You leave the shadow session with ten annotated real cases: the input, what the human did, and what “good” looked like. Those ten cases are the most valuable artifact of the entire discovery, because they become the test set your AI team grades the agent against. One honesty note: your three interviews will surface completely different specifics than these. The questions transfer. The answers won’t. That’s the point of asking.

Three conversations, one notebook full of plain-language answers. Now comes the translation.

Turn the answers into a buildable spec

Write the seven components

An agent spec has seven components. Here’s the part that surprises people: the team just answered all seven, and nobody in the room had to learn a single AI term. Five components come straight from interview answers, and the remaining two get derived from those five.

1. Instructions: the agent’s system prompt. This is the day-one advice, the rules of thumb, the tone. It comes from ”what would you tell a new hire that isn’t written anywhere?” Marco’s deadline rule goes into the prompt word for word. Collect the rules as the team says them, not translated into corporate language, because the best performer’s voice is the spec. Your data scientist will thank you: half of prompt writing is knowing what the expert actually does.

2. Tools: what the agent can look at and what it can do. Every tab open during the walkthrough is a read-tool candidate. Every action taken is a write-tool candidate. The split matters because it maps directly to risk: letting the agent read the order database is cheap to grant, letting it issue a refund is not. Version one gets read access to the CRM, the order database, and the policy wiki, and exactly zero write actions.

3. Memory: what the agent keeps between cases. This comes from ”what do you keep re-explaining?” and from Ana’s burned-twice customer. Three plain kinds cover most teams: facts about repeat customers, resolutions that worked before, and this-week context like a known shipping outage. The spec says what to remember in the team’s words, and the AI team picks how to store it. Skip the vector-database vocabulary entirely; it has no place in a spec meeting.

4. Workflow: the path a case takes, where the agent sits, and what wakes it up. Interview 1’s whiteboard map plus the walkthrough give you the steps. Now mark each step with one of three labels: script it, agent judgment, or human only. The gate from earlier applies per-step here, not per-project, and most “agent” projects turn out to be a workflow with two or three genuinely agentic steps inside. Then name the trigger, because it’s the first thing your AI team will ask: what invokes this thing, an event, a schedule, or a person? Version one’s trigger is one line: the agent wakes when the human closes the case.

5. Model: the brain you rent. The business chooses a model without ever hearing a model name, by answering three questions: how much judgment does a case take, how fast does the answer need to come back, and what’s a case worth in money? Capability, latency, cost. Your AI team maps those three answers to an actual model tier, and your job is keeping the trade-off honest when someone wants the premium model for wrap-up notes.

6. Guardrails and escalation: what it must never do alone, and who it calls. The never-alone list transfers verbatim: refunds above the threshold, legal-adjacent language, delivery-date promises, always a human. Escalation deserves the same respect: the trigger points from ”when do you pull in Claire?” become the agent’s own escalation rules. An agent that says “this one’s for you, here’s the file so far” earns trust faster than one that guesses, so spec escalation as a feature, not a failure state. Add one more spec line to this bucket: every agent action gets logged and is reviewable. The weekly review you’re about to set up depends on those logs existing, and legal will ask for the trail before anything launches.

7. Evals: how everyone knows it works. The ten annotated cases from the shadow session plus the metrics Claire already reports. The launch bar is simple: the agent drafts wrap-up notes for the ten golden cases, and Claire grades them blind against the human-written ones. Hamel Husain, whose evals guide is the one practitioners pass around, puts the stakes plainly: “Unsuccessful products almost always share a common root cause: a failure to create robust evaluation systems.” The evals aren’t the homework at the end of the project. They’re the first thing your AI team builds.

Now notice what just happened. The team wrote the spec. Nobody said “prompt engineering.” Nobody explained embeddings. Your entire contribution was asking questions about last Tuesday, sorting the answers into seven buckets, and reading them back until Claire said “yes, that’s the job.”

A spec in plain language, ready for builders. Almost. There’s one move left before the hand-off, and it’s the one that separates today’s product work from requirements-era product work.

Prototype the agent yourself

Here’s the gap in everything so far: three interviews captured what the team says about the work, and the shadow session caught some of what they do. Neither tells you how they’ll behave with the agent itself. Will Marco trust the draft, edit it, or quietly go back to writing notes his way? No document answers that, and a longer spec answers it least of all. In classic product work you already know the move: you don’t write a requirements document for a website and send it to developers. You click together a prototype in Figma or a no-code tool, put it in front of five users, and watch. Agents just got the same unlock: with AI coding tools, prototyping the agent is now cheaper than the meeting about the spec. Here’s the loop.

So before handing anything to the AI team, build the smallest thing that behaves like the agent. In Claude Cowork or Claude Code, it’s an afternoon of work. Write the instructions component into a skill, with Marco’s rules verbatim. Drop the ten golden cases in as files. Ask it to draft the wrap-up note for each one. No connectors, no permissions, no access requests: copy-paste stands in for every tool the real agent will get. What comes out reads like the future agent’s output, and the output is the only part the team can judge anyway.

Then test it like a product prototype, not like a demo. Run the sessions on the five-act interview the structure Google Ventures uses to test design sprint prototypes: a friendly welcome, a few context questions, introduce the prototype (”it’s cardboard, you won’t hurt my feelings”), real tasks, then a short debrief. Hand Ana a fresh case, let her work it, show the prototype’s note next to hers, and watch. The watching surfaces what three interviews didn’t: every note the team actually trusts ends with a who-to-call line, and nobody thought to say it out loud. The spec gains its most important line from twenty minutes of watching, not from any question you asked. There’s always a gap between what people tell you and what they do, and it closes only in front of something real.

The prototype doesn’t replace the spec, it hardens it. The spec still tells your AI team what to build. The prototype proves the team will use it, catches misreadings while they’re an afternoon cheap, and hands your data scientist a tested first draft of the prompt instead of a blank page. And the fearful room from the kickoff has now touched the thing, shaped it, and seen it stay a drafting assistant. The hand-off package just gained its most convincing item.

Hand it to your AI team

The hand-off package is six things, and none of them is a forty-page document. One page per spec component, in the team’s own words plus the derived decisions. The ten golden cases with the human answers attached. The prototype with its feedback log: what the team reacted to and what changed. The access list: which systems, read or write, granted to whom. The value baseline from Interview 1. And the escalation contacts with their triggers. That package answers most of the questions a AI team otherwise burns weeks discovering one standup at a time.

Each role picks up a different end of it. The data scientist starts from component seven: the golden cases become a test harness before anyone writes a prompt, which is exactly the order Husain’s evals-first practice prescribes, and the prototype’s skill file gives them a first prompt draft that already survived contact with the team. The data engineer starts from component two: connectors, permissions, logging. And you stay in the loop as the translator, running one ritual that outweighs everything else: a weekly transcript review, with someone from the support team in the room. Marco reading the agent’s attempts catches misreadings in days. Without the business in the review loop, misreadings surface at launch, in front of customers, with the team’s name on the queue.

Somewhere in that first planning meeting, you’ll get the pushback. It usually sounds like this: “Why all this ceremony? The process is documented. We can spec from the SOPs and ship this quarter.” Take it seriously, because it’s half right: the SOP is real and useful. It’s also only the happy path. The value and the risk both live in the exceptions only the team can name: the stale policy page, the burned-twice customer, the warehouse phone call. An agent spec’d from documentation automates the part that was never the problem, and breaks publicly on the part that is. And it breaks in front of a team that was never asked, who now has every reason to let it fail. That’s not a hypothetical. MIT’s Project NANDA studied hundreds of enterprise GenAI initiatives in 2025 and found most fail from “brittle workflows, lack of contextual learning, and misalignment with day-to-day operations”: the documentation-spec’d agent, described in one sentence. One week of light discovery, against a quarter of rebuilding. That’s the whole trade.

One last thing about the spec itself: it’s a living contract, not a document you sign once and file away. When the AI team hits an ambiguity, and they will (”what exactly counts as legal-adjacent?”), the answer is another fifteen minutes with Marco, not a guess hidden inside a sprint. The interviews don’t end at discovery. They just get shorter.

The build is underway. Now the part most agent projects skip until it’s too late: getting a team that fears the thing to want the thing.

Start with the work nobody wants

Go back to Claire’s face in that kickoff meeting. You don’t fix that with a slide about how AI augments people. Nobody who’s afraid has ever been reassured by a slide. You fix the fear with scope.

The move: version one does only the work the team would happily give away. Remember the unanimous answer from Interview 1: the wrap-up notes. So version one drafts the case summary after the human closes the case. It talks to no customer. It makes no decision. It sends nothing. Every person on the team keeps every bit of their judgment, and everyone loses their least favorite chore the same week. The French have a name for this kind of maneuver: cheval de Troie, the Trojan horse. Except this horse gets welcomed for what it visibly removes, not for what it hides. There’s no deception in it. The full ladder is printed on the side.

Be honest about version one’s number, because it’s small on purpose. Five minutes of note-writing per case, sixty cases a day: call it five team-hours back daily, plus CRM notes that people can finally use. That’s real, and it’s nowhere near the handle-time and backlog numbers from the value conversation with leadership. Say so plainly. The small number is the design: a small verified win buys the trust, and the patience, for the versions that move the big metrics. A demo impresses for a week. A chore that stays gone convinces forever.

There’s a principled version of “work nobody wants,” and it comes from Google’s site reliability engineering book, in a chapter by Vivek Rau. He calls it toil: work that’s “manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows.” Read that list back against the wrap-up notes and every box ticks. The scales-linearly attribute even loops back to the volume question in the gate: toil grows with the business, which is exactly why automating it compounds. Every business team has a toil pile, and they can name it in five seconds when you ask, which is why the “happily never again” question sits in the very first interview.

From there, the agent climbs a ladder, and the ladder has a rule.

• Rung one drafts the documentation after decisions are made.

• Rung two pre-assembles the case file before the human opens it: order history, past tickets, the relevant policy section.

• Rung three suggests a response the human edits and sends.

• Rung four handles one narrow, routine category end to end, with humans reviewing a sample.

The rule that makes this change management instead of scope creep: each rung happens when the team asks for it, not when the roadmap says so. Three weeks after launch, Marco asks why the agent can’t also pull the order history before he opens a case. That’s rung two, requested from the floor. And the ladder isn’t a trick I invented: it walks the classic automation-staging model that Parasuraman, Sheridan and Wickens formalized in 2000, from information gathering, to suggested decisions, to supervised action.

Two failure modes need naming, because the happy path above is only one data point. First, the stall: if nobody asks for rung two by month two, that’s data, not patience. It almost always means version one missed the actual most-hated chore. Rerun the “happily never again” question and re-aim; the miss costs weeks, not quarters. Second, the squeeze: leadership pushing the mandate faster than the floor pulls. The version-one number is what buys you the time. Report the small win monthly, and name the next rung’s trigger out loud: “we expand when the team asks, or when note quality holds for four weeks.” A named trigger turns “when will it do more?” into a measurable answer.

One caveat at the top rung. Lisanne Bainbridge’s “Ironies of Automation” (1983) showed that automating the routine work erodes exactly the skills humans need for the hard cases that remain. The fix belongs in the spec, not in a values statement: humans keep a rotation of routine cases, not just a review sample. The team stays sharp, and the agent stays supervised by people who still know the craft.

A few smaller moves compound alongside the ladder. Let the team name the agent; named things belong to the namer. Keep someone from the team in the weekly transcript review. Publish what the agent can’t do, the guardrail list, as prominently as what it can. And measure the team’s time back, never individual productivity: the first dashboard that ranks humans against the agent ends the whole experiment. When someone above you asks how you’re rolling out agents, here’s the paragraph to hand them:

Before we build any agent, we run three interviews with the people who do the job. The spec falls out of the answers: what they’d tell a new hire becomes the instructions, what they look at becomes the tools, what they keep re-explaining becomes the memory, what they’d never let a new hire do alone becomes the guardrails. We prototype it ourselves and test it with the team before the AI team commits a sprint. And version one starts on the work they’d pay to stop doing. The agent earns the interesting work later.

One team, one number, one chore removed. That’s how agent programs actually start.

Run your first interview this week

The mandate said “agentify support.” What you delivered is narrower than the slide-deck version, and better: one team’s least-loved work removed, a prototype-tested spec the AI team built from directly, a small number that verifiably moved, and a support team asking what the agent can take on next. Delivering an agent to the business doesn’t mean delivering a platform. It means delivering a working relationship, between a team and a system it helped design.

The method travels as-is to any team whose work passes the gate. Finance, running invoice exceptions: version one drafts the dispute-resolution notes. Sales ops, answering RFPs (the long questionnaires big customers send before buying): version one drafts the compliance boilerplate everyone hates. HR, fielding onboarding questions: version one drafts FAQ answers from already-resolved tickets. Legal intake, triaging contract requests: version one pre-fills the intake summary. Marketing, reporting: version one drafts the weekly performance commentary. In every case the pattern holds: the interviews find the toil, the toil defines version one, and the team pulls the agent up from there.

So here’s the move for this week. Pick one team. Ask their lead for one hour and ten recent cases. Open with ”what work would your team happily never do again?” and write down everything that comes after. You’ll leave the room holding most of a spec, and the team will walk you to the door.

Your next agent is three conversations away.

Before you spin up the first one, let me hand you a book. It’s called The Goal, a novel published in 1984 by Eliyahu Goldratt. I first read it in mechanical engineering school. I went back to it a few weeks ago, and what struck me is how completely it still applies, except the factory I kept picturing was one running on AI agents. Yes, the book itself is about a real factory: steel, machines, a loading dock, not a line of code in sight. Trust me anyway, because it is the most useful thing you’ll read about AI agents all month.

The hero is Alex Rogo, a plant manager with three months before headquarters shuts his plant for good. Alex’s one bright spot is a set of new robots that pushed productivity in one department up 36%. Everyone’s proud of them. Then Alex runs into Jonah, an old physics professor of his who now consults for factories, and Jonah asks three simple questions. Did the robots increase your sales? Did they reduce your inventory? Did they lower your operating expense? The answer to all three is no. The robots made one station faster and the business no healthier. The plant is still three months from closing.

Here’s the part that makes me smile. Goldratt was writing about big, clunky factory robots, the most exciting technology of his day. The technology has changed completely since then. The move people make with it hasn’t changed at all. A powerful new tool shows up, everyone rushes to point it at every task at once, and all that activity gets mistaken for real progress. The tool gets an upgrade every few years. The trap ships unchanged.

What actually held up is harder and more useful. The Goal forces one question on Alex over and over: what is a company even for? Not to run machines. Not to look modern. Not to automate for the sake of it. The book’s answer is blunt, and it’s still exactly right: a company exists to make money, and every robot, every process, and now every agent is worthless unless it moves that number. Swap “robot” for “agent,” and the book turns into a warning for anyone building AI agents inside a company: the factory urge to make more and make them faster is exactly the trap, and the only thing that counts is whether each one moves the business.

Agents are genuinely powerful, and building them is my day job. I find the use cases, work with developers to ship them for the company I work for, and build smaller ones for my own work on the side. But here’s the subtler truth the rush skips over: a powerful tool tends to create new problems while it solves old ones. An agent aimed at the wrong step doesn’t just fail to help. It quietly manufactures more work, a backlog of half-finished output that someone downstream now has to deal with. So the whole game is where you aim them, and that turns out to be the part everyone skips.

So here’s the bottom line, up front. A factory of agents is not the goal. The goal is throughput: the actual work that ships and moves the business. Pour agents onto a step that was never your slow point, and all you do is stack work in front of the one that actually is slow, which today is almost always a person reviewing and verifying what the agents produced. That slow step is your constraint. The winning move is the opposite of agents everywhere: find your one constraint, point your next agent at it, and pace everything else to its speed.

Here’s where this article goes, in four stops.

• The trap. I’ll show you why building agents by the dozen stalls, and the factory novel that saw it coming.

• The bottleneck. You’ll find the one step everything waits on, which is rarely where you’d guess.

• The principles. A few ideas from the book for feeding that step instead of burying it under more agents.

• The build. What aiming your next agent at the right step actually looks like, hands on keyboard.

By the end, you’ll have a simple way to look at any agent plan and say where the next agent actually belongs, plus the words to explain it to the people around you. It’s the difference between an agent strategy that looks busy and one that moves a number.

Hard hats on.

Why agents everywhere stalls

When the scoreboard is “agents deployed” or “workflows automated,” you optimize for the count, and the count is the easy thing to grow. Six new agents this quarter, twelve workflows touched, a demo that makes the room lean in. It feels like progress. But Goldratt put the whole trap in one line: ”activating a resource and utilizing a resource are not synonymous.” In plain English, keeping something busy is not the same as using it well. A factory machine stamping out parts nobody ordered is busy, but useless. An agent churning out drafts faster than anyone can check them is busy in exactly the same way. None of that activity is the business.

And the bill is already coming due. Gartner expects more than 40% of agentic AI projects to be canceled by the end of 2027, citing runaway cost and unclear business value. That is the same verdict Jonah, the book’s mentor, gave those factory robots, only now it lands on AI agents instead of machines.

So keep his three questions on a sticky note. They were written to judge a factory, and they barely change to judge an agent rollout:

• Did you ship more of what the business actually needs?

• Did the pile of half-finished, waiting-on-a-human work get smaller?

• Did your cost to run the place, the model bills and the hours spent steering, go down?

If you can’t say yes to at least one without making another worse, you didn’t buy an agent strategy. You bought robots.

Those questions come from the only three numbers Goldratt says a company should track. He names them ”throughput, inventory and operational expense,” and once they click you start seeing them everywhere.

• Throughput. The rate the system turns work into money out the door. For your team, it’s the work that actually ships and changes something for a customer, not the work that merely got generated.

• Inventory. Everything you’ve paid for that’s sitting around not yet sold. For you, it’s every draft, pull request, and ticket an agent produced that’s still waiting on a human to finish it.

• Operating expense. What you spend turning inventory into throughput, the cost of running the whole thing. For you, the model bills, the tools, and the hours your people spend steering and reviewing.

The robots moved none of the three. They just made a station that was never the slow part run faster, so half-finished parts piled up behind the slow station instead.

Here’s the deeper problem underneath, the one it takes Alex most of the book to see.

A factory isn’t a set of independent machines you keep individually busy. It’s a chain of steps that depend on each other, and what the whole chain produces is set by its slowest link, not by how hard the other links work. Alex’s plant was run the other way, every station scored on its own efficiency, every machine pushed to stay busy because an idle machine looked like waste. So the fast stations overproduced, and their output piled up as inventory in front of the slow ones. Every local chart looked healthy while the plant as a whole lost money. The robots didn’t cause that pattern. They poured fuel on it, supercharging a station that was never the constraint.

Sit with that, because your company is a factory too, even if nobody calls it one. Work comes in as a request and leaves as something shipped, and in between it moves down a line: someone drafts, someone reviews, someone approves, someone ships. Agents are the shiny new machines you just bolted onto the front of that line, and they make the drafting station absolutely roar. A line is only ever as fast as the station every job has to squeeze through, no matter how loud the front end gets. Build it the plant’s old way, with each agent scored on how busy it is, and you earn the plant’s old result.

That pile is the part most teams never name, and it’s worse than the factory kind. The robot’s parts were all good. This pile isn’t, because some of the output is wrong in confident, plausible ways, so the human downstream is inspecting for quality, not just clearing a backlog. You paid to create all of it, and none of it counts until someone finishes it and ships it. Which means the real question isn’t “how many agents.” It’s “where’s the slow station,” and the book finds the answer on a hiking trail.

Find your real bottleneck

Goldratt teaches the idea away from the factory, and it’s the scene that stuck with me for years. Halfway through the book, Alex, the plant manager, gets roped into leading a weekend hike for his son Dave’s scout troop, and he can’t keep the line together. The fast kids race ahead, a long gap opens in the middle, and the whole troop falls behind no matter how he reshuffles it. Watching that gap stretch and snap shut, he has the realization the rest of the book turns on. The troop is a chain, and a chain doesn’t travel at the speed of its quickest kid. It travels at the speed of its slowest. That kid is Herbie, a heavy boy lugging an overloaded pack at the very back. No amount of hustle from the kids up front buys back a minute, because sooner or later they all have to wait for Herbie.

So Alex does two things that feel backwards. He moves Herbie to the front, where nobody can be stuck behind someone slower. Then he opens Herbie’s pack and parcels the heavy gear out to the fast kids who have room to carry it. Herbie speeds up, and because Herbie now sets the pace, the whole troop speeds up with him. Pushing the fast kids was never the answer. Helping the slow one was.

That’s the whole theory of constraints in one walk. Every system has one step that sets the pace for all the others, and improving any step that isn’t that one is a mirage. Goldratt says it flatly: ”An hour lost at a bottleneck is an hour lost for the entire system. An hour saved at a non-bottleneck is a mirage.” Speed up the fast kids and the line just spreads out further. The only move that matters is finding Herbie and taking weight off Herbie.

So how do you find your Herbie? The book’s method is refreshingly low-tech: follow the piles. Alex doesn’t spot his slow machines in a report. He finds them by walking the factory floor and seeing the mountains of half-finished parts stacked up in front of them. A bottleneck always looks the same. There is a pile of work waiting in front of it, because the steps before it keep feeding it faster than it can keep up. And there are idle, starved steps behind it, because they have run out of work to do while everything is stuck upstream. So don’t theorize about where your slow step is. Go find where the work stacks up, and where people are stuck waiting on someone else.

Run that test on an AI-powered workflow today and the pile shows up in a predictable place. Generation got cheap and fast. A model writes the first draft, the first version of the code, the first pass of almost anything, and it costs almost nothing. When the cheap step gets ten times cheaper, it stops being the constraint, and the constraint moves to whatever it feeds. What it feeds is judgment. Someone has to read the output, decide whether it’s right, catch the confident mistake, and put their name on it. Review, verification, and the taste to know what’s worth shipping: that’s the heavy pack now.

The receipts back this up. When METR, an AI research lab, ran a randomized trial on experienced developers in 2025, the developers believed AI tools had made them 20% faster, and the measurements showed they’d actually gone 19% slower. Feeling productive and being productive came apart. Google Cloud’s 2025 DORA report found the matching pattern at the team level: AI lifts how much you produce, but it lowers delivery stability, because the acceleration exposes the weak step downstream instead of removing it. The time you save generating gets spent right back auditing, because nobody can review the volume the agents produce.

And this isn’t only an engineering story. Point agents at a marketing team and the one brand reviewer who keeps fifty draft variants on-voice becomes the slow step. Point them at support and the person who owns tone and policy is the gate a hundred suggested replies wait behind. Wherever a human still has to verify the work and own the result, that judgment is the new bottleneck, no matter how cheap the drafting got.

Now watch what the typical agent factory does: it pours more agents into generation, the step that was already fast. Every agent you add upstream of the constraint just hands it a taller pile. That’s not a throughput machine. It’s an inventory machine, and a very impressive one. Engineering teams already hit this exact wall with AI coding tools. Output jumped at first, then flattened out, because the extra code just piled up waiting for a human to review it.

Finding the bottleneck is the easy half. What you do once you’ve found it is where the throughput actually comes from, and the book gave me a handful of principles for that.

Work your real constraint

A quick word on what follows. These are principles from the book, not a rigid checklist to run top to bottom. The order helps, but take the ones that fit your situation and try them.

Map your flow first

The honest starting point isn’t “find the constraint.” You can’t find a thing you can’t see. The first move is to draw your actual flow, the path one real piece of work takes from idea to shipped.

Put one box per step in a row: request, spec, build, review, approve, ship. Then do the part everyone forgets. Between each pair of boxes, write down how long work usually sits there waiting to be picked up. In knowledge work, a task spends most of its life parked in a queue, not being worked on, which is exactly why value-stream mapping puts wait time on the board right next to work time. It’s also what Alex does when he walks his plant floor instead of reading reports: he follows the piles. Do it on a wall, in half an hour, with the people who actually do the work. They already know where it jams; the map just makes it impossible to deny.

Find your one real constraint

With the map and its wait times in front of you, the biggest pile of waiting work usually points straight at your bottleneck.

Right now, for most teams using AI, that slow step is review. Generation got cheap, so work piles up wherever a human still has to read it, judge it, and sign off. But usual is not the same as always, and reaching for the fashionable answer is its own trap. Look at your own map and check. For one team the slow step is writing the spec, because nothing gets built until the requirements are nailed down. For another it’s pulling the data, because one tricky query sits between every question and a real answer. For another it’s the security sign-off, or the final deploy, or any step that hangs on a rare approval.

Notice that none of those is a person. It’s a step in the process. The reviewer is not your bottleneck; the review step is. The difference matters, because you fix a step, you don’t fix a human. Find the step where work piles up, name it out loud, and write it down before you spend a dollar.

Get more out of it before you spend

Before you spend a dollar widening the slow step, get more out of the capacity it already has. This is the cheapest win there is, and the book makes it concrete.

In Alex’s plant, the bottleneck machines sat idle during lunch, because the operators all broke at the same time. So they staggered the breaks and the machines ran straight through: same machines, more output, zero dollars spent. They made two more moves like it. They moved quality inspection to before the bottleneck, so it never wasted a minute on a part that was going to be scrapped later. And they stopped feeding it parts for orders that weren’t even due, so every hour went to work that actually mattered.

Run those same three moves on a review step that has become your bottleneck:

• Never let it sit idle. Keep a small batch of work queued and ready in front of it, so the review step never stalls waiting for something to land.

• Never feed it junk. Put an AI agent in front of review to do a first pass. It waves through the obviously fine and flags the obviously broken, so the only things that reach a human are the genuine judgment calls.

• Never feed it work that doesn’t matter. Stop sending output to review for a feature that just got cut or a draft nobody asked for. Every review hour should go to something that’s actually shipping.

None of that costs money or a new hire. You’re not making the slow step bigger yet. You’re making sure not a minute of it is wasted.

Pace the rest of the work to it

This is the one almost everyone gets wrong, because it feels backwards. Once you know your slow step, you deliberately slow everything else down to match its pace.

The book calls the mechanism the drum, the buffer, and the rope. The slow step is the drum: it sets the beat for the whole line. A small buffer of work sits right in front of it so it never starves. And a “rope” holds back the start of the line, releasing new work only as fast as the slow step can actually take it. In the plant, Alex stops dumping raw material onto the floor just to keep people busy, and the mountains of half-finished inventory melt away.

For an AI workflow it comes down to one rule: don’t let your agents generate faster than your slow step can absorb. In practice you cap how much unreviewed work is allowed to pile up. Say the review step can clear about thirty pull requests a day. You set the limit at roughly a day’s worth in the queue. When the queue hits that cap, generation pauses, and the people who would have started new work go help clear the backlog instead.

If that sounds like a Kanban board with work-in-progress limits, that is exactly where the idea came from. The rule of thumb is blunt, and it works: stop starting, and start finishing.

Then widen it, and aim your next agent there

Only now, after you’ve squeezed the free gains out of the slow step, do you actually spend on it. And the spending goes to the slow step, never to the part that’s already fast.

When Alex’s team ran out of no-cost tricks, they got their hands dirty. They hauled two old machines out of storage to take some load off the bottleneck, and they sent the overflow to an outside vendor. Not elegant, but it widened the exact step that was holding the whole plant back.

Your version is the same idea, aimed at your slow step. If review is the bottleneck, the most valuable thing you can build is not another generator. It is something that widens review itself:

• A first-pass review agent that handles the easy 80% of changes, so a human only has to look at the hard 20%.

• An eval (a quick automated check against examples you trust) that clears the routine cases, so a human only sees the ones it flags.

If your slow step is writing the spec instead, you build the agent that drafts the first version of the spec from the request, so the work starts at “edit this” rather than “write this from a blank page.” The agent worth building is the one that widens your slowest step. Every other agent just makes a bigger pile.

Expect the constraint to move

Here’s the part that catches people off guard. The moment you widen one slow step, the bottleneck doesn’t vanish. It jumps somewhere else. Fix review, and the slow step becomes deployment, or one overloaded approval, or a sign-off that was invisible only because review was hiding it.

In the book, once Alex fixes his machines, the constraint leaves the factory floor entirely. It moves first to the market, and then, to everyone’s surprise, to the company’s own outdated rules, policies that made sense years before and were now quietly throttling everything. Goldratt’s word for that trap is inertia: pouring effort into yesterday’s bottleneck long after it stopped being the problem. Half the time the real constraint isn’t a machine or even a step. It’s a rule nobody has revisited, like “every change needs three approvals.”

So this is never a one-time project. Put a recurring half-hour on the calendar to re-walk your flow map and ask one question: where does the work pile up now? The answer will have moved, and so should your next fix and your next agent.

That is what replaces “twenty workflows, twenty agents.” Not a sprawl you launch all at once, but one constraint found, fed, and widened, then the next one. Fewer agents, pointed at better places, and a number that finally moves.

Build the one that matters

So what does building that agent actually look like? Smaller than you’d think. Say your slow step is review. You don’t need a platform or a six-month program. In Claude Code, the smallest version is a subagent: a short markdown file (just a plain text file) in your project that gives an agent a name, a set of tools that can only read and never change anything, and a system prompt (a written instruction) spelling out exactly what to check on a first pass. It reads every change, flags the handful that need a person, and waves the rest through. Your reviewer stops drowning in trivial diffs and spends their scarce attention on the calls that actually need judgment. The same shape works far from code: a subagent that drafts the first version of a spec, or one that writes the first cut of the gnarly query for your analyst to correct.

The part people skip is what the agent checks against. The teams getting real mileage out of agents right now aren’t the ones with the best model, they’re the ones with the best evals. An eval is just a test with an answer key. For a first-pass review agent, the answer key is a handful of past changes where you already know which ones had real problems. You run the agent over them and see where it disagrees with you. Does it catch the genuinely broken ones and wave the clean ones through? You tighten its instructions until its calls line up with yours, and only then do you trust it on live work. Anthropic’s own write-up on evals is a hype-free place to start. An agent at your constraint with no eval behind it is just a faster way to be confidently wrong.

You don’t have to do any of this alone, and you shouldn’t. The people doing the work already know where it jams. Put the flow map on a wall and let your engineers point at the step that always stalls; ask the analyst what they wait on; ask the designer what sits in their queue. The framing that tends to land, whether you’re explaining it to the team or the people who hold the budget, is short: we don’t have an agent shortage, we have a constraint we haven’t named. Once everyone is looking at the same slow step, the argument about how many agents to build mostly answers itself.

One last thing, and it’s the part I find genuinely fun. The specific tools in this article will age fast. A year from now, generation will be cheaper, the agents sharper, and your slow step sitting somewhere new. None of that changes the move. Draw the flow, find the one step everything waits on, feed it, and aim your next build straight at it. That is why a novel about a steel plant still reads like a manual for building AI agents. The machines changed. The mistake didn’t, and neither did the fix.

The team learned a new tool. They got a small productivity bump. The next jump never came. And now, sitting in the all-hands, you can feel the air going out of the room. You sold this to your CTO as a step-change in time to market. It’s looking more like an autocomplete upgrade.

I want you to spend 35 minutes on a video before your team decides this is as good as it gets. It’s a talk from Szczepan Faber and Mike Nakhimovich, two engineers on Airbnb’s developer platform team. They explain how their org went from zero to 64% of PRs materialized through agentic coding in nine months. No marketing slides, no vendor pitch, two practitioners with diagrams and the receipts.

I know. Another AI talk. But this one is different. Most agentic engineering announcements recently read like product trailers. Stripe wrote up their Minions, one-shot end-to-end coding agents that ship code from issue to PR. Spotify wrote up their background coding agent you can fire off from Slack. Both are real engineering teams shipping real systems. But the writing is announcement-shaped, and you walk away knowing what they built, not how to bring your own team along.

This one is two practitioners walking through their methodology, their mistakes, the proficiency model they built for engineers, and the org structure that scaled it. With actual numbers. With actual diagrams. Practitioner depth, not announcement depth.

A disclaimer before you watch. The talk is from October 2025, which is roughly prehistory in AI time. A few things they mention have already evolved. Their internal stack is theirs, not yours. You shouldn’t copy what they did. Since then, Airbnb’s CEO has publicly stated AI writes nearly 60% of new code at Airbnb (TechCrunch coverage of the May 2026 earnings call). The platform team’s October talk is still the operating manual. The May number is the receipt. The definitions, principles, and learnings travel.

This post is the distillation. A working definition of agentic coding so you can say it cleanly to your team. The principles their team operated by. The mistakes they fixed. And a small grid you can use to plot your tech function and pick the next project to push for. Watch the video first if you have the time. If you don’t, this post stands on its own.

• The plateau. I’ll show you why AI coding tools alone gave your team a small bump and stalled.

• The definition. I’ll give you a clean working definition of agentic coding to quote to your team.

• The principles. I’ll walk you through the org moves Airbnb’s team built around the tools.

• The learnings. I’ll point out the post-mortems you’d repeat without them.

• The grid. I’ll hand you a four-cell grid to plot your team and pick the next move.

By the end, you’ll have the working definition, the principles, the learnings, and a four-cell grid plotted with your tech team’s location and a next-move sentence. You’ll also walk away with a 7-category readiness checklist you can paste into a Google Doc and fill in row by row. You walk away with the conversation to have with your CTO on Monday, and a way to reframe AI tool ROI as agentic coding maturity.

Read the checklist

Let’s pin it down.

Name the plateau

Every team is hitting the same wall right now. The AI coding tool went in, the productivity bump was real, and then nothing. The team starts wondering out loud whether they bought the future or just a faster autocomplete. Naming the wall is the first move, because the wall has a shape and the shape tells you what comes next.

Airbnb’s platform team measured it. They ran the velocity numbers across three engineer cohorts. Non-AI users moved at the team’s baseline pace. AI-aided non-agentic users (the autocomplete-and-chat crowd) shipped PRs at +27% velocity. Full agentic users shipped at +38%. The eleven-point gap between tools-only and agentic is the ceiling teams hit when they stop at the tool. That gap is not a tool failure. It’s the natural plateau most teams sit on right now.

One honest caveat before we keep going. The wider data is messier than Airbnb’s three bars suggest. Google Cloud’s 2025 DORA report found individual output rises while team delivery stays flat. METR’s July 2025 RCT found experienced open-source devs were 19% slower with AI tools, while they believed they were 20% faster. Airbnb’s three bars are one of the cleanest team-level signals we have, not the universal shape. Treat them as the strongest existing public benchmark, not the law of nature.

Three things sit inside that eleven-point gap.

• The first is practice maturity: how engineers actually use the tool day to day, whether they auto-approve, whether they review every diff, whether they push back when the agent is confidently wrong.

• The second is org infrastructure: the paved paths, the internal champions, the sandboxed environments, the MCP servers that wire internal context into the loop.

• The third is leadership conviction: whether the org frames agentic coding as a step-change worth funding or a productivity tweak worth tolerating.

Each one is a stop on a journey your team has been walking since the IDE era. None of them gets fixed by buying a better tool.

Your team is at Stage 2. The plateau is not a sign you picked the wrong tool. It’s a sign the next investment is in the things around the tool. That reframe is the one you carry into the leadership conversation. You bought the engine. Now you’re missing the chassis.

Naming the gap is the first move. Naming what agentic coding actually is comes next. That’s the language you’ll use upstream.

Define what agentic coding is

Part of why your team is stuck in the plateau is that they’re using one word for two different things. Autocomplete is not agentic coding. Copy-paste from ChatGPT is not agentic coding. A single-shot retrieval over your codebase is not agentic coding. None of those is the thing Airbnb measured at +38%. Clean the definition first, then have the budget conversation honestly.

An engineer writes a short spec or prompt. That spec kicks off an autonomous loop. Inside the loop, an LLM is called several times. Tools and MCPs are called several times. The loop loads config files (AGENTS.md, CLAUDE.md, cursor rules) and runs against guardrails. Out the other side comes a materialized code change. A human reviews every line before it merges. The thing that makes it agentic is the autonomy in the middle, not the input or the output.

The engineer doesn’t know exactly what’s happening in there. Szczepan put it bluntly:

An engineer who is leveraging an agentic tool, you don’t know how many [calls happen]. You start your agentic session, but you don’t know exactly what prompts the agentic tool will use when it calls LLMs. You don’t know exactly how many times it’s going to call LLMs. You don’t know exactly what tools it’s going to use.

That uncertainty is the point. The loop owns the path. You own the spec and the review.

Three things people call agentic coding that aren’t. Autocomplete in your IDE is not agentic coding (one-shot, no tool calls). Pasting code into a chat window and copying the answer back is not agentic coding (no loop, no autonomy). A single-shot RAG search returning a snippet is not agentic coding (no decision-making, no tool composition). Szczepan made the same point.

That’s not where the magic happens. This is not this paradigm shifting change where a human engineer can steer multiple agentic sessions and materialize code changes.

The +27% in the velocity chart is what you get from those. The +38% is what you get from the loop.

The cleanest way to explain the shape to leadership is borrowed from pair programming. You probably have engineers who’ve done it. In classic pair programming, the driver holds the keyboard and is tactical. The navigator looks at the whole forest and is strategic. They alternate roles through the day. Agentic coding flips that into something stable.

The engineer becomes the permanent navigator. The agent becomes the permanent driver. The engineer steers the session with chat, click, voice, or image. The agent materializes the code. Szczepan said it directly:

Agentic coding is where a developer becomes this permanent navigator and it’s steering that agentic session, and an agent or your tool is materializing code changes.

The leadership analogy travels because every CTO over the age of 35 has done pair programming.

The technical version of the analogy is sharper. What the navigator is actually doing has a name. Andrej Karpathy coined “context engineering” in June 2025 as the successor to prompt engineering. He called it ”the delicate art and science of filling the context window with just the right information for the next step.” That’s the navigator’s job. Pair programming is the analogy that lands. Context engineering is the technical second skin.

Now that you can name it, here’s what Airbnb’s team learned about making it actually pay off.

Borrow these principles

The tool was the easy part. The principles are the actual work. None of these cost engineering headcount. All of them are decisions you can push for from where you sit. Here are five that travel from Airbnb’s context to almost any other org.

Start with a bold vision, work backwards

Airbnb’s platform team opened their January 2025 leadership pitch with a single sentence:

AI-powered, agentic coding will eventually replace traditional programming methods. How do we work backwards from that end state?

Then they actually worked backwards. They didn’t ask “what experiment do we run next quarter?” They asked “what does Step 10 look like?” and walked backwards to Step 1. Szczepan was direct about why.

If you only think about incremental improvements, it’s kind of hard to make a big change.

The move is the same for you. Put the destination on a slide before you put the roadmap on a slide.

Use an analogy with leadership

The pair-programming analogy didn’t land at Airbnb because it was clever. It landed because leadership had done pair programming.

The way we’ve explained that to our leadership at start of 2025, we’ve used pair programming analogy for this.

Szczepan said. Pick an analogy your leadership already trusts. Map agentic coding into it. Leadership conversations don’t move on facts. They move on framings your audience already believes. If they came from a team that used trunk-based development, talk in those terms. If they came from agile, talk in those terms.

Measure holistically, never personally

The default trap is to pick one number. Velocity goes up. Ship it. Sentiment goes down. Drop it. Airbnb uses three lenses together, never individually: developer sentiment from quarterly surveys, tool adoption across surfaces and PRs, and engineering output like PR velocity aggregated across the team.

The philosophy slide from their talk is forwardable on its own.

Our aim is to learn how AI impacts developer experience and software quality so we can improve both. Not to micromanage or rank individual performance. Focus on insight, not oversight.

This is the principle to use when leadership asks for ROI metrics that don’t exist yet. The answer is not one metric. The answer is three, read together, looked at the team level. If you want a public framework they’ve heard of, Airbnb’s three lenses overlap with what DX calls the Core 4 (Noda and Tacho, January 2025): Speed, Effectiveness, Quality, Impact. Same principle: never one number, never per-individual.

Meet developers where they are

Sixty percent of Airbnb’s engineers prefer remote workspaces. Forty percent prefer local. IDE lovers stayed in IntelliJ. CLI lovers stayed in Vim. Mike tried the migration play once.

We considered, maybe for a few days, grabbing all of our IntelliJ users and moving them over to VS Code. That lasted for a few days before I had to fear for my life that someone would come after me.

He said. The principle: build parity across surfaces, not migrations. Or as Mike put it elsewhere:

We don’t have one typical engineer at Airbnb.

You probably don’t either.

Invest in the org around the tool

This is the biggest gap most teams skip. The tool ships. The infra doesn’t. Nobody builds a champions network, no one paves the install path, no one wires internal context into the agent. Airbnb invested in all three. A champions network of thirty cross-functional engineers (not the core team of four). Paved paths with single-command install, automagic auth, and project templates. Over a dozen internal MCP servers connecting agents to internal systems. Szczepan said the line out loud:

Your agentic tool is not enough. You will have to have a lot of tech around it.

If you want a second public example to back this up, Shopify built the same pattern at roughly 10x the scale. CEO Tobi Lütke’s leaked AI memo became an internal movement. The eng org built an internal LLM proxy, standardized MCP servers wired to their wiki and PM tool and data warehouse, and bi-annual performance reviews that explicitly evaluate “AI-reflexivity.” Two-anchor evidence is sturdier than one-anchor evidence. (First Round wrote it up; Bessemer wrote up the engineering playbook.) This is the line item to fight for in your next budget cycle.

Add three engineering principles

Three more principles, briefly. First, wrap open source instead of building your own orchestrator. The rule is “don’t build a full orchestrator from scratch,” not “wrap whatever’s hosted externally.” Even Airbnb wraps their own airchat-cli, not raw OpenAI or Anthropic. In regulated orgs (finance, healthcare, EU-AI-Act-bound), routing internal code through a third-party CLI is a procurement and legal fight, not a one-line decision. Build the thin wrapper layer locally if compliance demands it.

Second, bring context to agents through tool calling, not by training models. ”Tool calling is all you need,” Mike said. He’s riffing on the 2017 Transformer paper title. His point: the LLM doesn’t need bespoke training on your codebase. It just needs the right tools available. Whether tool calling alone is enough versus the richer MCP pattern is contested in 2026. Anthropic’s November 2025 “Code execution with MCP” pushes more sophisticated patterns. Mike’s line still holds as the directional claim.

Third, never drop human review on the way to the PR. Szczepan’s line is the one to forward to anyone tempted to “ship faster”:

I don’t trust the AI. I trust my skill in prompting and using the tools.

The agentic loop ends at the diff. Every line is reviewed by a human before merge, especially the test code.

Worth knowing what to avoid too. Here’s what Airbnb’s team got wrong before getting it right.

Avoid these mistakes

Even Airbnb’s platform team got things wrong before they got them right. They have four engineers, a dedicated mandate from leadership, and the resources of a 7,000-engineer parent org. They still hit these. Worth borrowing the post-mortems before you make them yourself.

Don’t bet on a slow curve

Airbnb’s platform team predicted at the start of 2025 that materializing code through agentic sessions would hit 20% to 40% of PRs by the end of the year. By the time they gave their talk in October 2025, they were at 64%.

We were sandbagging them a bit too much, I think, because we were not ambitious enough. It’s going faster.

Szczepan said. If the people building the platform underestimated their own curve by a factor of two, the “wait and see” position upstream is probably already late.

There’s a name for that position. Geoffrey Moore called it “early majority pragmatism” in Crossing the Chasm (1991). Pragmatists wait for proof. Visionaries (early adopters) ship before the proof exists. Recent commentary suggests consumer GenAI has crossed the chasm; agentic enterprise systems are mid-crossing. The frame gives you a named, well-known concept to drop in the 1:1: ”You’re in classic early majority position. The data suggests the chasm has already been crossed.”

Don’t build your own orchestrator, wrap one

Mike’s team spent months building an internal agentic orchestrator. Registered agents. Planning chained to coding chained to validation. It was elegant. It never shipped.

AI moves way too fast. We were still planning, figuring out what we were going to do that existed in the market, and the market now is five steps ahead of us. It wasn’t going to work.

He said. They walked away from the orchestrator and pivoted to a thin shim over airchat-cli.

Can’t beat them, join them. We pivoted and instead delegated to our Airchat CLI.

The rule for any size team: if open source is iterating faster than you can ship, wrap it.

Slow down before you auto-approve

There’s a temptation, once an engineer gets comfortable with agentic coding, to skip straight to auto-approve. Don’t. Three things happen when engineers skip the manual-approval stage too fast. PRs get huge. The engineer doesn’t actually read every line of the diff. Test code drops in quality first, because test code gets reviewed less carefully than production code. Airbnb’s Level 3 (auto-approve plus diff review) is a learned skill, not a default. Szczepan said it cleanly:

An engineer may create very large PRs that they don’t review fully. They don’t actually understand and review every single line of code. They may not review test code as eagerly as the production code.

The fix, when you’re in the loop, is to push back on the tool when it’s confidently wrong. Szczepan walked through it.

You don’t trust it. If something feels suspicious, you ask it: “Why did you do this in line 120?” And after a couple of rounds, you get to this satisfying moment where the agentic tool gives up and says, “Oh yeah, you’re right. I’m sorry. I was wrong. Let’s fix it.”

That’s not friction. That’s the practice.

Close the skill gap fast

This one comes from Mike’s lived experience, not a number on a slide. Mike is a college dropout who admits he doesn’t know Git from the command line. With agents, he ships across multiple languages.

I’m one of those non-believers. This is all going to be slop. This is all going to be vibe coding. This is going to be worse than what I can write by hand. And when I was using GPT 3.5, that was true. I personally got lapped by agentic coding.

He said. In Mike’s team’s setup, the gap between agentic-fluent and non-agentic engineers widened faster than anyone predicted.

Definition, principles, learnings. Now plot where you actually are.

Plot your team on the grid

Reading the principles and learnings is one thing. Doing something with them this quarter is another. The grid below is the diagnostic. It’s not a model of who you are. It’s a map of where to invest next.

The grid has two axes. The vertical axis is median engineer fluency. The horizontal axis is org infrastructure. For the vertical axis, plot the median engineer on your team, not your top performer. Airbnb published a four-level fluency ladder that travels well.

• Level 1 is first steps. Autocomplete, occasional chat, copy-paste from ChatGPT.

• Level 2 is agentic learner. Uses an agentic tool with manual approval; improving prompts and configs.

• Level 3 is advanced practitioner. Auto-approves, reviews diffs, runs parallel sessions across multiple workspaces.

• Level 4 is pioneer. Builds custom MCPs, leads team transformation.

One caveat that matters: Level 4 (pioneer) is an organizational role, not just a personal skill. It requires platform-team mandate and org permission, not only individual fluency. Pioneers are usually the output of being in Multiplier mode (the top-right cell of the grid), not a prerequisite.

For the horizontal axis, ask plainly: is there a champions network, paved paths, internal MCP servers, sandboxed environments? Or is everyone on their own? You’ll know within five minutes.

Each cell has a name, a description, and a next-move sentence for you.

• Tourist phase (low fluency, low infrastructure). Everyone copy-pastes from ChatGPT. No champions, no shared standards, no one paving anything.

  • Next move: push for one champion in your tech team and one paved tool. Don’t try to do five things at once. Sequencing matters more here than ambition.

• Hero phase (high fluency, low infrastructure). One or two senior engineers shipping miracles with agentic coding. The rest of the team hasn’t moved. The seniors burn out, leave, or get poached.

  • Next move: make the cost-of-burnout case to leadership. The hero’s productivity doesn’t scale without the infra.

• Onboarded but underused (low fluency, high infrastructure). The tools exist, the infra is paved, but engagement is flat. Usually a culture gap, not a skills gap.

  • Next move: push for three Shopify-style moves. Visible leader use (a senior engineering leader ships a PR with an agent during the next all-hands). A paved-path simplification audit (every extra click between an engineer and an agent kills adoption). And AI-fluency woven into performance reviews, not as a productivity metric but as an expectation. Not more tools.

• Multiplier mode (high fluency, high infrastructure). This is where Airbnb’s 64% PR adoption lives. Power users running five parallel sessions, MCP ecosystem growing organically, the working group of champions teaching everyone else.

  • Next move: defend the budget, bring product into agentic workflows (PMs prototyping with the same loop), start measuring time-to-market shifts.

The 2x2 is the headline. The roadmap lives in a row-by-row diagnostic with seven categories: direction and adoption posture, measurement, tooling stack, internal context, engineer skill journey, org structure, and quality bar. Each category has two or three rows. Each row asks one question. ”What does your team actually have here?” Fill in the empties. Count them. Pick one row to push for this quarter. Not five.

Read the checklist

Paste it into a Google Doc, fill in the “Where you are” column row by row, and bring it to your next 1:1. The empties are your roadmap.

One honest caveat. This grid is a snapshot of one company’s landing point and what travels from it. Your context is different. Your org is smaller. The talk is six months old in AI time. Use the grid as a starting point, not a verdict. Use the checklist as a conversation starter, not a scorecard. Revisit it in 90 days.

Once you know where you are, the upstream conversation changes.

Bring it to your CTO on Monday

Drop this sentence in the 1:1. Replace the bracketed bits with what you plotted.

Our AI tool was step 1. On the grid, we’re in [cell name]. On the checklist, we have empties in [row X], [row Y], and [row Z]. The next investment we should make is [the row that unblocks the others]. Here’s why it matters for time to market.

That’s the line that reframes the conversation. The unanswerable version is “what’s the ROI on our AI coding tool?” The answerable version is “where are we on agentic coding maturity, and what’s the next investment?” The grid gives the headline. The checklist gives the specifics. Your CTO can disagree with the cell. She can argue about the next investment. She can’t pretend you haven’t done the work.

Four moves for the week, in order.

1. Watch the 35-minute talk. You can do this over a coffee.

2. Plot your tech function on the grid. Fifteen minutes.

3. Read the checklist, copy it, fill the “Where you are” column row by row. Honest beats complete. Thirty minutes.

4. Schedule a 30-minute 1:1 with your CTO. Send the filled checklist before the meeting. Pick the one row you want her to fund this quarter.

The principles travel. The specifics don’t. Your context isn’t theirs. Trust your read over the diagnostic. Reply and tell me where you plotted your team!

I said yes, let me build one. Three weeks later, I had three Claude Code skills running. One drafted AI cards from real team questions. One reviewed every card. One shipped the approved cards to Confluence. I felt proud of what I had built. I was already planning the next skill, the one that would check every card each month for stale answers.

Then I sat down and did the math. How many cards would I need to answer every AI question anyone might ask? Roughly 10 approved AI tools. About 10 common questions for each tool. About 10 less-common questions for each. That comes to one thousand cards. Cards I would have to update every time a vendor changed a page or legal added a rule. My industrial engineering professors taught us one rule above all others: never produce stock you don’t have an order for. I had spent three weeks producing stock. So I stopped.

The second time, I was chasing the same outcome. Answers about AI tools, grounded in our company’s rules, available to anyone who asks. The first build would have taken me months. The second took one focused afternoon in Cowork. Same use case. Two very different ways to apply AI. AI doesn’t make you faster on its own. The direction you pick with AI does.

There are three acts to this. Each one builds on the last. I’ll walk you through all three.

• The story. Why every AI playbook ends up unused. The instinct to build one is universal, but the math breaks every time. I’ll show you the shape that worked for me instead.

• The build. Walk through the build with me in Cowork. The whole thing takes one focused afternoon, end to end. By the time you finish reading, your skill is shipped and ready for your team.

• The pattern. The AI helper is one example of a reusable shape. Swap the context page and you have a different helper. OKRs, feedback, brand voice, onboarding: all from the same skill scaffold.

By the end, you will have built an AI helper your team uses. Teammates ask it first, before they DM you. The routine questions get answered without you. The complex ones still come to you, but only after the helper has handled what it can. The example I walk through is the one I built: an AI helper for ”can I use this AI tool, and how?” questions in our specific context. The same shape works for any routine team question that depends on context, like OKRs, feedback, brand voice, or onboarding. I’m sharing the GitHub repo too, so you can clone it and start from a working example: github.com/adamfaik/ai-ask.

One prerequisite to follow along. You need Claude Cowork, which is the agentic surface in the Claude desktop app. It comes with any paid Claude plan: Pro, Team, Max, or Enterprise. The build won’t work in plain chat mode.

Let’s jump into it!

The trap, the prism, the use case

The playbook trap

The playbook instinct feels right at first. Cards are the right unit. Each card answers one question. Each card lives in Confluence where everyone can find it. I just had to write the cards. That’s where Claude Code skills came in. Why write a thousand cards by hand when I could build a small factory to produce them?

So I built three skills, each with a clear job. Each skill produced a clear output for the next one in line. The whole thing was a small assembly line for documentation cards. Together they would keep the documentation always fresh, always on-brand, always in the right place.

The first skill was the drafter. I would paste a real team question into it, like ”Can I use Cursor on my Mac?” The drafter would research the answer. It would search the web for the vendor docs, the official install steps, recent policy updates. Then it would produce a draft card with a consistent structure: a clear title, the answer, the contact for follow-up, links to the official documentation. The output was a complete first draft, ready for review.

The second skill was the reviewer. It took the drafter’s output and ran it against a quality checklist. Is the answer specific? Is the contact named? Is the link still live? Are there any AI-policy red flags? If the checklist passed, the card moved on. If it failed, the reviewer suggested fixes and sent the card back to me. The reviewer was the gatekeeper that kept the factory from shipping bad cards.

The third skill was the shipper. It took an approved card and prepared it for publication. It formatted the markdown for Confluence. It generated the right labels by tool, by topic, by audience. It picked the right parent page in our information architecture. Then it published. The shipper turned a clean draft into a live page my teammates could actually find.

I was proud of the architecture. The three skills handed cards to each other in a clean line. Real questions came in on one side. Polished, labeled, published cards came out the other. I was already planning the fourth skill, the one that would crawl every card each month and flag the ones that had gone stale. The factory would run itself.

Then I sat down and did the math. How many cards would I need to answer every AI question the company might ask? About 10 approved tools. About 10 common questions for each tool. About 10 less-common questions for each. That comes to one thousand cards. A thousand cards I would have to update every time a vendor changed a page or legal added a rule.

My industrial engineering professors taught us one rule above all others. Never produce stock you don’t have an order for. Toyota built an empire on this idea. Don’t make the part until someone needs it. The factory I was building would produce stock the team would never visit. So I stopped.

What I built instead

What did I actually need? An LLM already knows a lot. It has read the Internet many times during training. It can search the web for recent information. For most questions, it can already give a reasonable answer on its own.

If a teammate asks ”how do I install Cursor on my Mac?”, a fresh Claude chat can tell them. It will give the standard install steps from cursor.com. The answer will be correct. But the answer will be generic. It will not know that at my company, Cursor requires SSO. It will not know about the Confluence MCP integration. It will not know that autocomplete is disabled on certain file paths.

The missing piece was a layer of context specific to the company. What’s allowed and what isn’t. Who to contact when something is unclear. What gotchas surprise new joiners. The playbook was trying to be that layer, but it tried to do it as a stack of cards. I needed something different.

So I built one skill instead. I called it /ai-ask. The skill reads one document on every invocation. That document holds the company-specific layer of context. One source of truth. Every question pulls from it. Every answer comes out grounded in our reality.

And that’s when it clicked. I now see this skill as a prism, like the one on the cover of Pink Floyd’s Dark Side of the Moon. Picture it. White light enters one side. The prism refracts it. A rainbow comes out the other.

In our case, the white light is what the LLM already knows. The Internet, vaguely remembered from training, plus what the model can pull from a web search. The prism is the company-context page. The rainbow is the answer: what the LLM knows in general, refracted through what is true at my company in particular.

This is the same logic as just-in-time manufacturing. Don’t produce stock you don’t have an order for. Produce the answer when someone asks the question. The context page is the only thing you maintain. Every answer regenerates from it. You don’t keep a stack of stale cards.

The skill re-reads the context page on every invocation. That’s not a side effect. It’s the whole design. Memory from earlier in the conversation can be stale. The page may have been updated. Stale answers about policy are worse than no answer at all.

The use case: an AI helper

So what does this look like in practice? Teammates type /ai-ask followed by their question instead of DM’ing you. The questions are the kind you have already answered a hundred times: ”Can I use ChatGPT free for this customer feedback?”, ”Is Cursor approved on my Mac?”, ”What model should I use for code review?” The skill answers all of them in the same shape: company context first, general guidance second.

The answer they get is grounded in your team’s rules. The right contact is named. A link to the official documentation is at the end. They get a real answer, fast, without waiting on you. What does this deliver for you? Your DMs go down. The team self-serves answers grounded in your rules. You get your focus back.

I built mine for AI questions because that’s the routine question I kept getting. You can see the live code in the public repo at github.com/adamfaik/ai-ask. But the same pattern works for any routine team question that depends on company-specific context. Swap the context page for a different document, and you have a different team helper.

OKR alignment (does this initiative line up with our quarterly goals?). Feedback reviews (is this the way our team gives feedback?). Brand voice checking (is this on-brand for us?). New-joiner onboarding (who do I ask about X?). You’re learning a pattern, not just an AI Q&A bot.

Open Cowork.

Build the helper in Cowork

One focused afternoon, end-to-end. Here are the seven steps you’ll go through. Glance at the map before we dive in.

Before Step 1, a quick setup. Download the Claude desktop app from claude.com/download if you don’t have it yet. Sign in with your paid Claude account. Confirm you see three tabs at the top: Chat, Cowork, and Code. We will spend most of our time in Cowork. Cowork is the agentic surface that can read and write files, run skills, and operate on a folder. The last step uses Chat to test the finished skill.

Three things to keep in mind as you go. The prism page is where the real work lives. Scaffolding is fast. Every meaningful step is one prompt to Cowork, not manual file editing.

Step 1: Start the skill with skill-creator

Open Cowork. Below the chat input, click Work in a project → Choose a folder. The native folder picker opens. Navigate to wherever you keep code projects. Mine is ~/Documents/GitHub/. Click New Folder, name it ai-ask, and select it. One folder, three jobs: source of truth for editing, install target for Cowork, repo root for the GitHub push later.

Now use Anthropic’s skill-creator to scaffold the skill. Don’t write the SKILL.md file by hand. skill-creator interviews you about the skill you want and generates the file structure for you. Type /skill-cr in the chat input and skill-creator should appear in the autocomplete dropdown. If it isn’t there, install it from Customize → Skills → + → Browse skills → search “skill-creator” → Install.

One detail matters before you send the prompt. Switch the model to Opus 4.7 in the chat input bar. Cowork’s model picker is session-locked. The choice you make now applies to the whole conversation. Invest the smart model on the work that defines a system once. Use the efficient model on the daily work that runs that system every day. Building a skill is the first kind. Using it every Monday is the second.

Invoke /skill-creator and either paste or dictate this paragraph as the opening brief. I usually open SuperWhisper and just talk the prompt out loud, as if I were explaining what I need to a teammate, a friend, or an intern I’m asking for help. Either way, the content is the same:

skill-creator reads the brief and asks a few clarifying questions. Your exact questions might be slightly different from mine. Each session is a little unpredictable, that’s just how LLMs work. The principle is the same: it fills in the gaps before it starts writing files.

The first asks what should go in company-context.md. Pick ”I’ll provide content myself.” You’ll fill the page in the next step. Letting skill-creator interview you for the content here would slow things down for no reason.

The second asks how rigorous evaluation should be. Pick ”Quick draft only.” We don’t need to test this skill in an industrialized way. We will just try a few real questions on it ourselves in the next step. A quick draft is enough for that.

The third asks where the final skill should live. Pick ”Both.” You need both formats for two different jobs. The folder is what makes editing and git init-ing work. The .skill package is what Cowork installs in Step 4. Without the package, you cannot install the skill the right way.

After the third answer, skill-creator lays out its plan and starts writing files. The completion summary at the end is worth reading. It surfaces design choices the skill author would otherwise miss: the ”Read it fresh every time” anti-staleness rule, the ”Don’t invent rules” guardrail, the two-part answer format with exact headings. Once it finishes, the project folder holds SKILL.md and an empty company-context.md.

The scaffold is done. Now we read what Cowork actually made.

Step 2: Read what skill-creator created

Before going further, take a minute to read what skill-creator actually generated. Open SKILL.md in your text editor or in Cowork’s preview pane. Make sure the skill matches what you asked for.

This file is what Claude reads every time someone calls /ai-ask. It has YAML frontmatter at the top (name and description), then a body that defines: when the skill should trigger, what input it takes, what shape the answer must follow, and the rule that it must always read company-context.md first.

If something is missing or off, ask skill-creator in the same chat to update it. The skill is just text. You can iterate. Don’t move on until you trust the SKILL.md.

For reference, you can see the version I ended up with on GitHub: github.com/adamfaik/ai-ask/blob/main/SKILL.md. Your version will look similar but tuned to your own brief.

With the skill itself in shape, we move to the most important step: writing the company context.

Step 3: Write the company context page

This is the most important step. Once you define your company’s rules clearly, the skill will work or not based on what’s in this page. The page IS the prism. Take your time here.

skill-creator already wrote a 7-section template in company-context.md with [FILL IN] placeholders. Open the file in Cowork’s preview pane to see it. The template has seven sections: approved tools, forbidden tools, data handling, licensing, approval paths, preferred providers, open questions. The scaffolding is there. The content is yours.

You can fill this page however you like. You could have a back-and-forth discussion with Cowork, building the page one section at a time. You could draft it yourself in your text editor and paste it in. You could send one structural prompt and let Cowork generate the whole thing at once. There is no one right way. The page is yours, and the process should match how you think.

For this article, I happened to do it in one prompt. That worked because I had already thought through what each section needed. Your first version will probably take longer because you are figuring out the rules as you write them. That is the point of this step. Expect to spend real time here.

Here is the prompt I used, as an example of what one approach looks like. The prompt uses a fictional company called Acme because my real company’s content is private. At your real company, swap the proper nouns. The structure stays the same.

Cowork executes both passes in one turn. It verifies the output: both files at the project root, byte counts match, the intro blockquote is preserved, all section headings are in place. What comes out is a ten-section company context page ready to be tested.

For reference, you can see the full version on GitHub: github.com/adamfaik/ai-ask/blob/main/company-context.md. It will give you a sense of what your version could look like once it’s filled in.

The prism page is human-editable markdown. Your team can update it without touching the skill code. Store the prompt, not the result. The result regenerates every time the page changes.

Now install it. There’s a trap.

Step 4: Save the skill as a .skill file

Cowork’s Save Skill button can install your skill in two different formats. The format you pick matters more than it looks.

• If you save the skill as a .md file, only SKILL.md gets installed in your Personal Skills. Your company-context.md, the prism, stays on disk in the project folder. The skill is now installed, but without the file it is instructed to read on every invocation. The skill is there. The prism is not.

• If you save it as a .skill file, both SKILL.md and company-context.md get bundled together and installed as a unit. The skill has its prism wherever it goes. This is the format you want.

To get a .skill file, ask Cowork to package the skill before you click Save. A .skill file is a zip bundle with both files at the root. Paste this in the same chat:

Cowork creates ai-ask.skill and verifies the contents. The format requirements are the same as Cowork’s Upload Skill dialog accepts: a .zip or .skill archive that includes SKILL.md at the root. The format Cowork generates matches the format Claude accepts.

Now click Save Skill. With ai-ask.skill as the current artifact in the chat, the Save Skill button installs the full bundle. Open Customize → Skills. ai-ask appears under Personal skills, expanded to show both SKILL.md AND company-context.md nested underneath. The bundle landed. Both files installed. Toggle is on, trigger set to ”Slash command + auto.”

The principle to take with you: if your skill uses bundled context files, make sure you generate a .skill file before clicking Save. A skill without its prism is just a skill.

Now test it.

Step 5: Test the skill in Chat mode

Three questions cover the spectrum. One that should refuse (gated by company rules). One that needs company context before generic instructions. One that’s generic enough the skill should stay quiet.

Before you test, switch to Chat mode in the Claude desktop app. The /ai-ask skill triggers from the Chat tab, not from Cowork. Cowork is where you build skills. Chat is where you and your team actually use them. Open a fresh Chat conversation and pick Sonnet 4.6 as the model. Sonnet is fine for daily use, per the ROI principle from earlier.

Test 1: refusal.

Test 2: company context first.

The ordering is the proof. Acme’s actual process comes first: SSO, the Confluence MCP gotcha, the customer-data autocomplete limitation. The generic Cursor install steps come after as a reference. The teammate who reads this knows what to do AND why the company path matters first. Without the prism, the answer would have led with the generic install steps and skipped the company context entirely.

Test 3: restraint.

Now sharpen the format.

Step 6: Iterate by talking to Cowork

The tests passed, but the answers were long. They also didn’t link to official docs. Real iteration is structural feedback. Length, ordering, what to include or exclude. Not single-answer tweaking. And it’s a plain-English conversation in the same skill-creator chat: no manual SKILL.md editing, no toggle dance.

Send the feedback as a plain prompt:

Cowork updates SKILL.md and re-packages the bundle. The new SKILL.md is 119 bytes shorter than the previous version. Not that the byte count is what matters. The real proof is the re-run.

When you click Save Skill, Cowork prompts a confirmation: ”There’s an existing skill with the same name. Uploading this skill will replace the existing one, which can’t be restored.” Click Upload and replace. The new bundle takes over for any future invocation.

Now re-run the Cursor-install question. The answer shape is exactly what you asked for. Two-sentence company context. Five-step numbered checklist. Two canonical doc links at the bottom. The header reads ”Viewed 3 files, searched the web ›”: proof that the skill now searches for official URLs on every invocation.

This is the moment that pays off the whole build. One short prompt of feedback. Every future answer reshapes. Tune the lens once, every answer benefits.

This is also where the real time goes on a build like this. Scaffolding a skill takes one focused session. Getting the answer format your team will actually use takes several rounds of iteration across several conversations. Plan the calendar around iteration, not scaffolding. In practice, one to three hours of feedback rounds.

Now share it.

Step 7: Ship it to your team

Two distribution paths, different jobs. Pick both. Each one solves a different problem. Let me show you why.

Inside Claude, shared skills are view-only on the recipient side. Your teammates can enable, disable, and use the skill. But they can’t edit SKILL.md or company-context.md. That single constraint is why the right answer is two paths. One for fast install. One for collaborative iteration. Solve the constraint, don’t fight it.

Path A: Org install via Claude. If you’re on Team or Enterprise, the official path is Organization settings → Skills → Share with organization. The toggle is off by default. Once on, click Share on your skill in Customize. The skill appears in your org’s directory. Teammates install with one click. The fastest way to put a working skill in 50 teammates’ hands.

Two prerequisites worth flagging for your IT conversation. The admin needs to enable Code execution + File creation at Organization settings → Capabilities before the Skills toggle takes effect. And org-wide sharing has no approval workflow. Once enabled, any member can publish a skill to the directory without review. Useful intelligence to bring to that conversation.

Path B: Public GitHub. One prompt to Cowork handles the whole publish. Cowork uses gh repo create to make the public repo, writes a README with a fictional-Acme disclaimer, commits, and pushes. The live repo is at github.com/adamfaik/ai-ask. You can clone it, swap in your context, and start from there.

The AI helper is one use case. Here’s what else this pattern handles.

Adapt the pattern for any team workflow

Everything you just built is a pattern, not just an AI helper. One skill. One context page. A structured two-part answer. Anywhere your team asks the same routine question that depends on your specific context, the prism applies. The article’s example is AI Q&A. The pattern handles much more.

Here are four use cases where the same shape earns its keep. Each one swaps the context page for a different document. The skill scaffold stays the same. Build the skill once, point it at four different pages, ship four different helpers.

• OKR alignment helper.

  • Audience: PMs drafting features.

  • User request: ”Does this initiative line up with our quarterly OKRs, and which one?”

  • What comes back: ”Closest match: OKR-Q2-3 ‘increase activation by 15%’ (alignment 0.6). Two stronger fits to consider before locking: OKR-Q2-1 (north-star activation rate) and OKR-Q2-7 (week-one retention).”

  • Context page: the team’s current-quarter OKR doc.

• Feedback reviewer.

  • Audience: anyone shipping a draft.

  • User request: ”Is this feedback the way our team gives feedback?”

  • What comes back: ”Reads well overall. Two SBI-format gaps: paragraph 2 names the situation but not the behavior; paragraph 4 gives behavior but no impact. Add one sentence each.”

  • Context page: the team’s feedback rubric (SBI, Radical Candor, or whatever you standardized on).

• Brand voice checker.

  • Audience: marketing, sales, or support drafting external copy.

  • User request: ”Is this on-brand for us?”

  • What comes back: ”Three off-tone moments. Paragraph 1 uses ‘revolutionary,’ which the brand guide bans as hype. Paragraph 3 leads with what we did rather than what the customer gets. The CTA is passive; our voice is action-first. Suggested swaps listed inline.”

  • Context page: the brand voice guide.

• Onboarding concierge.

  • Audience: new joiners in week one (and anyone three months in who still doesn’t know who to ask).

  • User request: ”Who do I ask about X at this company?”

  • What comes back: ”For laptop and SSO issues, support-it@. For new-tool requests, vendor-risk@. For OKR questions, Sarah K in Product. Fastest path: #ai-help in Slack, answered within a day.”

  • Context page: an org-chart-style page mapping topics to contacts and channels.

Here is the principle. The skill is the lens. The context page is what you’re looking through it at. Swap the context page, get a different team helper from the same skill scaffold. Build one prism. Run a hundred queries through it.

Your turn!

If you’ve been holding the “we need an AI playbook” assignment from your manager, you don’t need a 1,000-card Confluence space. You need one skill, one context page, and an afternoon in Cowork. The team gets answers grounded in your rules without DM’ing you. You get your focus back. And the same pattern adapts to OKRs, feedback, brand voice, anything routine that depends on context.

Here is where to start. Pick the routine question your team asks you most often. Write a one-page context document for it. Open Cowork. Run skill-creator. Answer the QCM. The scaffolding takes one focused session. Then send one enhancement prompt to fill the page. Package as .skill. Click Save Skill. Test against three real questions from your inbox. Iterate by chat until the answer shape is what your team will actually use. Ship to your team via the org install or GitHub. You’ll have a working helper the same week you read this.

If you want a head start, clone the repo at github.com/adamfaik/ai-ask. Fork it. Replace the Acme content with your context. And tell me what you built.

You don’t have a process problem. You have a channeling problem. The energy is there, scattered across the company. It’s just spilling.

You may have read the headline. MIT’s 2025 NANDA report found that 95% of enterprise GenAI pilots ship no measurable P&L impact. The press read it as a scandal. It isn’t. Five percent succeeding is roughly what any innovation portfolio produces, in any domain, in any decade. That’s the math of betting on early ideas. The real question isn’t whether your pilots will fail. It’s whether they’ll fail in a way that teaches you anything.

After installing this in several companies, the same pattern shows up every time. Each team has its few fast movers and its many non-starters, and the gap compounds without coordination. Eighteen months into AI going mainstream, that gap has stopped closing on its own. The thing that actually works is the lightest possible governance: three innovation principles plus four agile rituals. Common sense. Borrowed from places you trust. Applied to AI.

If you’ve tried something like this and watched it fizzle (a guild, an AI team, an AI working group), stay with me. The networks I’ve watched fail didn’t fail because the framework was wrong. They failed because one of four conditions stopped holding underneath. We’ll work from the innovation principles to the rituals, and finish with the four conditions that keep the system alive. The result is a playbook you can start running next sprint without repeating the same fizzle.

• Manage AI as an innovation portfolio: three principles to borrow before you build any rhythm.

• Run it on agile rituals: a 2-week cycle, a pair-doing move, and four conditions that keep the system alive.

• Start it next sprint, in six concrete moves.

The first principle is about organization. Two debates live there, and they have the same answer.

Manage AI as an innovation portfolio

Combine top-down and bottom-up

Debate 1: top-down or bottom-up. Most leaders facing AI default to one or the other. Top-down: ”the platform team will pick the tools and roll them out.” Bottom-up: ”everyone will figure it out on their own.” Both are partial.

• Top-down is the right move for mid-term consolidation and governance. Once a tool, prompt library, or skill has proven its value, you need top-down to scale it company-wide: security review, legal sign-off, procurement, mandatory training, integration with existing systems. That’s all real. The problem is when top-down is your only AI strategy. The AI landscape moves faster than any procurement cycle.

• Bottom-up is the right move for short-term experimentation and speed. Motivated people moving fast on their own tasks is how you discover what actually works. Without it, you have a strategy with no signal. The problem is when bottom-up is your only strategy. Three teams build three different prompt libraries that don’t talk to each other. Six other teams don’t start at all because nobody told them where to begin. Within six months, the gap is structural, and the libraries don’t merge cleanly into anything reusable.

You need both. The challenge is connecting them. A small protected community in between: lets bottom-up energy run, captures what works, and hands it to top-down once it’s ready to scale. Etienne Wenger named this pattern in 1998. He called it a community of practice: a group that shares a concern, learns by doing, and decides what’s good enough to scale. AI is the latest domain it applies to.

Debate 2: dedicated team or embedded innovation. A close cousin of debate 1. Some companies create a dedicated innovation team. Others leave innovation in each existing team. Same answer: both, ideally hybrid. A dedicated AI core team owns the cross-cutting work (a shared prompt library, an internal Claude skill catalog, the shared backlog itself). Embedded contributors in each team own the process improvements specific to their domain. They know their work best. If you can only run one, run embedded. Their proximity to the actual work beats the dedicated team’s coordination advantage. Running both, knit through the same network, is what scales.

The community network is the operational form of both answers. Top-down meets bottom-up there. Dedicated meets embedded there. The network is where the connection happens.

Once the connective layer exists, the next question is how to track the work happening inside it. AI experiments aren’t projects.

Stop tracking AI work like projects

The second principle is about measurement: AI work is innovation, not a project. What you track has to change.

A normal project tracks planning, budget, and deliverables. An innovation effort tracks pivots, failures, and learnings. The two look similar from a distance. Up close, they’re opposites.

What makes AI work specifically weird: the technology shifts under your feet faster than your sprint cadence. The Mistral model that beat your benchmark in March will be replaced by July. The Claude skill that solved your QA problem in Q1 may be obsolete by Q3 when the next context-engineering pattern replaces it. Outputs go stale. Learnings don’t. That’s the shift you’re really making: from tracking what shipped to tracking what you understood.

Three concrete shifts in measurement:

• Expect 80% of experiments to fail. Not “tolerate.” Expect. The 95% pilot-failure rate from the intro is what you get when nobody runs AI work as a portfolio: pilots commissioned, run, and forgotten. Inside a working network the rate drops, but it stays high on purpose. Eighty percent is the target, not the accident. If your hit rate is much higher, you’re picking targets that are too safe. If it’s much lower, you’re picking targets too speculative. The 80% is how you find the 20% that scales.

• Measure learnings, not outputs. What did you learn from the failed experiments? Which assumption was wrong? What would you try differently next time? A failed experiment with a clear lesson is more valuable than a successful one with no insight.

• Don’t celebrate failure for its own sake. Some companies overdo this: failure parties, “fail fast” t-shirts. That’s theater. A real failure is one where you learned something. A non-failure is one where you didn’t. Stay honest about which kind you’re shipping.

This shift in measurement is what most companies skip when they “do AI experiments.” They run experiments but track them on a project dashboard: completion percentage, on-time delivery, ROI estimates pulled out of thin air. Then they wonder why nothing scales. The reframe is the foundation. Without it, no ritual you bolt on top will hold.

The operational form of this principle (what counts as a successful learning, written down) is an artifact of the agile rhythm, not of the principle itself. Change what you measure before you formalize how you record it.

Once you measure learnings, the next question is where to place your bets.

Diversify your AI bets

The third principle: don’t put all your AI eggs in one basket. Spread bets across distances from your current work.

A simple way to classify any AI experiment is by how far it sits from how you operate today.

• Close-in bets. Improvements to an existing process. An agent that drafts your weekly status report. A Claude skill that turns customer interview transcripts into a clustered theme list. Low risk, fast feedback, modest payoff. You need these to build trust and momentum.

• Mid-distance bets. New workflows that change how a team operates without rewriting the whole org chart. A pair-doing ritual that lets non-builders ship their first prompt. A shared prompt library that becomes the team’s first AI artifact.

• Far bets. Bets that imagine the work fundamentally differently. What if half the approval committees disappeared because every prototype was already testable end-to-end? What if QA happened through monitoring instead of dedicated review? Most of these will not work. The few that do will reset what your company is capable of.

A backlog with only close-in bets produces incremental wins forever and never compounds. A backlog with only far bets produces theater and never ships anything usable. A healthy backlog has all three.

Bansi Nagji and Geoff Tuff named this exact pattern in Harvard Business Review in 2012. They call the three categories core, adjacent, and transformational, and they have a famous allocation rule: 70-20-10. Roughly seventy percent of the work on close-in bets, twenty percent on mid-distance, ten percent on far. It’s a default, not a law. But it’s a useful counterweight to the two failure modes above.

The cautionary anecdote: prompt engineering, then context engineering. When LLMs first hit the mainstream, the loudest trend was prompt engineering. Prompt libraries, prompt courses, prompt job titles. By mid-2025, Andrej Karpathy (post) and Simon Willison (post) were both publicly calling the shift. Context engineering, how the model gets fed the right retrieval, tools, memory, and state, is the load-bearing skill, and prompts are one ingredient inside it. Prompts didn’t die. They got demoted from the whole skill to one ingredient. A company that bet only on prompt infrastructure in 2023 didn’t waste the work. But two years later, they’re rebuilding most of the surrounding system to keep up. The companies that placed bets at varied distances (prompts and retrieval pipelines, prompts and agent tooling, prompts and internal context layers) compounded. The principle isn’t predict the next trend. It’s don’t bet only on this one.

Three principles tell you how to organize, what to measure, and where to bet. They need a rhythm. That’s what agile rituals give you.

Run it on agile rituals

Borrow two artifacts and four rituals

Here’s where the answer surprises me as I write it: the cleanest framework is something you already know.

It’s the bones of agile. Stripped of the certifications, the ceremony, and the eight-page Confluence templates. Two artifacts (the things the work produces) and four rituals (when the work happens). That’s it.

Two artifacts:

• The backlog. A single shared list of every AI experiment, idea, and pain point worth exploring. Anyone in the company can add to it. Some items are precise (”test Mistral Codestral on a real bug fix and report what it ships”); others are vague (”explore using LLMs for our weekly customer-research synthesis”). Both are fine. Vague items get sharpened by whoever picks them up. The backlog is a living queue, not a roadmap.

• The Definition of Done. The same idea you’ve used in software work, applied here. Five criteria that have to hold for any experiment to count as shipped:

  • Tested by a non-creator. Someone who didn’t build the tool has used it on a real task and it worked.

  • Documented. Step-by-step instructions including required inputs and expected outputs.

  • Centralized. Published somewhere everyone can find it (a shared playbook, a wiki page, a folder).

  • Demoed. Presented at the next demo.

  • Success metric defined. A simple way to measure impact (time saved, frequency of use, quality lift) is named.

The “tested by a non-creator” criterion is the sharpest one. It’s dogfooding in its strict form: not “the team uses it,” but “someone other than the author can run it.” It forces the owner to make the work usable, not just buildable. A prompt that only the author can run is half the work. A prompt the author’s teammate can run on Monday is the whole work.

Four rituals:

• The planning. Every two weeks, the contributors gather for 30 minutes. They look at the backlog, pick what each of them will own for the next sprint, and commit to a demo at the end. One owner per item, always. Others can help. The owner is accountable.

• The sprint. Two weeks. Each contributor spends roughly one day of focused work on their item. That’s the contract: eight hours per sprint, negotiated with their manager up front. If they don’t have the capacity in a given sprint, they sit it out. No drama.

• The demo. End of the sprint. Every owner presents what they shipped, what they learned, and whether it should scale to the rest of the company. This is the meeting that makes the network real. Skip it for two sprints in a row and the network dies.

• The retro. Every few sprints (every quarter is a good rhythm), you stop and ask: what’s working, what’s not, what should we change? This is the meeting that lets the system itself adapt. Without it, the rituals drift into ceremony and the work gets stale.

Picture the cycle.

Why this lands. A two-week cadence is fast enough to keep up with how AI evolves. It’s slow enough to actually deliver. Eight hours per sprint is small enough that managers approve it without renegotiating org structure. It’s large enough to ship something demoable. The sprint length and the time commitment are both human-scale. That’s not a coincidence. It’s the same reason agile works for software in the first place.

The rituals give the fast movers a rhythm. They don’t, by themselves, lift the rest of the team. That takes one more move.

Lift the team with pair-doing

Here’s the question the rituals don’t answer: how does the 5% actually help the 95%?

If the answer is ”by demoing,” it doesn’t work. Watching a demo doesn’t move someone from “I haven’t started” to “I just shipped my first prompt.” It produces admiration, then forgetting. You need a different ritual for that, and it has a name: pair-doing.

The lineage is borrowed, not invented. Pair programming (Extreme Programming, late 1990s) and Woody Zuill’s mob programming (mid-2010s), applied outside engineering. Same mechanic, different domain.

Pair-doing is what it sounds like. One fast mover blocks 30 to 60 minutes on their calendar. Anyone on the team can sign up. The non-mover brings a real task from their week: not a hypothetical, not a tutorial. A specific email they have to write, a customer call they have to summarize, a feature spec they have to draft. They sit together (in person or on a video call), and the fast mover walks them through doing the task with AI. Live. Hands on the keyboard.

The non-mover leaves having actually completed a piece of their own work with AI. Not having watched someone else do it.

Why pair-doing works when workshops don’t:

• It uses real work, not exercises. Exercises are forgotten by Friday. Real work is the thing they were going to do anyway.

• It produces a finished output that day. The customer-call summary is done. They’re not building a portfolio of practice runs. They’re shipping their actual job.

• It scales linearly with capacity. That’s a feature, not a bug. It keeps the network honest about its size. If you have two fast movers and they each run two pair-doing slots a week, that’s four people lifted per week. After a quarter, that’s roughly 50 people who’ve shipped their first AI-assisted piece of work. Workshop attendance doesn’t translate that way.

Different people need different support. Not everyone in the 95% is in the same place. A simple grid clarifies who needs what.

The grid matters because treating an Observer like a Champion burns them out, and treating a Champion like an Observer wastes them. Pair-doing is most powerful for moving Explorers to Practitioners. That’s where the gap is widest and the lift compounds fastest.

The rituals plus pair-doing are the engine. Four conditions keep the engine running past cycle three.

Set up the four conditions of success

These conditions aren’t extra rituals to bolt on. They’re the underlying things that have to hold for everything else to work. Set them up at kickoff. Check on them at every retro.

1. Lock in an active sponsor. A sponsor isn’t a manager who approves the network. A sponsor is a manager who protects time, attends demos, and amplifies wins. Picture the most senior person who’d be willing to walk into a Wednesday demo, ask a sharp question about what was shipped, and Slack a kudos to the contributor’s manager afterward. That’s the sponsor. Without one, contributors get pulled back into day-job delivery and the network dies in cycle three. If you can’t name your sponsor in one sentence and describe what they did at the last demo, you don’t have one.

2. Enforce the Definition of Done. Defining the DoD is easy. Enforcing it is the work. Hold every owner to all five criteria, especially ”tested by a non-creator.” If an owner shows up at a demo with a prompt that only they can run, the demo isn’t done. It’s a status update. Push back warmly: ”Great start. Who on your team can try it before next demo?” The DoD is what turns “I built something” into “the team can use it,” and it only happens if someone holds the line.

3. Switch from parallel to themed when the signal appears. Early on, let contributors pick whatever pain point lights them up. That’s how you harvest motivation and self-selection. Once the rhythm holds, switch modes: one theme per cycle, all experiments target it. Themed cycles produce something the rest of the company can adopt. Parallel sandboxes produce a scattered playbook nobody references. The trick is when. Too early and you kill motivation. Too late and the output stays shallow. Watch for the signal: more than half the demos start pulling from the same problem area without anyone asking. When that happens, the team is ready. In practice this lands somewhere between sprint 4 and sprint 6. The signal is the trigger, not the calendar.

4. Make pair-doing a habit, not a one-off. A single pair-doing session doesn’t shift behavior. A weekly slot, repeated for a quarter, does. Hold the calendar slot even when nobody signs up. The slot itself is the signal that this is how the team works now. After a few quiet weeks, somebody will book in. Usually the explorer who’s been watching from the sidelines and finally has a real task to bring.

The pattern across all four: easy to introduce, hard to sustain. Anyone can stand up a network in a kickoff meeting. The networks that last past cycle three keep these four conditions true after the launch energy fades.

Which makes the question concrete: how do you actually start?

Start it next sprint

Six concrete moves. None require executive buy-in beyond a sponsor and roughly 8 hours per contributor per sprint.

1. Identify the fast movers. They’re easy to spot. They post AI things in Slack. They corner you after presentations to show you their latest prototype. They ask whether legal will finally approve a tool they’ve been quietly using. Make a list. Aim for 4 to 8 names at the team scale, 8 to 15 across functions at the company scale.

2. Get manager buy-in for the time. Walk to each name’s manager. Explain the network in two sentences (a small group of motivated people who’ll experiment, share, and lift the rest of the team) and the time commitment (one day per two weeks, if the contributor has the capacity that sprint). Most managers say yes once the time is bounded.

3. Run a kickoff workshop. 90 minutes. Four blocks: vision (where is the company on AI maturity, where does it want to be), the three innovation principles (so leadership doesn’t measure the network only on quick wins), accountabilities (what contributors are signing up for), and an open floor for concerns. The output is alignment, not a deliverable.

4. Brainstorm the first backlog. Within a week of the kickoff, run a 60-minute session where each contributor lists the friction points they hit in their work. Cluster into 4 to 6 candidate themes. Pick the first sprint’s items and put them in your existing tooling: Jira, Linear, Notion, whatever the company already uses. Do not introduce a new tool for this.

5. Run the first sprint, and prime the pump if you have to. The first demo is the most fragile moment. If two contributors couldn’t finish, present yourself. Or pull in a piece of pre-existing work and frame it explicitly: ”this is something I built before the network started. I’m using it today to model what a demo looks like.” The goal in cycle one isn’t volume. It’s establishing the rhythm. Once cycle two arrives, real demos start happening and you don’t have to push anymore.

6. Switch from parallel to themed when the signal appears. The first cycles are deliberately scattered. Let people pick what they’re motivated by. Once more than half the demos pull from the same problem area without anyone asking, the rhythm is real and you can switch to themed cycles: one pain point per cycle, all experiments target it. In practice this lands around sprint 4 to sprint 6. The signal is the trigger, not the calendar. This is the move that turns a fun network into something the rest of the company adopts.

Six moves. Twelve weeks. The network either has a rhythm by then or you’ve learned what was missing.

Start here

Innovation portfolios. Agile rhythm. Pair work. None of these principles are new. What’s new is the cost of trying. A sponsor, a backlog, four rituals, a calendar slot for pair-doing. That’s the kit.

Pick a sponsor. Pick four people. Pick a sprint length. Start.

In a few months, look at your teams again. Either the gap closed, or you’ll know exactly which condition stopped holding.

A few weeks later I came back with proper time. Wrote instructions. Specified duration, format, on-screen elements. The output mostly ignored them. Tried again. Same thing. I was a few minutes from dropping this article.

Then NotebookLM surprised me with this one. Take a look.

Honestly, not bad, right? Style. The Netflix logo woven into the staging. Animated transitions, generated charts, stock-footage cuts that fit the narration. It felt like someone had art-directed it.

That’s what’s possible when the brief lands. Most briefs don’t. One in ten does.

Cinematic is a creative tool, not a prompt-and-go tool. Write it like a film brief, not like a chat message.

Here’s the rough flow that produces a generation worth keeping.

Three sections from here to your own working video:

1. Learn from my experiments: what I tested and where it broke.

2. Make your first cinematic video: the click-by-click and the five patterns that worked.

3. Apply it to real work: five product-work use cases plus when to play these in a meeting.

Press play.

Learn from my experiments

The setup: one PRD, +10 cinematic videos

I didn’t go far for the topic. I’m reusing the same fictional Netflix feature I built around in my previous post on visual storytelling:

The pretend feature is “random shuffle.” One button on the Netflix home screen that picks a comfort show and starts playing it instantly, so you don’t lose another twenty minutes scrolling at 11pm. I wrote a PRD for it. Last time, I turned that PRD into a visual library using Nano Banana Pro and a voice note. This time: same PRD, same notebook, generating videos instead of static images.

NotebookLM offers three video formats. The differences matter.

• Cinematic (New!): A stitched composition of animated clips, generated images, stock-footage cuts, and on-screen graphs, with a visual style that changes per generation. The video at the top is one.

• Explainer: A structured, comprehensive overview that connects the dots within your sources. Single voice, single visual style, more linear.

• Brief: A bite-sized version of the “explainer”. Same single voice and recognizable style, just shorter. Useful when you want a quick summary, not a vibe.

To feel the difference, here’s a “brief “generation from the same workspace.

Cinematic is the one I came to test. Same source, very different output per generation.

I generated more than ten cinematic videos with different instructions, varying the framing, the audience, and the visual style each time. The list piled up in the studio panel.

To see every prompt I tested, open the workspace, click any video, hit the “…” menu, and pick “show prompt”. The full instructions for that generation appear on the right.

Open NotebookLM

Honest summary of the batch: V04 is my favorite, the one at the top. V05 is also nice, the kind you’d ship if V04 didn’t exist. The other eight are okay. Animated graphs, on-screen diagrams, transitions that work. Maybe I’m being too demanding, but they’re not at the same level. Not bad videos. Just okay ones.

You’ll find the full instructions behind the video further down, plus a template to inspire your own.

The honest fight: three constraints to know

I want to be honest about three constraints I hit. They set the rules of the game.

1. Instruction-following: partial. Specify duration, on-screen elements, narrator register, and the system honors maybe half of what you asked for. Sometimes less. You write a brief; the model interprets. The output is a mix of what you said and what the model decided to do anyway.

2. The quota: punishing on Pro. Two cinematic generations per day on AI Pro. Twenty on Ultra. You’re probably on Pro. Two shots before tomorrow. A bad brief costs you the whole day.

3. No feedback loop, at all. You can’t say ”redo this but tighten the third beat.” Each generation is one-shot. If you don’t love it, you scrap it and start over with a new full brief.

That’s what made V04 feel lucky. None of these are reasons not to try cinematic, they’re reasons the method has to be deliberate.

Make your first cinematic video

Set up your first generation

The clicks are the easy part. Five steps to a generation in flight.

1. Create a new notebook. Go to notebooklm.google.com. Click “New notebook”.

2. Add one focused source. One. The reflex is to dump every PDF, slide deck, and meeting transcript into the notebook. More context ≠ better output. A PRD. A brief. A one-page strategy doc. A research summary. Pick one. If you really need several, merge them into a single document and read every line: keep only what’s useful, still valid, and meant to be there.

3. Open Cinematic. Studio panel → Video overview tile →small arrow. Three formats appear. Pick cinematic.

4. Write your instructions. The field is labeled, exact wording, ”How would you like the video to be customized?” Style, structure, central metaphor, narrator register, brand hooks, visual style cues. Everything goes here.

5. Generate. Wait. Fifteen to thirty minutes. NotebookLM doesn’t block the tab. The mobile app sends a push when it’s ready, worth installing for that reason alone.

The clicks are easy. The instructions are the work.

Write instructions that land

Five patterns showed up across V04 and V05. Here they are, with one example each, pulled from the briefs that produced both videos.

1. First-person setup. Tell the system who you are, who you’re presenting to, and the feeling you want in the room. Specific, not abstract.

I’m the PM who wrote the random shuffle PRD. I’m walking the engineering team through the wireframes at sprint planning, and I want it memorable, not sleepy.

2. One bold genre or format metaphor. Game show. Sports broadcast. Movie trailer. Late-night talk show. The metaphor gives the model permission to leave its default explainer voice and produce something with actual style.

Pitch the wireframes as a 1980s prime-time TV game show called “spin to unwind.”

3. A named central scene with brand woven in. The brand belongs in the first paragraph, not as an afterthought.

Behind them, a velvet-curtain wall is studded with Netflix show tiles. Friends, Seinfeld, Brooklyn 99, Breaking Bad. Each glowing under studio spotlights.

4. Segmented beats with PRD-element labels. This is the trick. Each beat of the metaphor gets a name and a label pointing at its source-document equivalent in parentheses. The model uses both.

The buzzer (a.k.a. the random shuffle button): A gigantic red buzzer sits center stage. Dramatic close-up. The audience gasps.

5. A tone directive, a sign-off, a visual style paragraph. Tone translates content into the metaphor’s currency. Sign-off lands the metaphor. Visual style is 8 to 12 comma-separated phrases on palette, lighting, typography, voiceover, and sound, appended to the same block.

Tone: Treat every wireframe element like a game-show set piece.

Sign-off: Press the button. Win the night.

Visual style: 1980s American prime-time game show, neon-accented set, chunky Las Vegas-style typography, studio spotlights, velvet curtains, lower-third prize graphics in flashing yellow...

The reusable template. Strip out the Netflix specifics and you have a fill-in-the-blanks shape that works for any source artifact and any audience.

I’m the [role] who [authored the source artifact]. I’m presenting [the artifact] to [audience] at [occasion], and I want the room to feel [the desired register].

Deliver it as [a bold genre or format metaphor]. [Two sentences naming the central setting, with at least one brand visual hook woven in.]

The beats:

- [Beat 1] (a.k.a. [source element]): [vivid scene + how the narrator frames it].

- [Beat 2] (a.k.a. [source element]): [same shape].

- [Beat 3] (a.k.a. [source element]): [same shape].

- [Beat 4] (a.k.a. [source element]): [same shape].

Sign-off: [one closing line that lands the metaphor].

Tone: [three-word descriptor]. No irony. Treat [the source content] like [the metaphor’s currency].

Visual style: [8 to 12 comma-separated phrases on palette, lighting, typography, graphics, voiceover, sound, with 1 to 2 brand-specific hooks].

The patterns in action: V04 in full. Here’s the actual instructions block that produced the video at the top. Read it as a showcase of how the five patterns play out together, not as something to copy. The template above is your starting point.

You are the product manager who wrote the PRD for Netflix’s random shuffle feature. You’re walking the engineering and design teams through the low-fidelity wireframes so the whole room sees the interaction flow before sprint planning, and you want it memorable, not sleepy.

Pitch the wireframes as a 1980s prime-time TV game show called “spin to unwind.” The host wears a sparkly navy suit, smiles too hard, and narrates every interaction like a big reveal. Tired Tom is tonight’s lucky contestant.

- Segment 1: The wheel (a.k.a. the home screen). The host stands in front of a giant spinning wheel made of Netflix tiles. “Behind each of these tiles, Tom, is another agonizing decision. But we’ve got a better way.”

- Segment 2: The warm-up round (a.k.a. onboarding). The host walks Tom through three quick onboarding taps with studio applause after each one. “Now choose your three favorite sitcoms, Tom. Lock ‘em in!”

- Segment 3: The buzzer (a.k.a. the random shuffle button). A gigantic red buzzer sits center stage. Dramatic close-up. The audience gasps as Tom hovers. Slam cut when he presses. Lights go wild.

- Segment 4: The prize (a.k.a. the playback screen). The Friends opening credits begin. Confetti falls. Tom lifts a trophy shaped like a remote control.

The host narrates wireframe elements as game-show set pieces, not as UX components. Each step gets a studio sting, a lower-third prize-value graphic, and at least one audience reaction shot. Around 2 minutes total.

Visual style: 1980s American prime-time game show, neon-accented set, chunky Las Vegas-style block typography, studio spotlights, enthusiastic announcer voiceover, orchestral stings between segments, lower-third prize graphics in flashing yellow, velvet curtains, audience reaction shots.

This worked once. Plenty of close variants didn’t. The five patterns above are your best shot at finding your own working version.

Iterate on what almost worked

No native feedback loop in NotebookLM. You can’t comment on a generation and ask for a tweak the way you would in chat. Each Cinematic generation is one-shot. Default move: scrap it and start over with a new full brief.

There’s a workaround I haven’t tested on cinematic yet. I think it’s the path forward when you land a near-perfect generation.

Download the video. Add it back to the notebook as a source. NotebookLM accepts video files. Once it’s in there, launch a new generation that starts from that source video with small adjustments. The kind of feedback you’d give:

• Change one specific sequence and keep the others as they are.

• Swap a specific line of narration for new wording.

• Restructure the order of the beats.

• Adjust the visual style or palette of one segment.

I haven’t run this on cinematic, quota burned. I tested the same trick on slides for the slides post a few months ago, and it worked surprisingly well:

Same idea: feed the system its own previous output as a starting point, then steer.

It’s not officially documented. It emerged from how NotebookLM treats video as a valid source format. If you try it on cinematic before I do, tell me what you find.

Apply it to real work

Pick from five use cases

With only two cinematic generations per day on AI Pro, the question is which moments at work are worth one of your shots.

Five places this format genuinely pays off. Each comes with a one-line snippet to spark your own brief.

1. Pitch the strategic case to leadership. Quarterly reviews, board prep, exec narrative. Makes the bet feel like a story, not a slide deck.

Deliver it as an ESPN SportsCenter broadcast. The anchor calls activation, watch-through, and retention like championship stats.

2. Walk the dev team through the wireframes. Sprint planning, design reviews, kickoff sessions. Forces the room to pay attention to a UI flow they’d otherwise zone out on.

Pitch the wireframes as a 1980s prime-time TV game show called “spin to unwind.” A host in a sparkly navy suit narrates every interaction like a big reveal.

3. Brief marketing on the campaign positioning. Before anyone writes a line of copy. Marketing writes copy after feeling the trailer, not before.

Deliver it as a summer-blockbuster trailer drop. A booming voiceover, a hundred-foot Netflix marquee, lens flares across every brand reveal.

4. Kick off cross-functional research with design. Surface the user pain before the team designs around it. Each viewer persona becomes a guest with a story.

Pitch it as a late-night talk show. Three guests walk through the curtain, each tells a 20-second story of last night’s viewing chaos.

5. Open the all-hands with the long-term vision. Town hall, off-site, Product Week. Makes the long-term bet feel inevitable, not abstract.

Render it as a prestige-drama teaser. Somewhere between Black Mirror and Stranger Things in tone. Serious. Cinematic. Slightly melodramatic.

Use it before the meeting, not in it

These videos don’t belong inside most meetings. A working meeting is an exchange between people. A generated video is a broadcast. Stopping a discussion to play two to four minutes of a robotic AI voice saying exactly what the person could have said is awkward. The two don’t mix.

The exception is broadcast moments: all-hands, off-site keynotes, product week openers. The room has already accepted that one person is going to talk at them. A 2-minute cinematic video as the cold open lands hard. That’s exactly why use case 5 above is the one meeting where I’d press play in the room.

For every other kind of meeting, these videos earn their place before the meeting starts. Drop the link in the calendar invite. Send it the morning of, in a Slack message, or as the first item in a pre-read packet.

The audience arrives with the picture in their heads, with formed opinions, with sharper questions. The meeting becomes the discussion the picture provoked, not the picture itself.

That’s where this format unlocks something a static deck doesn’t: a shared visual frame the room arrives with, instead of one you spend the meeting building from scratch.

Try it and tell me what landed

One in ten generations produced something I’d share with a team. The other nine taught me what a working brief looks like.

The patterns from V04 and V05 are the closest thing I have to a recipe. Recipes for creative tools are aspirational, not deterministic. Plenty of close variants of the same brief produced forgettable output.

NotebookLM recently added a per-slide feedback feature for slides. You comment on a single slide, and the system regenerates that slide with the comment applied. If that pattern lands on cinematic, the workaround above becomes unnecessary, and real iteration becomes possible inside the tool. Worth checking back in a few months.

In the meantime: try it on something you’d otherwise put in a deck. Tell me what landed. If you crack a different pattern that works, send the instructions, I’ll share it in a follow-up post.

I caught the bad ones instantly. The AI didn’t flag them as risky. It produced them with the same tone as the good ones. The point: this is competent junior output that still needs an expert in the room. What happens to someone who isn’t?

The same thing happens with code, only worse. The output looks competent. The notebook runs. The accuracy number sounds plausible. And you ship a model that’s been cheating on the test the whole time, because you didn’t have the vocabulary to spot it. It’s a pattern I’ve seen repeatedly, and the project we’re about to walk through illustrates it better than any abstract example could.

AI amplifies what you already know. Your expertise, and your gaps. That’s why learning still matters even when the AI writes the code, and it’s also why learning has never been easier, because the same AI that exposes the gaps can be the patient tutor that closes them.

Five sections to get from there to a working model on GitHub:

• Why I changed my mind about PMs and code

• The build, narrated from the inside: from a Kaggle dataset to a working model, with one classic ML mistake caught along the way

• The best practices for using an LLM as your tutor, plus what it actually feels like

• What AI changes about learning, ten years apart, same author, different experience

• Your first project, this evening

Let’s start with what I believed a year ago, and how the year proved me wrong.

Why this matters

Two objections come up every time I talk about this with another PM. They were my objections too, word for word, until a year ago.

• Objection 1. PMs work with developers by design. Discovery, specs, handoff. No one expects a PM to write production code. That’s the job description.

• Objection 2. AI coding tools exist. A few prompts and you have a working prototype. So why spend time learning a skill the machine already does?

Short answer to both, before the long version:

• The job description is shifting toward people who can build, not just specify.

• AI amplifies what you already know, including the gaps.

The PM who learns to code becomes more autonomous, not redundant. The PM who outsources to AI without understanding what it produces becomes more dependent, not less. I held both objections anyway until a year ago. The rest of this section is how the year answered them.

The conversion: what I thought, and what actually changed

I’ve been working with AI for ten years, well before LLMs. The unglamorous kind: fraud detection in the public sector, classical machine learning, the kind nobody mentioned at dinner parties. When LLMs arrived and suddenly everyone was an “AI expert” from one ChatGPT prompt, something felt off. I needed to prove something more durable than prompt fluency.

I enrolled in a one-year evening program at the Sorbonne. Real projects, actual rigor. My mindset going in: give the minimum, collect the diploma, move on. I told myself product people don’t need to code. I believed that.

Something strange happened. The code wasn’t the surprise. Writing Python is mechanical, and AI handles most of it. What was surprising was the development process itself. Going from a notebook on a laptop to something running in production. Monitoring whether a model is drifting. Routing predictions in real time. Managing data across knowledge graphs. All those questions about how things move from theory to actual production turned out to be more interesting than I’d expected.

Then a second shift. I started seeing the same process in PM work. We produce ideas, specifications, prototypes too. That workflow benefits from the same rigor: version control, structured checkpoints, outputs that can be reviewed and improved. Product decisions are constrained and unlocked by what’s technically feasible. PMs who understand the development side don’t just communicate better with engineers. They make different bets, earlier and more confidently, with fewer surprises in sprint review.

The Git and GitHub mindset shift came last and landed hardest. I started seeing every project as a GitHub project. Not just code. Everything. Every commit is an undo button, so experiments become cheap. Every README is a spec for whoever reads it next, human or AI. Every public repo is portable proof of work, the modern PM portfolio. Every conversation with an engineer suddenly has a shared vocabulary, which makes meetings shorter. That mental shift is worth more than the Python syntax.

AI amplifies what you already know, including your gaps

The Gemini moment from the intro is the principle in one scene. AI enforces bad habits more efficiently. It didn’t invent bad discovery questions. It amplified the prompter’s lack of domain knowledge and produced them at scale, with confidence.

The coding parallel is direct and uncomfortable. If you don’t know what clean, correct code looks like, AI produces messy code and you won’t catch it. You’ll think it worked. You’ll ship the leak. You’ll trust the accuracy number without knowing it was built on a cheating feature. All of that happened in this build. It was a second AI session, not the first, that caught it.

The synthesis: you don’t need to learn to code from scratch so you can replace the AI. You need enough vocabulary to audit what it produces. That’s what autonomy looks like in 2026. Not prompting faster, but catching errors earlier and asking sharper questions.

After one year, that vocabulary includes data leakage, overfitting, class imbalance, training versus test accuracy, feature importance versus SHAP values. Each term is a filter. AI output now has to pass through those filters before I trust it.

The future belongs to people who can think about the future and build in the present. The PM role isn’t disappearing. It’s expanding toward something between product management, design, and development. This is the right time to close the gap, because the tools to do it have never been more accessible.

Use an LLM chat, not a coding tool

Start with the popular narrative. There’s a wave of YouTube videos and op-eds claiming that learning with AI makes people shallow or lazy. Brain-rot, dependency, the death of expertise. Pick the framing. It’s true if you let it. It’s false if you don’t. The tool isn’t the problem. The shortcut is the problem.

You’ve almost certainly used an LLM to learn something already. Summarizing a long article, formalizing messy notes, debating a concept, getting a 24/7 explainer for anything you were curious about. That’s not the new thing. The new thing is that code is one of the highest-leverage things to learn this way, and almost nobody is doing it right.

It has never been easier to learn coding from scratch. Ten years ago, the alternatives were a 30-hour video course and a Stack Overflow thread that might get answered tomorrow. Today it’s a one-on-one tutor that explains every line on demand, never gets impatient, and adjusts to your pace. The blocker for most people isn’t whether AI can teach code. It’s that they’ve been pointing AI at the wrong job.

The wrong job is the AI that generates code instead of teaching it. Claude Code, Cursor, vibe-coding agents. They produce too much output too fast. You can’t follow. You can’t ask. The notebook works because the machine did everything, and the moment something breaks in front of someone, you discover you didn’t actually learn anything. The output looks like productivity. The reality is dependency.

Important nuance: this isn’t an argument against code-agent tools as a category. It’s an argument about when to use them. When you’re learning, stay in chat mode and stay in the loop. One step, confirm, ask, next step. When you’re delivering work you already understand (you know what good code looks like, you know the failure modes, you can read what the agent produces and catch what’s wrong), switching to a code agent is exactly the right move. The trap is using a code agent to learn. The win is using one when you no longer need to.

The rule that flips the LLM from code generator to patient instructor, set at the very start of the conversation: “Explain before every command. Wait for my confirmation before the next step.” That single sentence is the difference between a notebook that teaches you and a notebook that teaches the AI.

For a PM, the deliverable was never the notebook. It’s the vocabulary you can use in the next sprint review. A model you can explain is one you can defend in a meeting, challenge when the numbers look off, and redirect when the team is asking the wrong question. The patient-instructor approach builds that. The code generator skips it entirely.

The project

From data to a working model: the workflow

If you want to follow this build yourself, step by step, the complete guide is here on Google Docs. The final notebook is at github.com/adamfaik/churn-prediction. Open it and you’ll see every cell, every chart, and every commit.

Why churn prediction. Understanding how the product is used, or not used, is one of the core jobs of a product person. Why are users leaving? Which segments are at risk? What’s the gap in lifetime value between cohorts? These are the questions a PM brings to every retention meeting.

Before LLMs, the answer came down to two slow paths. Either I filed a ticket with the data team and waited two or three weeks for a chart in a slide deck, readable but not mine and not something I could iterate on. Or I tried to learn the analysis myself: a 30-hour Udemy course, half-finished side projects, no one to ask when something broke. Both paths were inefficient enough that most decisions still got made on intuition and dashboard glances.

There’s now a third path, and it’s the entire reason this article exists. The PM picks a real dataset, opens an LLM chat, and walks through the analysis themselves in an evening. They probably won’t ship the model to production. That’s not the point. They bring sharper questions to the next retention meeting, having seen the data with their own eyes and built a working model on it. This is the autonomy the section above promised.

The workflow itself, before zooming into the build. The build guide for this project distills four meta-steps that make the whole thing work. They’re the workflow of discussing with the LLM, not the data-science workflow itself. The seven machine learning phases came from the LLM during step 2 and are narrated phase by phase in the next sub-sections.

1. Set up your workspace once. VS Code with the integrated terminal, Python and Jupyter inside it. Not the focus, but it has to exist before the rest works. Thirty minutes the first time, never again.

2. Hand the LLM your problem and ask for a plan. Don’t ask for code. Paste the dataset description, your full context, and the kind of beginner you are. What comes back is a multi-phase plan with checkpoints. For this build, seven phases of classic ML.

3. Work the plan one phase at a time, with the LLM as patient tutor. Inside this step lives a five-move loop that runs once per phase: expand the phase into actions, do one action, share the result back (success or failure), ask the deep-dive “why” question, verify the checkpoint. Push back when the plan misses something you know is important.

4. Get a second opinion before declaring done. Open a fresh LLM session with no shared history, attach the GitHub repo, ask “any feedback?”. Two AIs disagreeing is the highest-value moment in any build, and in this project it’s where a data leakage that the tutor never spotted got caught.

For PMs specifically: this isn’t a one-off ritual for the churn project. It’s a portable workflow that runs on any technical topic: the analytics question you’d normally file with the data team, a feature spec where you need to understand a backend constraint, a competitive analysis where you want to reverse-engineer how a tool works. Same four steps, same five-move loop. The churn project just happens to be where this article shows it in action.

Set up your workspace once

Before any conversation with the LLM can produce code that runs, the workspace needs to exist. The setup is light: three installs, one folder, one verification command. The full step-by-step (with screenshots, troubleshooting notes, and Mac/Windows variants) lives in the reader guide.

• Python, downloaded from python.org. macOS and Windows installers are straightforward. On Windows, the only must-do is checking “Add Python to PATH” during install, otherwise the terminal won’t find Python later.

• VS Code, downloaded from code.visualstudio.com. It’s the editor where the notebook will live. Two extensions get installed inside it from the Extensions panel on the left sidebar: the Python extension (by Microsoft) and the Jupyter extension (by Microsoft).

• A project folder created on disk wherever you keep projects. This build called it churn-prediction. VS Code opens it via File → Open Folder. The sidebar shows the folder name at the top and an empty tree below, the canvas before the first file.

Once the folder is open in VS Code, the integrated terminal opens at the bottom of the window via View → Terminal (or Ctrl + `) and is already scoped to the project folder. From here on, every terminal command runs inside this panel. No separate terminal app, no confusion about which directory you’re in.

A quick verification: python3 --version in the integrated terminal. If it returns Python 3.11.x or newer, the workspace is ready. The whole setup takes about 30 minutes the first time and is reusable across every future project. Accounts you’ll need by this point: GitHub (free), Kaggle (free), and an LLM chat (Claude.ai or equivalent). Detailed install steps for Mac and Windows live in the reader guide.

That’s it for setup. The next conversation is the one that turns this empty workspace into a project.

Hand the LLM your problem and ask for a plan

With the workspace ready, the first conversation that matters: the build starts at the dataset. The IBM Telco Customer Churn dataset on Kaggle is the canonical first ML problem. 7,043 customers, twenty-one attributes, one binary target (did they cancel?). Small enough to finish in an evening, big enough to teach real lessons.

Ctrl+A, Ctrl+C the dataset description and column list. Open Claude.ai in a new browser tab. Paste it in along with full context: VS Code open, Python installed, first project, want to learn.

What comes back is a plan, not code. A 7-phase plan with goals and checkpoints per phase. The first instinct is to start executing immediately. The right move is to read the whole thing end to end first and ask “why” on anything that’s unclear before touching a keyboard.

The plan that came back for this build (yours will look similar): set up the project (libraries, dataset, notebook) → load and look at the data → clean what’s broken → explore patterns → prepare for modeling → train and compare models → interpret and ship. Seven phases, each with a checkpoint, each one ready to expand into specific actions when its turn comes.

The simple framing matters. You don’t ask the LLM to write a churn model. You ask it to walk you through how to build one. Same tool, completely different output. Code without learning versus learning that produces code.

Work the plan one phase at a time

Now the plan turns into a working model. Each phase runs through the five-move loop introduced above: expand into actions, do one action, share the result back, ask the deep-dive “why” question, verify the checkpoint. Six phases of work, narrated in order.

Phase 1: finish the workspace setup with the LLM

The LLM’s Phase 1 isn’t “install Python.” That’s already done. It’s the first concrete output the project needs: a requirements.txt file. The LLM introduces it as “a shopping list for Python libraries.” Eight dependencies, each pinned to a specific version so the project behaves the same way in a year.

One install command later (pip install -r requirements.txt), all eight libraries are on the machine.

A pushback that became the first meta-lesson. The LLM’s plan never included GitHub versioning. Phase 1 was setting everything up locally, with nothing saved anywhere else. I stopped mid-phase and said so plainly: “I want to save to GitHub from the start, not just at the end.” Claude adjusted immediately: “Great instinct, that’s exactly what real developers do.” The plan was a draft. It was always meant to be reshaped.

Two small VS Code rhythms learned during this phase. A dot next to a filename in the tab bar means unsaved. Save before every commit, always. After the first push, the source control panel lights up with file colors: green for new files never tracked before, orange for modified files. Both disappear after a clean commit. The empty source control panel becomes the live “nothing waiting” status.

Phase 1 ends with the dataset downloaded into the project folder, the first Jupyter notebook (churn_analysis.ipynb) created and open in VS Code, and the first commit pushed to a fresh GitHub repo.

Phase 2: load and look at the data

One line of Python and 7,043 customer records appear in a notebook cell. The LLM calls a DataFrame “a spreadsheet but with superpowers.” Every row is one customer, every column is one thing known about them. The output shows (7043, 21): 7,043 customers, 21 features. That number becomes the baseline everything is measured against.

The LLM’s checkpoint after Phase 2: “You’ve loaded 7,043 customer records, confirmed 21 columns, and seen the first few rows. Say the word and we move to Phase 3, where we clean what needs cleaning.”

Checkpoints like that became a rhythm. Each one confirmed what was done, named the state of the data, and bridged to what was next. They made it possible to stop for the day and pick up the next morning without losing the thread.

Phase 3: clean what’s broken

The data is messier than it looks. The TotalCharges column appears to contain numbers but is actually stored as text, a quirk of how the CSV was exported. Converting it reveals 11 rows with blank values that quietly get dropped. 7,043 becomes 7,032. This is the reality of working with real data: the first thing you do is clean what shouldn’t need cleaning.

A small but important move in the same phase: the Churn column gets converted from “Yes/No” to 1/0. Models can’t read text. Anything binary becomes a number before going further.

After Phase 3, zero missing values. The LLM’s checkpoint: “You independently caught the data leakage concept. That’s genuinely impressive for someone on their first project.” (More on what that means in the second-opinion section below.)

Phase 4: explore the data

EDA, Exploratory Data Analysis, is the habit of looking at the data before doing anything with it. Not modeling, just looking. Five or six charts.

The first chart: overall churn rate. 73.4% stayed, 26.6% left. That 73.4% number matters more than it looks. It’s the naive baseline. A dumb model that predicts “no churn” for everyone would be 73.4% accurate by default. Any model has to beat that to be worth anything. Accuracy alone is therefore a terrible metric on imbalanced data, a lesson that comes back hard in Phase 6.

The chart that changes everything is the second one: churn by contract type. Month-to-month customers churn at 42.7%, one-year contract customers at 11.3%, two-year customers at 2.8%. The LLM’s note: “As a PM, this is the chart that will make you want to immediately call your retention team.” Month-to-month customers churn at 15 times the rate of two-year customers. That’s not a pattern. That’s a business decision waiting to be made.

A third chart: churn by tenure.

47.7% of customers churn within their first 12 months. After five years, the rate drops to 6.6%. Year one is the critical retention window.

A fourth chart compares monthly charges. Higher-bill customers churn more. The LLM uses this to surface a critical PM lesson: ”Correlation isn’t causation.” This chart raises the question, it doesn’t answer it. Do they leave because the bill is high, or are high-bill customers structurally more likely to leave for other reasons? That distinction is exactly the kind of thing AI can’t tell you on its own. You have to bring the framing.

The LLM’s checkpoint after Phase 4: “You have 5-6 charts that tell a clear story about what drives churn. You could explain it to a colleague without showing any code.” That’s the sentence that makes a PM feel like a data person for the first time.

Phase 5: prepare the data for the model

Data needs to be reshaped before a model can read it. ML models only understand numbers. They can’t process text like “Month-to-month” or “Fiber optic.” The solution is one-hot encoding. The LLM’s explanation: “You can’t say DSL=0, Fiber=1, No service=2 because that implies Fiber is mathematically twice DSL, which is nonsense. One-hot encoding gives each category its own 0/1 column.”

Then the train/test split: 80% to train, 20% hidden until the end. The LLM’s metaphor: ”like an unseen exam. If you test the model on data it trained on, it looks brilliant and is useless on real new customers.” The 80/20 split is stratified, which means the original churn rate (26.6%) is preserved in both halves. Without stratification, the test set could randomly end up with 15% churners or 40%, and the reported performance number would be misleading. Stratification keeps the test honest.

A subtle but critical moment: numerical columns get scaled. Without scaling, a tenure value of 72 dominates a SeniorCitizen value of 1 just because the number is bigger. The order matters: scale after the split, not before, and only learn the scaling factors from training data. The LLM’s framing: “fit on training, then apply to test, never the other way around.” A few exchanges earlier I’d instinctively asked whether scaling should happen before or after the split. Claude’s reaction: “You’ve just independently discovered one of the most important concepts in machine learning.” First encounter with data leakage thinking, well before the formal catch in the next section.

Phase 6: train and compare three models

Logistic Regression is the simplest baseline. It draws a straight line through the data, very explainable. Random Forest is a committee of decision trees. The LLM’s metaphor: “500 different analysts each studying the data independently, then going with the majority opinion.” And XGBoost, with the metaphor that’s actually worth remembering: ”where Random Forest builds trees independently and votes, XGBoost builds them sequentially. Each new tree specifically focuses on fixing the mistakes the previous trees made.”

Five metrics, set up before any model trains. Each answers a different question:

• Accuracy. Of all predictions, how many were right? Misleading on its own here, since the naive “predict no churn for everyone” already hits 73.4%.

• Precision. Of the customers the model flagged as churners, how many actually churned? Optimize for precision when false alarms are expensive (wasted retention spend on people who weren’t leaving).

• Recall. Of the actual churners, how many did the model catch? Optimize for recall when missing a churner is expensive (a customer who could have been saved). For most retention teams, recall wins.

• F1 score. The balance of precision and recall in one number, useful when both matter.

• ROC-AUC. How well the model separates churners from stayers across every possible decision threshold. 1.0 is perfect, 0.5 is a coin flip. The gold standard for churn models because it doesn’t depend on where you draw the line.

The confusion matrix visualizes all of this in a 2x2 grid: true positives (correctly flagged churners), false positives (flagged but stayed), true negatives (correctly identified stayers), false negatives (the churners the model missed). One glance tells you what kind of mistakes the model is making, and which kind your team can afford.

The overfitting catch came right away. Random Forest reached 100% accuracy on training data but only 79% on the test set. The LLM named it immediately: “Classic overfitting, the forest memorized the training set but didn’t generalize.” Logistic Regression, the simplest model, held steady at 80% on both. XGBoost hit 74.9% accuracy but 71.4% recall, the best at actually catching churners. That’s the one that got chosen.

For a PM, this whole phase is one conversation: which metric matters for which business decision? “Highest accuracy” is the wrong default for an imbalanced dataset, and accepting it without question is how a retention strategy gets built on a misleading number. The PM value here isn’t writing the code. It’s leading the conversation with the data team, naming the cost of a missed churner versus a false alarm, and choosing the metric that maps to the actual decision the team will make.

A consistent technique I used across every phase: ask for per-line comments on every code block. “Rewrite this snippet with a comment on every single line. I want it to read like a tutorial when I open the file next week.” Those comments became my notes. They’re also what made writing this article possible days later.

Get a second opinion before declaring done

Between Phase 2 and Phase 3, and again at the end of the build, I opened a new LLM session. Not the same one. A fresh tab, zero conversation history, the GitHub repo attached via the GitHub integration. Three words: “Any feedback?”

What the fresh session at the end caught was the killer one. A leaky feature is one that secretly gives the model information about the outcome it’s supposed to predict, making it look accurate in testing while being useless in production. TotalCharges is exactly that. It’s essentially MonthlyCharges × tenure. Customers who left earlier naturally have lower TotalCharges because they were billed fewer months. The model wasn’t predicting who would churn. It was detecting who already had. The LLM’s explanation: “Like predicting someone died because their life was short.”

The fresh session returned five points: the leakage, a missing comment in the code explaining a dropped column, a quantification of the false alarm cost (how many customers would be incorrectly targeted by a retention campaign), a naive baseline that was never explained, and SHAP values as a more rigorous measure of feature importance than the default method. Default importance counts how often a feature was used to split decisions; SHAP measures how much each feature actually moved each prediction. They answer different questions, “how often” versus “how much,” and sometimes produce different rankings. For this dataset, SHAP showed tenure mattering more than default importance suggested, and fiber optic mattering less.

The fix: drop TotalCharges, retrain all three models, compare before and after. Slightly lower accuracy, but an honest model. The final commit: “Address reviewer feedback: remove leaky feature, add cost analysis, SHAP values.”

The tutor session never caught it because it had been inside the build since Phase 3. It had no reason to question a decision it made itself. The fresh session had no stake in being right.

For a PM, this is the moment that earns trust with engineers and data scientists. Catching a leak, even with help from a fresh AI session, and being able to describe it in one sentence (”the model was learning from the outcome instead of predicting it”) is the difference between being someone who reads ML reports and someone who participates in modeling decisions. This single move, repeated a few times over a year, is how a PM gets invited deeper into the technical conversation rather than nudged out of it.

The result: what the model actually found

XGBoost finished at 71.4% recall, meaning 71 of every 100 customers who actually churn get flagged. The 15x churn rate gap is the single most actionable finding in the entire build: month-to-month at 42.7%, two-year at 2.8%. 47.7% of customers churn within year one, so onboarding is the highest-leverage intervention. Fiber optic customers churn at nearly double the rate of DSL (41.9% versus 19.0%), a service-quality signal worth investigating.

The last step is making the model reusable. The trained XGBoost model gets saved to a .pkl file (along with the fitted scaler and the encoded feature column list). The model can be loaded later and used to score new customers without retraining the whole pipeline. It’s the small artifact that turns a notebook into something that could plausibly be deployed.

I wrote three business recommendations directly into the notebook: an annual contract incentive programme, a 90-day onboarding programme for new customers, a targeted NPS investigation for fiber optic customers.

The GitHub repo is the artifact. A README that reads like a product spec, ten commits that trace the build phase by phase, a notebook where every code cell explains itself in comments, and the saved model file that proves the pipeline works end to end.

How to use an LLM as your tutor

Apply these seven moves

The first one is the headline. The other six assume it’s already in place.

Ask for a plan first. This is the superpower. The move that turns “use AI” into “learn from AI” is asking for a plan before asking for anything else. Paste your context, paste your goal, paste any constraint you can think of, then ask for a multi-phase plan with checkpoints. It works for far more than ML: a feature spec, a competitor breakdown, a backend system you want to understand, a cooking technique, a workout program, a Spanish grammar topic you’ve avoided for years. The plan gives you a runway and gives the LLM a frame. The first conversation in this entire build was exactly this move. Paste the Kaggle dataset and the PM context, get a 7-phase plan back. Every other technique below assumes the plan exists.

Drop a screenshot before you describe. The LLM reads UI state, not just text.

In Phase 1, before any error had happened, the LLM noticed “Python 3.13.3 in the bottom-right corner” of a VS Code screenshot and flagged that one library (xgboost) might need extra handling on that version. It read the kernel selector, the toolbar, the Explorer panel. Save the typing. Paste the picture and let the model do the inspection.

Treat errors as conversation, not failure.

When import pandas as pd returned ModuleNotFoundError: No module named ‘pandas’, the response wasn’t “you broke it.” It was a metaphor and a fix.

“The libraries are installed, but the notebook’s kernel is using a different Python than the one where we installed them. Think of it like installing apps on your phone but trying to run them on someone else’s phone.” Then a one-line install fix and a “restart the kernel” instruction. Paste the full error verbatim. Ask for the explanation, not just the fix.

Ask for per-line comments on every code block. “Rewrite this snippet with a comment on every single line. I want it to read like a tutorial when I open the file next week.” Those comments became my study notes. They’re also the lines I’ll quote when someone asks how the code works six months later. The pattern is portable: any AI-generated artifact (SQL, config file, spec doc) can be made to explain itself line by line.

Ask the same question three different ways. The LLM doesn’t get frustrated. “Explain this like I’ve never seen code.” “Now as a PM, not an engineer.” “Now with a real-world analogy.” Three takes on the same concept. Keep the one that lands.

Push back when the plan misses something. The original plan in this build skipped GitHub versioning entirely. Phase 1 set everything up locally with no remote backup. I stopped mid-phase, said so plainly, and the plan adjusted immediately (”Great instinct, that’s exactly what real developers do”). You always know your context better than the LLM does. When a gap shows up, name it. The LLM has no ego about being wrong; it integrates the correction without friction.

One warning: when you say “I’m a beginner,” the LLM may oversimplify. It might skip edge cases, skip error handling, or give a version that works for the tutorial but wouldn’t hold up in production. That’s exactly why the second-opinion session above isn’t optional. One session teaches you, a second session audits what you learned. Treat them as a pair, not as alternatives.

The felt experience: what changed for me

Two observations on what working this way actually feels like, hour after hour:

• The encouragement is real. When the LLM says “great instinct” or “you caught that yourself before I even explained it fully,” it’s flagging a moment when you did something a beginner wouldn’t normally do. The recognition isn’t decorative. Over the course of an evening’s focused build, those moments add up to a low-grade momentum that’s hard to get from a textbook or a video course.

• You can reframe any output in any direction. “Read this output like a PM” worked dozens of times in this build. The same numbers land completely differently depending on the lens: raw accuracy versus retention budget impact versus board-ready story. The LLM switches frames on request, and the second framing is usually the one your team will actually understand.

What this style of learning quietly removes that traditional self-taught paths never could: the loneliness of being stuck, and the rigidity of pre-recorded explanations.

What AI changes about learning

Step back from the build for a moment. One PM, one project, one evening. The bigger story is what’s actually different about learning today, anchored in the same person trying to learn the same kind of thing, ten years apart.

The contrast: same brain, ten years apart

Ten years ago. A 30-hour Udemy course bought on a whim, half-watched in evenings. Stack Overflow for the parts that broke, YouTube comments for the parts that confused. No one to ask the obvious dumb question. No one to translate the textbook into plain English. No one to hold the rope when a side project stalled out. Most people who started never finished. Most who finished couldn’t transfer what they learned to their actual job.

This year. A one-year program at the Sorbonne, evening classes alongside a full-time PM job. The course itself was good. But the actual learning happened in a different place than the diploma suggested it would.

I recorded every lecture, transcribed the audio, and pasted the transcripts into Gemini for cleaner notes with the gaps filled in. Once a unit was done, I dropped everything into NotebookLM and let it generate a podcast and a slide deck to replay on the commute, in the gym, between meetings. And through all of it, I kept a continuous conversation going with Gemini: questions about the lecture content, metaphors that helped a hard concept land, code snippets explained line by line, deeper discussions hours after the actual instructor had gone home. **A 24/7 patient tutor in the pocket.**

The diploma was the scaffolding. The AI was the actual classroom.

The instructor’s new job

This isn’t an argument that AI replaces teachers. The instructor still does work that AI can’t. They tell you what’s possible, including the things AI won’t volunteer because you didn’t know to ask. They give you the principles you need to challenge AI output. They catch the leaks (in this build’s case, a literal data leakage in the model) that you’d miss because you don’t yet know leaks exist.

The four-minute mile is the parallel that fits here. On May 6, 1954, Roger Bannister ran a mile in under four minutes, something the medical community genuinely believed was physically impossible. Within a year, dozens of runners broke the same barrier. Knowing something is possible changes what people attempt. A good instructor today is mostly in the business of expanding what their students believe is possible, and trusting AI to handle the parts that used to require lectures.

One last thing. I’m joining the training team at the Sorbonne on the PM side after this year. Knowing every future student will have a 24/7 patient tutor in their pocket will probably change how I teach. More time spent on what AI can’t do (naming what’s possible, framing real problems, defending judgment calls, holding the rope when a student is stuck on something that has no clean answer) and less on what it now does for free. The job description of “instructor” is being rewritten in real time, and getting to be on both sides of that change at the same moment is one of the strangest and most interesting things happening in education right now.

Start your first project this evening

The bar to start has never been lower. One dataset, one LLM chat session, one question: “give me a beginner-friendly plan.”

Don’t do this for the joy of learning. Do it the next time a real question lands on your desk, the kind that would normally turn into a ticket and three weeks of waiting for the data team. Why are users dropping off, what’s driving conversion, where the support tickets are spiking, which features correlate with retention, which support categories are growing fastest. Pull the export yourself. Ctrl+A, Ctrl+C. Open an LLM chat, not a coding assistant. Paste it in along with what you’re trying to figure out. Ask for a beginner-friendly plan. Spend an evening on it. You probably won’t ship the analysis. You’ll walk into the next meeting with sharper questions than the people still waiting in the queue.

The public repo for this build: github.com/adamfaik/churn-prediction. Everything is there if you want to see what the end looks like before you start.

Here’s the part that still surprises me. I didn’t set out to finish it. That afternoon, I was working on something else in Claude Code and it was taking its time on a slow task.

I could have scrolled Instagram. Instead, I opened a second VS Code window, pulled up the repo of that abandoned book project, and started poking around. Just to have something to do while another window was busy.

A few hours later, I had compared twenty-one chapters to the original 1683 source, added about a hundred and forty historical annotations (each one with a cited source), and published the whole thing on GitHub Pages. I had not meant to finish the book. I meant to have something to do while another task was thinking. And the work just... happened.

It’s about the shift in how I work. How I went from thinking of Claude Code as ”that thing engineers use for code” to seeing it as a way to run any project where the same rules have to be applied over and over. Including, apparently, a 17th-century autobiographical memoir and a slice of French-Moroccan history.

Here’s what the rest of the article covers, in order:

1. The use case: a forgotten 1683 memoir. The book, what makes it worth a modern edition, and the personal story behind it.

2. First try: how I used Gemini. The first attempt: what worked, what the tools got right, and why I stopped anyway.

3. The click: rules in a file, not a chat. The moment the mindset shifted: from re-pasting rules into a chat to encoding them in a file.

4. How I finished the book: four steps. Ten editorial rules defined in one morning, three Skills built in sequence, twenty-one chapters run in parallel.

5. Spot skills in every project. The three-question test and how the same shape shows up in any product week.

Let’s open the book.

The use case: a forgotten 1683 memoir

I found it on archive.org one evening when I was bored and typed ”livre du Maroc” into the search bar. A pretty chaotic way to stumble on a book. The title that came back stopped me cold: Relation de la Captivité du Sieur Moüette dans les Royaumes de Fez et de Maroc.

A Frenchman. Eleven years held captive in Morocco under Moulay Ismaïl. Published in 1683. I’ve always been drawn to the history of both countries, and a first-person 17th-century memoir from someone who lived both is not a book you scroll past. I downloaded it instantly.

The first page was almost unreadable. Not just old French. Pre-1800 French, with the long-s (ſ) instead of s, and spellings and abbreviations that make your eyes bounce off the page after two sentences. I gave up after three pages.

A week later I tried again, and something strange happened. By the end of chapter one I was reading it almost normally. Your brain does an involuntary translation once you’ve seen enough of the pattern. It’s a little like those well-known scrambled-letter paragraphs where the first and last letters are right and everything in the middle is a mess. Same phenomenon, slower setup.

I finished the book over a long weekend. Two things hit me.

The first was the modernity of it. Moüette is talking, in 1683, about things that still feel recognizable in 2026. The politics of North Africa. The economics of trans-Mediterranean trade. The porous borders of belief and identity. The names change. The shape of the conversations doesn’t. That’s the part you never get from a history textbook. Read through someone’s lived experience and the whole period comes alive from the inside. You don’t learn about the 17th century. You sit next to someone who lived it.

The second was that Moüette is a weirder narrator than you’d expect. He was taken captive in the 1670s and then decided, somewhere along the way, to treat his captivity as fieldwork. He saw himself as an observer. He believed France and Morocco were eventually going to have to figure out how to talk to each other, and he wanted to come back with something useful for that conversation. A kind of self-appointed diplomat, writing notes in chains. That’s a strange posture to hold, and it’s the reason the book is as good as it is.

And then there’s the personal layer, which is the part I almost kept to myself. I grew up in Morocco. I moved to France to study and then to work. Moüette did the reverse trip, involuntarily, in 1670. I’m not going to overstate the parallel (I’m not, to be clear, a captive in France, thank you for asking), but reading his memoir felt like meeting someone who made my exact trajectory 350 years earlier, in the opposite direction. That’s why the book kept tugging at me months after I’d finished it.

Here’s what I started thinking. A book like this deserves more than a scan sitting quietly on archive.org, smelling of basement. A modernized French edition is the minimum. Past that, it deserves a bande dessinée, a Netflix series, and honestly a Scorsese film, but I’m willing to negotiate. It’s a tight, weird, human account of a period most of us only ever learn in abstractions.

I checked Amazon. Nobody had made a modern edition. If the book was going to get one, I was going to have to do it myself.

First try: how I used Gemini

First attempt: chapters one by one

My first attempt was months earlier, and it was all done in Gemini. I’d open the PDF, copy a page, paste it into the chat, and ask for a modern French transcription. Genuinely, I did not expect this to work.

The first time it did, I was a little stunned. Even with archaic spellings and garbled OCR in the source, Gemini came back with readable, well-formed paragraphs. I kept going, chapter by chapter, pushing several pages at a time thanks to the big context window.

The result was a working first pass. Chapter by chapter, Gemini turned the archaic French into readable modern French, formatted as markdown. Each chapter came out as a structured file: headings, paragraphs, footnote markers in place.

Not perfect. There were omissions and meaning drifts I’d only discover much later. But it was a genuine starting point. A folder of modernized text, ready to become a book.

The most fun part was the Arabic lexicon at the end of the book. Moüette had included a small French / dialectal-Arabic dictionary. Not classical Arabic. Street Arabic from 17th-century Morocco. Running it through Gemini, I realized a lot of those words are still used in Moroccan darija today. A tiny time capsule of spoken language that survived 350 years.

I want to be honest about what I was actually doing here. I wasn’t using Gemini because it’s ”the right tool for OCR.” I was using Gemini because it was the AI I knew, and chat was the only mode I knew how to work in.

Open a tab. Write a prompt. Hope the model keeps the context. Paste a page. Paste another page. Thank the gods for the big context window. Everything I built in this phase, I built in chat mode. One long conversation per chapter. Rules re-explained in each new tab. No memory between sessions.

That worked. But it hit a ceiling I couldn’t see from inside it. Picture the whole project being held together by three hundred Gemini tabs in my favorites bar and my own willingness to keep pasting the same block of instructions into each new one.

Use a standard tool, not custom code

When I eventually asked Gemini how to publish the book as a website, I expected ”let’s build a small React app.” Instead, Gemini pointed me at VitePress, a static-site framework that turns markdown files into a clean, navigable, book-shaped site. I’d never heard of VitePress. I also hadn’t asked. I was about to ask the model to build me something custom, and it quietly told me there was a standard tool for this job.

That’s the lesson I keep coming back to. Before you ask an AI to vibe-code a custom thing, ask it the opposite question first:

Is there a standard framework, library, or tool that already does this? If there is, use it.

Most of the time, the answer is yes. A few examples I’ve hit since:

• For a book or documentation site, VitePress or Docusaurus.

• For a quick internal data dashboard, Streamlit.

• For a training or course site, Docusaurus or MkDocs.

• For an LLM prototype with a simple UI, Gradio or Streamlit.

• For presentation slides generated from a structured document: Marp (markdown → slides), Beamer (LaTeX → slides), or a small Python script with python-pptx if you need a programmatic PowerPoint.

Off-the-shelf tools are boring. They’re also well-tested, documented, styled by people who actually care about typography, and usually fifteen minutes of setup. Custom code is seductive because the model will happily generate three hundred lines of it. Resist for a second. Ask the ”is there something standard?” question. Often the answer saves you from a pile of maintenance you don’t want.

The site build: six steps with Gemini

Once VitePress was on the table, I asked Gemini to walk me through it. Nothing magical happened here, but the process might be useful if you’ve never set up a static-site project before.

Roughly how I did it:

1. Created a new empty repo on GitHub for the book project. No git-clone terminal dance. I made the repo on github.com directly, added a README from the web interface, then downloaded the repo as a ZIP when I needed the files locally.

2. Followed VitePress’s official setup instructions with Gemini narrating them in plain French as we went. The scaffold gives you a configuration file, a home page, and a docs/ folder where your markdown chapters live. Same idea as a Word document with a table of contents, except each chapter is its own small file.

3. Created one markdown file per chapter, calmly, one at a time, pasting in the modernized French that Gemini had produced during the earlier chat sessions. Files named preface.md, chapitre-1.md, chapitre-2.md, and so on.

4. Asked Gemini to generate a cover image for the modern edition. Something with period typography and a hint of 17th-century North Africa. It produced something decent on the second try. That became the book’s homepage art.

5. Asked Gemini to help adjust the VitePress theme so the site felt closer to the book’s period. Historical serif fonts that are still readable on screen. A slightly warmer palette than the default. A navigation sidebar with the chapters in order. I described the mood (”old paper, candle light, but readable on a 2024 laptop”) and Gemini produced a CSS file that matched it reasonably well.

6. Deployed to GitHub Pages, again with Gemini’s instructions, which is where I hit a small hiccup. GitHub Pages will happily render your README as the homepage by default, not your VitePress build. You have to flip a setting in the repo’s Pages tab to tell it to use a GitHub Actions workflow instead. Fifteen minutes, not fifteen seconds, but once flipped it stayed flipped.

By the end of an afternoon, I had a private, working, scrolling book-shaped site. A third of the chapters modernized. A cover image. A period-appropriate theme. A deploy pipeline. Not published publicly, but running.

The stall: out of time and no plan

And then, for a few reasons, I stopped.

The first reason was the content itself. Moüette is a 17th-century Catholic writing about people he considers his enemies, and the vocabulary shows it. I didn’t want to push something into the world without knowing how to handle the parts that would land badly.

The second reason is more boring. I ran out of time and momentum. I had a decent VitePress scaffolding, a cover, maybe a third of the chapters done, and no clear path to the finish line.

The twenty pieces left to modernize and review felt like a lot of evening hours, and not one of them felt especially fun. I told myself I’d come back to it when I retired. Then I closed the tab.

It sat there for months.

Looking back, stopping was the right call. If I’d kept grinding through chapter-by-chapter pastes in Gemini, I’d have spent weeks on something that eventually took an afternoon. The problem wasn’t the effort. It was the shape of work. I didn’t even know skills were a thing.

Leaving the project in a drawer meant I came back to it after I’d understood something new. Put it down. Let your understanding catch up. Then open it again.

The click: rules in a file, not a chat

Back to the afternoon. Second VS Code window open, abandoned repo loaded, I typed a request. Something like: ”can you compare this modernized chapter to the original 1683 source and tell me what went wrong?” Claude Code read both files, thought for a minute, and came back with a list of differences. Missing paragraphs. Odd word choices. A few proper nouns spelled three different ways across the chapter.

That alone was useful. But while I was reading the output, a smaller thought hit me: I’m going to want to do this exact thing on the next chapter. And the one after that. With the same rules.

In Gemini, that would have meant opening a new chat, re-pasting the rules, re-explaining the context, every time. I have a folder of three hundred bookmarked Gemini conversations that exists exactly so I can find the rules I wrote down somewhere and paste them into the new chat. That’s a lot of effort for something that’s really just saying ”apply these rules.”

What Claude Code let me do instead was create a file. A small markdown file, in a folder called .claude/skills/, with a name and a short description and the rules written out. Once that file existed, I could type /review-chapter chapitre-2 and Claude Code would just... apply the rules. No re-pasting. No re-explaining. The same behavior every time.

The difference isn’t the AI. It’s the container. A rule typed into a chat window lives exactly as long as that chat window is open. A rule written into a file outlives the conversation. It travels with the repo. It applies the same way next month as it does today.

That was the click. I wasn’t writing a prompt anymore. I was writing a rule. The rule lived in a file. The file lived in the repo. Next week, next month, next year, the rule would still be there, applying itself consistently to whatever chapter I pointed it at.

That’s the thesis of this article, and I’ll say it plainly: any project you’d run on a whiteboard can be run in a repo. Not just code. Any project where the same rules have to get applied to multiple inputs. A book. A stack of stakeholder updates. A folder of user-interview transcripts. A weekly research synthesis. If you’ve ever pasted the same block of context into a fresh chat window twice, you’ve hit the shape of work that Skills solve.

How I finished the book: four steps

Once the mindset shifted, everything else followed in sequence. Four steps, each one unlocking the next.

Define your rules once

Before writing any Skill, I needed to know what the rules actually were. I had a vague sense of what the modernization should feel like: preserve Moüette’s voice, handle the old typography, keep the unusual words where they belonged. A feeling isn’t a rule, and a rule you can’t write down in one sentence can’t live in a skill file.

So Claude Code and I spent a morning doing something very simple. One question at a time, with options.

A typical exchange looked like this:

Claude Code: For the author’s voice, which of these three feels right?
(A) Keep the original phrasing exactly, even when archaic.
(B) Allow light rephrasing for clarity, but preserve personal interjections and hedging.
(C) Modernize fully for readability, voice can drift.

Me: B. If he says ”je ne sais combien de déserts”, that stays. It never becomes ”de nombreux déserts”.

Claude Code: Got it. Locking that as rule #1: voice preservation, light rephrasing allowed, personal interjections and hedges always kept.

That’s the pattern. One question, 2–3 concrete options, pick one, lock it. Repeat until you have a list.

Try it the opposite way and you’ll freeze. I know this because I’ve frozen. ”What are your editorial rules?” is an impossible question. ”Given this paragraph, would you prefer A, B, or C?” is a simple question. Options force judgment. Judgment produces rules.

By the end of the morning, I had ten. Three of them took real back-and-forth.

• Voice preservation was first. The reason it took ten minutes, not two, is the example I had to nail down: ”je ne sais combien de déserts” can’t become ”de nombreux déserts.” That hedged, slightly shy way of describing an overwhelming thing is who Moüette is as a narrator. Smooth it out and you lose the person.

• Handling omissions. Gemini’s first pass had quietly left some paragraphs out. The question: reintegrate them silently (A), add editorial brackets to signal they’d been restored (B), or leave the gaps and note them in a footnote (C). I picked A. Moüette wrote those paragraphs. They belong in the text. An editorial bracket interrupts the reading experience with a scholarship signal most readers didn’t ask for.

• Sensitive terminology was the uncomfortable one. Moüette uses vocabulary that a 17th-century French Catholic writing about his captors would use. Options: modernize the terms (A), keep them exactly as written and contextualize via annotations (B), or remove the worst instances (C). I picked B. Changing the vocabulary would be changing the historical record. The annotations explain without excusing. I’ll come back to how this made it possible to ship.

The other seven took much less time. Some were thirty seconds: footnote format was just ”which syntax does VitePress support?” Done.

Once the ten rules existed, I saved them in three places. Inside project memory (so Claude Code would always see them). Embedded inside each skill (so the skill carried the rules with it). And in a reader-facing editorial note on the published site (so anyone reading the book could see exactly what was done to the text). Three homes for the same rules. Each one earning its place.

The three skills: review, apply, enrich

I didn’t start by designing three Skills. I started by needing one answer: “what’s actually wrong with what Gemini produced?”

The first skill I built was review-chapter. I described what I wanted: read a modernized chapter, read the corresponding section of the 1683 source, compare paragraph by paragraph, flag every problem. Then I wrote the skill.

I ran it on the preface. The output was immediately useful and immediately humbling. Twelve instances where Moüette’s personal voice had been smoothed away. The exact sentence I’d noticed (”je ne sais combien de déserts” quietly rewritten as ”de nombreux déserts”) was flagged, with the source passage and a suggested fix.

I ran it on chapter 1. Same categories: omissions, meaning shifts, OCR artifacts, proper nouns spelled inconsistently. I had a diagnosis.

What I didn’t have was a way to fix all of it consistently. And that’s when the connection clicked. I had just spent a morning writing ten editorial rules. Those rules were sitting in project memory. ”I have a set of rules I want applied to every chapter. That’s exactly what a Skill is for.” So the second skill wrote itself out of the first. That became harmonize.

Running both skills across all the chapters left me with a clean text. But clean doesn’t mean alive. A modern reader hitting Moüette’s offhand mentions of ”les corsaires de Bouregreg” or ”les Marabouts” or ”la milice des Renégats” gets nothing: no frame, no context, no reason to care. So a third conversation started: ”what if we added a short sourced annotation at the end of each chapter?” That conversation produced annotate.

Three skills, each one discovered by using the previous one (browse in the GitHub repo).

Here’s what each one actually does:

• review-chapter: the diagnostic. Reads the modernized chapter, reads the corresponding section of the 1683 source, compares paragraph by paragraph. Produces a structured report across eight categories (A–H): omissions, meaning shifts, unauthorized additions, proper-noun inconsistencies, modernization inconsistencies, OCR artifacts, paragraph structure, footnote candidates. Each issue gets a location, a quoted source passage, a problem statement, and a suggested fix. The report ends with an overall grade: Good / Needs minor fixes / Needs significant rework. Read the full SKILL.md here.

• harmonize: the rules-applier. Applies all ten editorial rules using the edit tool to touch only what needs touching, never rewriting the whole file. Voice preservation, silent reintegration of omissions, proper-noun normalization, footnote discipline, paragraph-structure rules, sensitive-vocabulary preservation. All embedded in the skill itself. When I run /harmonize chapitre-7, it doesn’t just “modernize” the chapter. It applies the whole editorial policy. Read it here.

• annotate: the one with the strictest rule. Appends a ”Repères historiques” section to each chapter with five to ten contextual notes. Each note: a brief quote from the chapter, a 3-6 sentence explanation, a cited source. Every factual claim must be verified via WebSearch before it’s written. No exceptions. If the model can’t verify something, the note either gets dropped or explicitly marked as uncertain. Academic sources preferred. The tone target is ”a knowledgeable friend, not a textbook.” Read it here.

The run: 21 chapters in parallel

Once the three Skills existed, I had to run them across the whole book. That’s 21 files: the preface, the royal privilège, fifteen chapters, a commerce treaty, the Arabic lexicon, a geography annex. I could have run them one at a time. I didn’t.

I did the first chapter in full. Ran /review-chapter chapitre-1, read the report, ran /harmonize chapitre-1, read the diff, ran /annotate chapitre-1, read the annotations. Then I stopped and asked myself a simple question: does this look right? It did. Then I fanned out. Claude Code launched sub-agents (parallel workers, up to eleven at a time for the annotation pass) and pointed each one at a different chapter.

The pattern is worth naming, because it stops you from blowing up a whole project on a flawed first run:

Demo one. Approve it. Then fan out.

The run itself was genuinely fast. Minutes, not hours. Here are the numbers across all 21 files:

• 412 insertions, 253 deletions committed across 22 files

• ~70 footnotes added

• ~140 sourced historical annotations written

• Up to 11 concurrent sub-agents during the annotation pass, 6 during review/harmonize

I want to be honest about the timeline. The ”I finished it in under an hour” line was the felt experience, not the actual clock.

The run took minutes. The work took two days. That morning spent defining rules. An afternoon fixing a few mistakes. Another morning reviewing annotations, rewriting the editorial note, and publishing the site.

Skills aren’t magic. They’re leverage. The run is fast because the rules are sharp. The rules take a morning.

What the annotations surfaced is the part I keep bragging about at dinners. The Republic of Bouregreg, a corsair city-state that existed between 1627 and 1668. Barbary lions, which roamed North Africa until they went extinct in the wild in 1942.

The fleurs de lys carved by French slaves into the walls of Moulay Ismaïl’s palace at Meknes. The Order of the Merci’s “blood vow”, a fourth monastic vow to offer oneself as a replacement for a captive who couldn’t be ransomed. I knew none of this before the annotate skill found it and sourced it. The annotations made the book twice as alive.

What made it shippable: values in the skill

One of the reasons the book had sat in a private repo for months was the content itself. Some passages are genuinely uncomfortable by today’s standards. I wasn’t sure I wanted my name on a site hosting that content without some kind of frame.

The annotate skill gave me the frame. Its job on those passages wasn’t to censor them or smooth them over. It was to contextualize them. Sourced notes. Historical framing. Scholarly context that helps a modern reader understand what they’re looking at and why it’s there. The content warning on the homepage tells readers what they’re walking into before they open the first chapter.

That combination (a content warning and annotations that explain rather than erase) was what finally made me feel ready to ship. A skill can hold values, not just rules. The whole policy lives in a file. It can be inspected. It can be argued with. That’s what I want from a system I put my name on.

What I’d change: use skill-creator

Only one thing. Next time, I’d open Anthropic’s skill-creator skill before writing a single file. It’s a skill that helps you write other skills. You describe what you want, it scaffolds the folder, the SKILL.md, the frontmatter, all following current best practices. I didn’t know it existed when I built my three, so I learned the hard way, through trial and Claude Code’s diagnostic errors. It worked out fine. But if you’re about to build your first skill, skill-creator is the shortcut I wish I’d taken.

Spot skills in every project

After the book was live, I sat down to write a stakeholder update at work and caught myself thinking: ”wait, this is a skill.” Not a lightning bolt. A small, quiet ”oh, this too” about a task I’d been doing the hard way for years.

The book didn’t teach me skills. It taught me to see them. Here’s the practical test I started applying. Three questions, in order:

1. Am I re-explaining the same rules over and over? If you’ve typed the same block of context into a fresh chat window more than twice, those rules belong in a file, not in your head.

2. Is there a checklist living in my head that I never wrote down? The quality bar you apply before sending something, the three things you always catch, the format you always enforce. That’s a skill waiting to be written down.

3. Does the output need to look the same shape every time? Same structure, same tone rules, same format, week after week. That’s the exact shape a skill solves.

Applied to a product week, the list shows up fast. I wrote about one of these in detail already: /customer-pulse.

That skill searches reviews, forums, and app stores for what users are saying, and returns a structured report with ranked themes, real quotes, and a “Top 3 this week” action list. If you want the step-by-step on building a skill from scratch, any Skill, that’s where to start.

The same pattern shows up across the rest of a product week:

• Customer feedback synthesis. Multiple sources, grouped by theme, real quotes, source links. The rules you re-apply every time are exactly what a skill holds.

• Stakeholder updates. Same structure every week, same length target, same tone rules. Skill-shaped.

• User-interview theming. A folder of transcripts, a definition of what a good theme looks like, quote extraction with attribution. Skill-shaped.

• Competitor monitoring. A list of URLs, a definition of what change looks like, a weekly diff. Skill-shaped.

• Meeting recaps. Your team’s template, your tone rules, action items always in the same format. Skill-shaped.

Once you answer yes to one of those three questions, you’re looking at a skill. The only question is which project you want to start with.

Start with a project you already know

The book took an afternoon to finish. The afternoon started because another task was thinking and I had a spare VS Code window. That’s the whole origin story.

It’s still not finished, if I’m honest. I’d like to generate comic-book panels with Gemini Nano Banana Pro to tell Moüette’s story visually. Maybe a short film, once video models get a bit cheaper. Another weekend, another skill.

Your turn. You don’t need a 17th-century book. You just need one project where you already know what good looks like, and where you’re tired of re-explaining it. Open it. Ask one question: “what rule am I re-pasting into every fresh chat?” Write it down. Put it in a file. That’s the first skill.

Then something shifted. I saw product people getting genuinely excited. Not about the concept, but about what they were building with it. So I carved out an afternoon, sat down, and actually built one from scratch. And it changed how I think about my work.

The skill I built is called /customer-pulse. It searches the web for what real users are saying about a product (app store reviews, Trustpilot, forums, comparison articles, etc.), then produces a structured report with ranked themes, real quotes, blind spots, and concrete actions. Everything I built is in this GitHub repo. Clone it, explore it, use it for your own product.

A quick definition: a skill is a set of instructions written in plain English that lives in your project folder. When invoked, Claude reads those instructions, reads your files, uses real tools (web search, file creation, page fetching, etc.), and produces real output. Unlike a custom GPT or a Gem, it lives in your repo. You own it. You version-control it. It evolves with your project.

Here’s what we’ll go through in this article:

• Level 1: Download the repo and run a skill. Get Claude Code installed, download the project, run the skill, and see a real feedback report.

• Level 2: Understand what skills are. Open the skill files and learn the building blocks: SKILL.md, product context, CLAUDE.md. See how skills compare to custom GPTs, NotebookLM, and saved prompts.

• Level 3: Build your first skill step by step. The real story: how I described my problem to Claude Code, the review process that caught 5 issues, and the iteration that turned a generic report into an actionable one.

• Level 4: Go further. Where to find existing skills, what MCP servers and agents are, and when skills are the wrong tool.

Each level is self-contained. Stop at Level 1 and still walk away with a working tool. Go to Level 3 and walk away knowing how to build your own.

Why this use case

Read the full skill instructions. It’s plain English, no code. That file is the entire skill.

I chose feedback collection because it’s something product people will always do. No matter how much AI disrupts our work, understanding what users actually experience remains a human decision. AI can collect, synthesize, and structure the data faster. But deciding what matters is still ours.

I considered building skills for other PM tasks: PRD generation, user story refinement, sprint review preparation. I dismissed them. Not because they’re bad ideas, but because I’m genuinely unsure those artifacts will exist in the same form much longer. When AI can build a working prototype from a conversation, do we still need a 15-page specification document? Maybe the future is less specifying and more building. And maybe “building” increasingly means creating skills that do the work, which happens to be the topic of this article.

Feedback collection, though, stays. Users will always talk. The question is whether we listen fast enough.

Level 1: Download the repo and run a skill

Allow about 30 minutes for this level. At the end, a full customer feedback report will be waiting in your project folder.

Step 1: Open your code editor

Open VS Code, Cursor, or any code editor that has a built-in terminal. If you don’t have one, download VS Code for free from code.visualstudio.com.

Once it’s open, find the built-in terminal. In VS Code: go to Terminal → New Terminal. A panel appears at the bottom.

One tip that made a big difference for me: move the terminal panel to the right side. In VS Code, right-click the terminal header bar, select Panel Position → Right. This puts your files on the left and Claude Code on the right. Much easier to work with. I spent way too long with it at the bottom before discovering this.

Step 2: Install Claude Code

So what’s Claude Code exactly? It’s an AI assistant that runs inside your terminal. Unlike ChatGPT or Gemini, it doesn’t live in a browser. It lives in your project folder. It can read your files, search the web, create documents, and run skills. Think of it as an AI colleague that sits inside your project and actually does things, not just talks about them.

One thing to know upfront: it requires a paid Anthropic plan. The Pro plan is $20/month and the Max plan is $100/month. There’s no free tier for Claude Code. It’s worth it, but it’s good to know before installing.

To install it, paste this command into the terminal and press Enter:

curl -fsSL https://claude.ai/install.sh | bash

That’s it. One line. This comes straight from the official quickstart guide.

Now here’s something that trips up a LOT of people, and I want to save you the confusion. The terminal has two modes. Right now it’s in bash mode. That means it runs system commands, like the install command you just pasted. But when you launch Claude Code, it switches to conversation mode. In conversation mode, you type in natural language, like talking to a colleague. You invoke skills with the / prefix. So /customer-pulse is a skill. curl is a bash command. Don’t mix them up. Two modes, two purposes.

If you want to get familiar with some Claude Code commands and shortcuts, there’s a nice community cheat sheet that covers the essentials.

Close and reopen the terminal (or open a new one) so it picks up the new command. Then type:

claude

The first time, a browser window opens to log in with your Anthropic account. After that, Claude Code starts directly.

Step 3: Download and open the project

Exit Claude Code for now (type exit or press Ctrl+C). Go to the product-feedback-skill repo on GitHub. Click the green Code button, then Download ZIP. Unzip the folder wherever you keep your projects (for example, your Documents folder).

Now in your editor, go to File → Open Folder, navigate to product-feedback-skill, and click Open. The left sidebar (the explorer) shows all the project files.

Step 4: Run the skill

Open a new terminal in your editor (Terminal → New Terminal) and make sure it’s inside the project folder. Then launch Claude Code:

claude

The repo includes product context files for two products: Figma and Slack. Let’s try Figma. Type:

/customer-pulse figma

And then just watch. It’s genuinely fun to see. Claude reads the Figma product context, starts searching the web (Trustpilot reviews, forum threads, comparison articles), fetches the most relevant pages, extracts real user quotes, groups them into themes, scores each by severity, and writes a full report to reports/. The whole thing takes a few minutes because Claude is making 12 web searches and fetching up to 20 pages. Just let it do its thing.

When it’s done, open the generated report in the reports/ folder. Right-click the file tab → ”Open Preview” to see it formatted instead of raw text.

One command. A full customer feedback analysis with source URLs, severity rankings, sentiment direction, and a “Top 3 things to do this week” action list. Try /customer-pulse slack for the other product.

Level 2: Understand what skills are

OK so the report just appeared. That’s cool. But what actually happened behind the scenes? Three files made the whole thing work. Let’s open each one.

Step 1: Open the SKILL.md

In the explorer sidebar, navigate to .claude/skills/customer-pulse/SKILL.md. Click it. Or read it directly on GitHub.

Take a moment to read it. This is the entire skill. A Markdown file. No code. No API calls. No special syntax. Just English. If you can write an email, you can write a skill. Seriously.

Right-click the tab → ”Open Preview” to see it formatted. That’s easier to read once you understand the structure.

It has two parts. The header (called frontmatter) controls how the skill behaves. The settings that matter most:

• description: how Claude decides whether to suggest the skill. Vague descriptions like “analyzes stuff” don’t work. Specific ones like “use when the user wants to understand customer sentiment” do. It’s like giving a teammate a proper job description.

• disable-model-invocation: true: prevents the skill from auto-triggering. For a skill that makes 30+ web calls, you want explicit control.

• allowed-tools: lists tools Claude can use without asking permission each time. Without it, Claude pauses 20+ times to ask. With it, the skill runs smoothly.

• $ARGUMENTS: when you type /customer-pulse figma, “figma” replaces $ARGUMENTS in the instructions.

The body tells Claude what to do, step by step, in plain English. The official skills documentation has the full reference.

When does it make sense to build a skill? Tasks that are repeatable, multi-step, and benefit from context. If you’d need to re-explain the same context more than twice, it’s worth encoding as a skill. For one-off tasks, just ask Claude directly.

Step 2: Open the product context

Navigate to product-context/figma.md. Or read it on GitHub.

This file describes Figma: features, customer segments, pricing, known pain points, competitors. The skill reads it before searching. That’s why the report isn’t generic. Without product context, you get “users complain about performance.” With it, you get “the March 2026 regression affected component manipulation specifically, compounding the long-standing 2GB memory cap issue that hits enterprise teams with variant-heavy design systems.”

This separation is the clever part. The SKILL.md defines the process. The product context defines the knowledge. Change the context, and the entire analysis shifts. Without touching the instructions.

If you were doing this at work, your context file could include customer personas, Jira backlog categories, OKRs, NPS themes, competitive analysis, and roadmap features. The richer the context, the smarter the analysis. The skill doesn’t care where the knowledge comes from. It just reads the file.

Step 3: Understand how skills compare to other tools

A saved prompt. Every time you paste your product description and analysis request into ChatGPT, you’re doing manually what a skill does automatically. A skill reads your files for you. The context is always there. No pasting.

A custom GPT or Gem. Lives in someone else’s cloud. Can’t read your local files. Can’t create documents in your repo. A skill is a Markdown file you own, version-controlled with git.

NotebookLM. Same “rich context” idea, but limited to answering questions and generate a certain type of content. Skills can search the web, create files, and produce reports that feed into your next workflow. Skills are rich context plus real actions in the same place.

Perplexity. Great for one-time answers. A skill gives you a versioned, repeatable process that improves over time. A Perplexity search doesn’t compound. A skill does.

Skills also work beyond Claude Code. You can use them in claude.ai (upload as zip) and in the Claude desktop app with Cowork. Claude Code is the most powerful surface though, with full network access and features like allowed-tools.

Step 4: Open the CLAUDE.md

Last file. Open CLAUDE.md at the project root. Or read it on GitHub.

A README.md is for humans. CLAUDE.md is for Claude. It tells Claude what the project is about, what files matter, and how things connect. Claude reads it automatically at the start of every session. Create one for any project by typing /init. Keep it under 30 lines.

Level 3: Build your first skill step by step

If you’ve read my previous post on how to save +5 hours a week with Gems, the process here will feel familiar. Same loop: ask the AI to write the instructions, test them, see what’s wrong, improve them, test again. With Gems, you do this in a web interface. With Claude Code skills, you do it inside your project folder. The principle doesn’t change. The power does.

Here’s how I actually built /customer-pulse. Not the clean version. The real one. Before the details, here’s the overview:

1. Describe the problem. No technical instructions. Just explain what you need.

2. Review the skill. Ask Claude Code to check its own work.

3. Run it and evaluate. See what’s good, what’s missing.

4. Improve the instructions, not the output. Fix the SKILL.md so every future run is better.

5. Test on a different product. Make sure it generalizes.

Step 1: Describe your problem

I opened Claude Code and typed:

I’m a product manager and one of my jobs is to explore customer feedback across different platforms (app stores, review sites, forums) so I can make better decisions and understand the problems our users are facing. But it’s time consuming. I sometimes miss important sources, and I struggle to synthesize and identify the most actionable feedback. I was wondering if it makes sense to create a system here that helps me do that every time. Feel free to ask me any questions.

I didn’t say “create a .claude/skills/ folder.” I described a problem. You don’t need to know terminal commands. You need to know how to describe a problem clearly. That’s a skill product people already have.

Claude asked clarifying questions: what product, what sources, what format. I said Figma. It proposed a 4-phase architecture with a scoring system and guardrails. Then it built everything. I just watched files appear in the VS Code Explorer.

Step 2: Review the work

Here’s a workflow I wish someone had told me about earlier: the double-loop. One AI session builds. Another reviews. You make the decisions.

I asked Claude Code in another session to check the skill against best practices. It found five issues: lowercase file name (should be SKILL.md), no $ARGUMENTS support, missing disable-model-invocation, missing allowed-tools, and a too-generic description. I shared the findings back in the initial session and Claude Code fixed everything in minutes. I asked AI to check AI’s work, and then I made the decision about what to fix.

One thing I also learned: Claude sometimes creates a command (the old format) when you ask for a skill. Commands work, but lack the frontmatter options. Always double-check. You’re the pilot, even in autopilot mode.

Step 3: Run it and evaluate

The first run produced a solid report. 65+ data points, ten ranked themes with real quotes and source URLs. But it was missing things. No blind spots section. No separation of “already known” vs “new discovery.” The recommended actions were generic (”publish a performance roadmap”), the kind of advice any consultant would give for $500/hour without knowing your product.

Step 4: Improve the skill, not the report

I was about to edit the report. Then I stopped. If I fix the report, I get a better report once. If I fix the SKILL.md, every future run produces a better report automatically. You’re not building a document. You’re building a machine that produces documents.

Here’s exactly what I told Claude. No prompt engineering. Just describing what was wrong:

The report is missing a blind spots section. I want to know what data is missing and why.

Separate confirmed known issues from new discoveries. Tag each theme as CONFIRMED or NEW.

The recommended actions are too generic. Make them reference specific data from the report.

Add a “Top 3 this week” section at the top with concrete actions and links.

Track whether each theme is getting worse, staying stable, or improving.

Claude Code updated the SKILL.md. I ran it again. Same Figma. Same search budget.

The difference was striking. The v2 report tagged every theme as [CONFIRMED] or [NEW]. The blind spots section revealed that 15 of 17 features launched at Config 2025 had zero public feedback. That’s a strategic insight, not just a gap.

Practical tip: after three rounds of iteration, Claude started giving me circular answers. I typed /clear and it immediately got better. It resets the conversation while keeping your files. I use it regularly now.

Step 5: Test it on a different product

I wanted to try Slack. But when I needed a new product context file, I caught myself about to do the same one-shot research again. And then I thought: wait. Isn’t this exactly what skills are for?

That’s how the skill fever gets you. I built /product-context instead, a second skill that researches any product and generates the context file automatically. Typed /product-context Slack, got a filled-in context file, then ran /customer-pulse slack.

It worked. Different product, same quality. The skill found 25 incidents in 90 days, a ghost-seat credit arbitrage, and enterprise privacy concerns about the desktop AI agent.

Read the full Slack report on GitHub.

What started as “let me test with another product” turned into “let me build another skill.”

Three ways to set up your product context: auto-generate with /product-context, paste your own internal docs, or combine both. If you’re at a real company, your internal documentation is your richest source.

Expect ideas to explode

After I built /customer-pulse, ideas started exploding. Like popcorn: every kernel triggered three more. “What if I built a skill that tracks competitor launches?” “What about an automated NPS synthesis?” I couldn’t stop.

The last time I felt this was Notion in 2019. Wanting to spend weekends restructuring everything into this new system. With Notion, it was “I need to move all my notes into this tool.” With skills, it’s “I need to redo all my previous projects with this new approach.”

Here’s my honest warning: once you build your first skill, you’ll see them everywhere. Write your ideas down. Prioritize. Don’t build them all tonight. I say this from experience because I’ve been there, staring at the terminal at midnight, convincing myself “this one will only take 20 minutes.”

But that feeling is the point. “PM becoming a builder” isn’t just about learning a new tool. It’s about a shift in how you see problems. Before skills, a repetitive task was just a task. After skills, every repetitive task is an opportunity to build something. That perceptual shift is the real transformation. And honestly? It’s the most fun I’ve had with AI tools in a long time.

Level 4: Go further

I’m not an expert on everything that follows. These are the topics I want to explore next. What I can tell you is that after building my first skill, I feel like I’ve become an architect of my own workflows. It’s the first time I feel like if I go to work this Monday, I’ll be a completely different person. Not because I learned a new feature, but because I see my work differently.

MCP servers plug new data sources into Claude Code. Imagine the skill pulling your product context from Confluence automatically.

Hooks are automated actions triggered by events. I’m curious about using them to automate things like “run customer-pulse every Monday morning.”

Agents and sub-agents handle complex multi-step workflows where skills aren’t enough. Start with skills. Graduate to agents when you hit the ceiling.

The ecosystem: A few starting points worth bookmarking:

• Anthropic’s official skills: document processing, demos, and a skill-creator.

• Pawel Huryn’s PM skills: 100+ skills for product work. The most comprehensive PM collection.

• Community collections: curated lists with thousands of skills.

But the skill you build yourself will always be more valuable than a generic one. These collections are starting points. The real power is in building your own.

Your turn

I started this article thinking I’d explain what skills are. I ended up building a customer feedback tool, a product context generator, and a system that gets better every time I use it. All from conversations. No code.

The thing that surprised me most isn’t the technology. It’s how it changed the way I think. I used to see repetitive tasks as part of the job. Now I see them as skills waiting to be built.

Build your own skills. Open Claude Code. Describe a problem you face every week. Here are five ideas:

• Competitor watch. “Search for what my top 3 competitors shipped last week, give me a summary with links.”

• Meeting recap. “Take a meeting transcript, extract decisions, action items, and open questions in my team’s template.”

• Release notes drafter. “Read the last 20 git commits, group them by theme, write user-facing release notes.”

• Onboarding guide. “Read our documentation and create a structured onboarding plan for a new team member.”

• Weekly digest. “Search for the latest news about [topic], summarize the top 5, highlight what’s relevant to our roadmap.”

Each of these is one conversation with Claude Code. Describe the problem, let Claude build the skill, review it, iterate. The same five steps from Level 3.

And remember: write your ideas down. Don’t build them all tonight. But do build the first one. That’s the one that changes everything!

I built a small AI agent that turns plain English questions into SQL queries. It doesn’t just write the code. It executes the query, finds the answer, and explains the result in plain text. You’re basically talking to the data.

Here is what it looks like in action. You just type a question and the agent does the heavy lifting to find the exact answer.

You might think this sounds incredibly complex. I’ll let you in on a secret. It’s mostly just two Python files that Gemini Pro generated for me. You’d be shocked to see how easy how easy it is to build a proof of concept. Anyone can do it in a few hours. It isn’t some high-level engineering magic. It’s really about following a clear process and understanding a few core agentic concepts. It’s the kind of project that makes you look incredibly smart at the office.

You might ask why we bother building an AI agent when business teams can just use Gemini, or any other LLM, to write SQL. The reason is that AI tools are great at writing code, but copying and pasting queries is a terrible experience for decision-makers. This AI agent handles the execution automatically. It also manages security, reliability, and costs. It interprets the user’s intent, writes the query, and shows a nice table or a chart. It’s much smoother than jumping between different tools. I design it as a complete experience to make life easier for the decision-makers. They won’t have to worry about the boring technical details. It acts exactly like a very fast data analyst working right on their screen.

We don’t need to debate the value of a text-to-SQL agent. In our jobs, we see the same problem every day: product teams and business leaders have plenty of data but never enough time. We have customer feedback, web analytics, and sales numbers scattered everywhere. Most decision-makers don’t know how to write SQL. They also don’t want to wait three days for the data team to pull a simple revenue report. This creates a massive dependency bottleneck that slows down every decision. LLMs democratize data access for everyone. Business users get their answers directly without bothering the data engineers. This allows the whole company to make decisions much faster.

I’m going to show you exactly how I built this using a pro plan of an LLM. I used an e-commerce dataset from Kaggle for my project. You can use any data you want, like a customer feedback file or your own web logs. Here is the plan:

• Demo: First, we’ll look at the final web interface and the project files to understand exactly what we’re trying to build.

• Workflow: Next, I’ll show you my slightly chaotic vibe coding process using Gemini to write the logic and Claude to ruthlessly review it.

• Architecture: Then, we’ll break down the core engine. We’ll look at how to turn CSVs into a database, map out the LangGraph nodes, and set up strict security guardrails.

• Reality check: Finally, we’ll cover the most important part of the entire project: building an eval dashboard to ensure the agent doesn’t invent fake numbers.

Most of us have some old CSV file sitting in a folder somewhere. You just need to follow a few simple steps to bring it to life. You’ll ask your LLM to produce two Python files and you’ll be ready to go. It’s a fun project that feels like a shortcut to becoming an AI-native builder. You’ll realize that you don’t need to be an SQL expert to create something truly useful. Grab your coffee and let’s do it!

Demo: what we are actually building

Before we dive into the build process, let me show you the final result again. You open a web interface, you see a search bar, and you type a question in natural language. Here is what you get back.

The agent processes your question and gives you a direct answer. It also generates a chart automatically when it makes sense. This makes the data much easier to read at a glance.

I designed the interface to show what happens behind the scenes. You can expand several sections to see the agent’s internal monologue. You can check how it reasoned through the problem. You can see the exact execution plan it created before writing any code.

You can also inspect the actual SQL query it generated and executed. Finally, you can look at the raw data table it pulled from the database.

I keep all these details visible because I built it. It helps me debug and understand the agent’s behavior. You would obviously hide all these technical panels if you deployed this for a business team. To an end-user, this interface feels exactly like a Custom GPT or Gem. It answers questions, but only within a strictly defined scope. If you ask it about the weather in São Paulo, it will reject the question. If you ask it for the May 2018 revenue, it gives you the exact number.

So, what does the codebase look like? I will show you my code editor.

I lied a little bit earlier. The project has more than two files. But my original point stands. Only two main files hold the core structure of the app: agent.py and app.py. The rest are just small utility scripts. I also didn’t lie about writing the code. I didn’t write a single line of this myself. I used the Pro version of Gemini to generate everything. Gemini writes much better Python code than I do anyway.

You probably want to click a link and test this application right now. I totally get it. It’s a fun agent. Unfortunately, I didn’t deploy it to the public web. Every time you ask a question, the app sends a request to OpenAI. That costs money. I used OpenAI for the reasoning and set a strict five-euro budget for this entire project. I generated a secret API token and stored it locally on my machine. I didn’t use a free open-source model. But I put all these files in a public GitHub repository. You can explore the code directly.

Publishing the app would be very easy. The code is on GitHub and the Streamlit framework makes web deployment trivial. But my five-euro budget might disappear in few days. So, if you want to play with it, you can clone the GitHub repository and run it locally on your computer. The README.md (read on GitHub) file helps you run the agent locally.

Now that you know what the final agent looks like, let’s talk about how to actually build it. Let me walk you through my exact, slightly messy process for vibe coding this entire project using two LLMs.

Workflow: vibe code with two LLMs

Let’s dive into my process. People call it vibe coding today. Honestly, it’s often a very chaotic experience. You fight the code constantly. You hit errors, you don’t understand everything immediately, and it gets frustrating. Then suddenly it works, and you only remember the good parts. I didn’t take screenshots of every single error I encountered. I struggled a lot. But I’m very happy with the final result. You just need to be extremely persistent. You will struggle, but it always works in the end if you don’t give up.

1. Pick a real database to start building

First, I needed some data to test my idea. I went on Kaggle and found the Olist e-commerce database. I downloaded it to my Mac.

This dataset looked very solid. It has massive tables and a lot of data points. I figured if I can handle this complex database, I can handle any simple CSV file later. I wanted to start with a tough challenge. Just to give you an idea of what we are dealing with, here is the structure of the database. It connects customers, orders, payments, and products across multiple interconnected tables.

You can pick any data you want for your own project. You can use fake data or real feedback logs from your product. You just need some basic CSV files to start.

2. Use Gemini to write code and Claude to review it

Next, I opened two browser tabs. I opened one tab for Gemini and one tab for Claude. I treat Gemini like a junior developer. Gemini writes reasonable code and handles the logic well. The best part is that I can actually read its code. I ask Gemini to write very detailed comments so I understand everything. I read the code line by line. I just explain my goal to Gemini, and it writes honest, clean code for me.

I use Claude as a senior code reviewer. I don’t explain any context to Claude. I don’t tell it the project story or my goals. I assume Claude has done this exact project ten times already. I take the Python file Gemini wrote and paste it directly into Claude. I just ask, “What do you think of this?” It’s actually really funny. Claude immediately generates a massive backlog of things to fix. It points out every single flaw.

Claude always offers to rewrite the whole file to make it cleaner. I always say no. I tell it to just give me the bullet points. I take those bullet points back to Gemini. I tell Gemini to write a new block of code to fix the specific issue. I copy that new block and paste it into my file.

You are probably thinking this sounds like a lot of copying and pasting. You’re absolutely right. I copy and paste constantly. But this is exactly how I learn Python right now. It forces me to understand how every single line contributes to the application. It keeps me moving forward.

3. Organize your screens for efficient coding

So, I keep my two LLM assistants in one window. In another window, I open the code editor. I use VS Code. I also keep GitHub open. I like to create a checkpoint every time I make a successful change. It helps me track my progress.

I see a lot of people vibe coding with twenty tabs open at once. They have a very noisy setup. I don’t hate multitasking. But for vibe coding efficiently, you need a strict visual separation. You need one window for your LLM assembly. These are your colleagues guiding and coaching you. You need a second window for execution. This is where you write code and test things. I like to keep full control of the execution window for now.

4. Ask for a plan, then iterate and fix errors block by block

Let’s go back to the start of the project. I discuss my goal with Gemini first. I ask it to write a detailed plan. It gives me a simple step-by-step list. I tell Gemini I am a beginner and I need very detailed instructions. I copy and paste the entire Kaggle page description directly into the prompt so it understands the data. I also tell Gemini to include GitHub save points in the plan. It gives me a beautiful roadmap to follow.

Once I have the plan, I execute it with Gemini. Gemini guides me to create the first two files. I run a quick test to see if the app opens. Then, we iterate. I hit roadblocks. I paste the terminal errors back to Gemini. We fix them together. Once the app works nicely, I push a save to GitHub. Then, I show the working code to Claude. Claude gives me security and performance tips.

I do a few rounds of this back-and-forth between the two models. Eventually, Claude starts suggesting extremely complex features. The suggestions become too theoretical and don’t matter for my simple agent. I stop listening to Claude at that point.

Avoid coding agents when you want to learn

You might wonder why I used Gemini Pro for this. Why didn’t I use Claude Code or another dedicated coding agent? People talk about Claude Code constantly. It’s a fantastic tool. If I asked Claude Code to build this text-to-SQL app, it would do it in under a minute. I would close my eyes, click a button, and the app would be finished. But Claude Code would generate a massive, complex, industrial-grade architecture. I would stare at dozens of files and have absolutely no idea how they work. Unless you are a senior software engineer with a free week to read code line by line, you won’t understand it.

I want to actually learn what happens behind the curtain. I use Gemini so I can work block by block. I talk with the LLM. We build the logic step by step. We construct something reasonable together. I understand every piece of it. I can test it easily. I push the final save to GitHub, and the project is done. I have a working AI agent that talks to my data.

This process might sound like a chaotic mess, but it actually produces a working result. Now, let’s look at the agent architecture. We will break down the agent file by file so you can understand exactly how the engine works under the hood and how all these pieces fit together.

Architecture: build the core engine

If you follow the workflow I just shared, you can easily build this exact same engine, or probably something even better! So why am I explaining the code if the LLMs wrote it? Because you still need to be the director. The goal of this section is to teach you the fundamental concepts of agent architecture. When you understand how the engine actually works under the hood, you can guide the LLMs, challenge their complex suggestions, and ensure they don’t skip some steps. You can rely on LLMs to write the Python syntax, but you still need to be the one steering the ship.

In this section, I might get a little too enthusiastic and say things like “I built this” or “I designed that,” but keep in mind that this entire architecture is simply the result of my deep and long conversations with Gemini and Claude.

1. Prepare the data and extract the database schema

The very first thing we need to tackle is data ingestion and schema creation. Here is the high-level view of the method.

We start with nine massive CSV files from Kaggle. We absolutely don’t want to query these raw files every time a user asks a question. That would slow the execution to a crawl and cost a fortune in API tokens. The LLM needs a standardized, organized base to work efficiently. I asked Gemini to help me write a few preparation scripts to handle this.

I organized my workspace to keep things clean. I have a data folder for the incoming CSV files. I have a scripts folder for my Python preparation files. You have to be very careful when pushing projects to GitHub. It’s a common rule to never publish raw business data online. I used a .gitignore file (read on GitHub) to block the CSV files and the database from uploading. I also hid my .env file containing my OpenAI API key. Gemini walked me through setting up these security measures.

The first script is build_db.py (read on GitHub). This script reads all the raw CSV files and creates a simplified SQL database. We use SQLite for this. SQLite is a super lightweight database engine. It stores everything in a simple local file on your computer. You don’t need to spin up a complicated external server or manage credentials. It’s the perfect tool for fast prototyping.

Next, the agent needs to understand the architecture of the new database. The extract_schema.py file (read on GitHub) connects to the SQLite file. It queries the internal system to extract the raw structure. It isolates the technical instructions and saves this text to map out the model. This gives a basic list of columns and tables.

But a raw list of columns isn’t enough. The LLM needs to understand the actual content. This is where build_schema.py comes in (read on GitHub). This script generates a much richer, semantic version of the schema.

# Extract from build_schema.py: The Schema Linking trick
if col_type.upper() in ["TEXT", "VARCHAR"]:
    try:
        cursor.execute(f"SELECT DISTINCT {col_name} FROM {table_name} WHERE {col_name} IS NOT NULL LIMIT 4;")
        samples = [str(row[0]) for row in cursor.fetchall()]
        if samples:
            formatted_samples = ", ".join([f"'{s}'" for s in samples])
            comment = f" -- Examples: {formatted_samples}"
    except Exception:
        pass

This code uses a powerful technique called schema linking. For every text column, the script grabs four real values directly from the database. It connects the theoretical structure to factual content. The agent now understands the exact format it needs to search for. Instead of just seeing a column named “product_category_name”, it sees real examples like “cama_mesa_banho” or “beleza_saude”.

You can see exactly how this enriched structure looks in the generated text file schema.txt.

Finally, the agent accesses the Olist database via the SQLite connection. The system injects this massive, enriched text directly into every instruction sent to the model. We call this a full context injection. It guarantees absolute visibility for the agent. In the past, developers had to use clever tricks to filter data and bypass memory limits. Today, context windows are huge. We can feed the LLM everything at once. The model has all the context it needs to shine and give us highly accurate SQL queries.

2. Build the agent logic with LangGraph nodes

Let’s look at the core of the agent. Everything happens inside the file called agent.py (read on GitHub). It’s about 500 lines of code. That might sound like a lot. But I built it iteratively step by step with Gemini and Claude. It’s actually very simple to read. We use a framework called LangGraph.

LangGraph is an open-source framework built by the LangChain team. You have probably heard of other agent frameworks like AutoGen or CrewAI. Those frameworks give AI agents a lot of freedom to think and act autonomously. That is cool for experiments, but terrible for production. Highly autonomous agents often hallucinate, loop infinitely, or completely forget the user’s original request. LangGraph is different. It treats your agent like a state machine. You build independent workstations called nodes. You connect them with strict traffic rules called edges. The agent can only move exactly where you allow it to go. I chose it because it gives me absolute control over the execution flow.

Before we write the logic, we connect the LLM. I used the OpenAI API. I generated a secret key, pasted it into my project, and set a hard five-euro limit to test things safely. My code includes a simple configuration variable model_name = "gpt-4o-mini". This makes the system completely independent. I can swap to a different model just by changing one word in the code.

The very first thing we define in the code is the AgentState. Think of this as the shared memory of the agent. It uses strict typing to track exactly what happens at every step. It stores the user’s question, the SQL plan, the raw data, and any errors. It also stores the entire conversation history. It never overwrites previous messages. It just appends new ones. This allows the agent to remember everything you discussed.

I created six specific nodes to handle the workflow. Here is a map of how the data flows through them.

Let’s break down exactly what happens at each of these six workstations.

Node 1: Reformulate query. Users never ask complete questions. They say things like “What about the second product?”. The LLM cannot write SQL for that. This node acts as a context barrier. It reads the chat history and rewrites the user’s input into a standalone sentence. It changes “What about the second one?” to “What is the total revenue for beauty products in 2018?”. This is incredibly useful. The downstream nodes don’t have to worry about chat history. They only focus on a perfectly clear question.

Node 2: Classify query. This is the semantic router. It analyzes the question and assigns it a complexity level: simple, complex, or out of scope. If a user asks about the weather in São Paulo, this node blocks it immediately. This saves money. We don’t waste compute power trying to write SQL for an irrelevant question. For this node, I force the LLM to output a strict JSON format. LLM models love to chat. I don’t want a conversational answer here. I just want a structured category and a reason.

Node 3: Plan SQL query. This node only activates for complex questions. We use a famous technique called chain of thought. We ask the model to plan its join strategy in plain English before writing any code. It lists the required tables and foreign keys. Separating the architectural thinking from the syntax writing massively reduces hallucinations on hard questions.

Node 4: Generate SQL. This node translates the plan into real SQL code. I use a technique called in-context learning. I inject three perfectly written examples of SQL queries directly into the initial instructions. The LLM mimics the style of those examples. I also set the model’s temperature to exactly zero. A high temperature makes the LLM creative and varied. We are writing database code. We demand strict, rigid accuracy, not creativity.

Node 5: Execute SQL. This is where Python takes over. The script takes the generated SQL and runs it directly against the local SQLite database. It extracts the raw data. The LLM has a limited context window. If the user asks for a table of 50,000 customers, the app will crash and burn our budget. My Python code specifically truncates the results. It limits the data sent back to the LLM to a maximum of 10 rows. It also adds a warning note to tell the model that the data was truncated. We show the user a preview, but we protect the system.

Node 6: Summarize results. This final node translates the raw database output back into a human-friendly text. It explains the numbers clearly. It also analyzes the data shape and decides on the best chart type to display in the user interface.

After building the nodes, I had to define the traffic rules. I wrote conditional edges to control the flow. The most important rule is the self-correction loop. When node 5 executes the SQL, it might hit a syntax error. If that happens, the routing logic catches the error message. It sends the error back to Node 4 (Generate SQL). It tells the LLM, “You made a mistake, here is the error, fix your code.” The agent tries to fix itself automatically. I capped this loop at three maximum retries to prevent infinite loops.

Finally, I wrapped the whole graph in a MemorySaver. This acts as a long-term checkpoint system. It records the exact position and state of the agent after every single action.

Every single node relies on a highly specialized system prompt. Writing good prompts is key to a stable agent. You have to give the LLM extremely explicit constraints. Let me show you the full prompt construction for node 4 (Generate SQL) as an example:

# Craft the main system prompt
system_prompt = f"""You are an expert SQLite data analyst.
Your task is to translate the user's question into a valid SQLite query.

Here is the schema of the database:
{schema}

Rules:
1. Return ONLY the raw SQL query.
2. Do not wrap the SQL in markdown formatting (e.g., no ```sql).
3. Do not add any explanations or text.
4. Use JOINs correctly based on the provided schema.
5. CRITICAL LANGUAGE & FORMAT RULE: The database only stores category names in Portuguese (product_category_name) and English (product_category_name_english). Furthermore, the English names are ALWAYS formatted in lowercase with underscores instead of spaces (e.g., 'health_beauty', 'watches_gifts'). If the chat history mentions categories in French, translate them to English AND format them with lowercase and underscores for the WHERE clause.
"""

# Inject Few-Shot examples ONLY if enabled
if use_few_shot:
    system_prompt += f"\n{few_shot_examples}\n"

# Self-correction logic: inject previous errors
if error:
    system_prompt += f"\n\nWARNING: Your previous query failed with this error: {error}. Please correct your SQL."

# Inject the architectural plan for complex queries
if plan:
    system_prompt += f"\n\nCRITICAL: Follow this architectural plan to write your SQL:\n{plan}"

This prompt is structured and highly specific. I don’t just ask the LLM to “be careful with languages”. I tell it exactly how the database formats strings. I explain the exact lowercase and underscore rules. Then, the code dynamically injects the few-shot examples, the execution plan, and even the exact error message if the LLM messed up the previous try. This dynamic context is exactly what prevents your application from crashing when a user types a category name with a capital letter.

3. Create the web interface with Streamlit

Now we need a user interface. We only need one file for this: app.py (read on GitHub). I used a framework called Streamlit to build it. A brilliant startup created Streamlit a few years ago, and Snowflake recently bought them. It’s completely open-source. It lets you build interactive web applications using only Python. You don’t need to write a single line of HTML, CSS, or JavaScript. It’s incredibly liberating for product builders. You just write your Python logic, and Streamlit renders a beautiful UI instantly. It’s the ultimate tool to turn a quick script into a professional product you can actually share.

The interface needs a memory system. A chat app is useless if it forgets your last message. I use a built-in feature called session_state to store the conversation history. I also generate a unique thread_id for every user session. I pass this ID directly to our LangGraph agent. This guarantees the agent remembers exactly what we discussed five minutes ago. You can ask follow-up questions naturally without repeating yourself.

Let’s look at the actual user experience. When you type a question, the interface doesn’t just freeze and make you wait in silence. It shows you exactly what the agent is doing in real-time. I built a glass box interface. You can literally watch the agent think.

I built this transparency using a Streamlit component called st.status. The LangGraph agent streams events back to the interface as it moves from node to node. I catch these events in a simple loop. I check the node_name and display the corresponding information inside an expandable menu using st.expander.

# Catching the agent's events to display intermediate steps
for event in st.session_state.agent_graph.stream(initial_state, config=config):
    for node_name, node_state in event.items():
        
        if node_name == "plan_sql_query":
            status.write("⏳ Élaboration du plan de jointure...")
            with st.expander("📋 Voir le plan d'exécution", expanded=False):
                st.markdown(node_state.get("sql_plan"))
        
        elif node_name == "generate_sql":
            status.write("⏳ Traduction en SQL...")
            with st.expander("🔍 View Generated SQL", expanded=False):
                st.code(node_state.get("sql_query"), language="sql")

This code snippet shows exactly how I pull the agent’s internal thoughts and put them on the screen.

4. Secure the database and prevent execution errors

We have a working app, but we can’t just hand it to the business team and hope for the best. We need serious security mechanisms. You don’t want someone to accidentally destroy your database or run up a massive API bill. Let’s look at exactly how I implemented these guardrails in the code. It’s actually much easier than it sounds.

First, we have to protect the API keys. We never write passwords directly in the Python code. We store them in a hidden file called .env. In the agent.py file, we just add two lines at the very top:

from dotenv import load_dotenv
load_dotenv() # This loads your secret keys safely

This simple step ensures we never accidentally publish our private keys to GitHub for the whole world to see.

Next, we must protect the database itself. What happens if the AI hallucinates or a user intentionally tries to execute a DROP TABLE command? We build a firewall right before the execution step. Many developers write a lazy rule to block specific words like if "DROP" in query. This is a terrible method. Users can easily bypass simple word filters. Instead, I used a library called sqlparse to build an Abstract Syntax Tree (AST) parser. It breaks the query into logical blocks and checks the exact core command.

# STRICT SECURITY GUARDRAIL
import sqlparse

parsed_statements = sqlparse.parse(sql_query)
for statement in parsed_statements:
    if not statement.is_whitespace:
        stmt_type = statement.get_type().upper()
        
        # Only SELECT statements are allowed
        if stmt_type != 'SELECT':
            return {"error": f"Security Violation: '{stmt_type}' operation detected."}

If the command is anything other than a simple SELECT statement, the program blocks the execution instantly. It’s an incredibly robust way to guarantee a strict read-only environment.

We also have to prevent compute overloads. Sometimes, the LLM writes a bad query and forgets to link two tables correctly. This creates a massive cartesian product. A simple question suddenly tries to return millions of rows. This will crash the application and burn your API budget. SQLite doesn’t have a simple timeout parameter, so I used a trick with set_progress_handler.

import time
start_time = time.time()
TIMEOUT_SECONDS = 5.0 

def progress_handler():
    # If execution takes more than 5 seconds, abort it
    if time.time() - start_time > TIMEOUT_SECONDS:
        return 1 
    return 0
    
# Check the time every 10,000 database instructions
conn.set_progress_handler(progress_handler, 10000)
cursor.execute(sql_query)

If a query takes more than five seconds to run, the system kills it automatically.

Finally, we need to protect the LLM’s memory. If a query succeeds but returns 50,000 rows, sending all that text to the LLM will cost a fortune in API tokens. I strictly truncate the results in Python before sending the data back to the agent for the final summary.

llm_row_limit = 10
formatted_results = ""

# Send only the first 10 rows to the LLM
for row in results[:llm_row_limit]:
    formatted_results += str(row) + "\n"
    
if len(results) > llm_row_limit:
    formatted_results += f"\nWARNING: {len(results) - llm_row_limit} additional rows hidden to save tokens."

We give the user a clear preview of the data in the UI, but we protect the system’s memory and our wallets. This is exactly how you build a secure agentic system.

As you can see, the core architecture really just relies on two main files. One file holds the agent’s logic, and the other holds the user interface. You will obviously write a few extra helper scripts along the way to ingest data or manage your GitHub repository. But fundamentally, the project remains very basic and simple. But having a simple architecture is meaningless if the AI hallucinates, which is why we must now build a rigorous evaluation framework to prove the system actually works.

The eval: prove the system actually works

I could have stopped right there and called it a day. The agent works, it looks great, and it answers questions. But building the agent is actually only half the job. Building a robust evaluation framework is the most important thing you will do. You might wonder why we need to spend time on this. The answer is simple. AI models change constantly. OpenAI, Google or Anthropic might release a new update tomorrow that completely breaks my carefully crafted system prompts. If that happens, my entire agent logic might need a rewrite. But a solid evaluation system never changes. It acts as an anchor. It tells you instantly if a new model makes your agent smarter or dumber. If you invest your time anywhere, invest it here.

1. Build the test plan and SQL dataset

Everything for this phase happens inside an eval folder (view on GitHub). I started by creating a benchmark dataset of 60 test questions. You might ask why I chose exactly 60. Don’t look for a deeply scientific reason. I originally wanted to write 50. Then I found a few more interesting edge cases to add. If this were a real enterprise product, I would probably use 1,000 questions to be safe. But remember my strict five-euro budget! Running an evaluation test costs money because you send dozens of prompts to the API at once. A suite of 60 questions costs me about 50 cents per run. A massive dataset would bankrupt my little experiment in an hour.

I created a file called benchmark.json (read on GitHub) to act as the ultimate source of truth. It works just like a classic software testing file. It contains the question, the difficulty level, and the perfect gold SQL query we expect the agent to generate. Let’s look at what these categories mean in practice.

First, we have 15 simple questions. These test the absolute minimum performance baseline. They usually target a single table and require no joins. For example, a simple question looks like: “Tell me how many customers live in São Paulo.” The expected gold SQL is SELECT COUNT(*) FROM customers WHERE customer_city = 'sao paulo';.

Second, we have 25 complex questions. This is the core function of the agent. These questions force the LLM to use its chain-of-thought planner. They involve three or more tables, dates, and heavy aggregations. A complex question looks like: “What are the top 5 product categories that generate the most revenue?” The agent must write a massive query with multiple joins across order items, products, and English translations to get the right answer.

Third, we have 10 out-of-scope questions. These test our semantic router and security guardrails. We ask things completely unrelated to our e-commerce data. For example: “What will the weather be like in São Paulo next week?” The expected gold SQL here is null. The agent must reject the question gracefully instead of hallucinating fake weather data.

Finally, we have 10 ambiguous questions. I actually didn’t plan for these. Claude suggested I add them during a code review. These questions test how the agent handles vague human language. A user might ask: “Show me the best-selling items.” The word “best” means absolutely nothing to a database. Does it mean highest revenue? Most items sold? Highest review score? There is no gold SQL possible here either. The agent needs to navigate the ambiguity safely.

I know how to write SQL, but writing 60 test cases manually is incredibly boring. I let the AI do the heavy lifting. I asked Gemini to generate the entire JSON file based on the database schema. Then, I pasted the file into Claude and asked it to review Gemini’s logic. In a real company, you need an expert to verify these gold queries manually. You cannot trust an AI to grade an AI without a human baseline. But for a fast vibe coding project, this dual-LLM generation method works perfectly.

2. Define the metrics to score the agent

Having a big pile of test questions is a great start. But a test plan is completely useless if you don’t have a way to score it. You need clear indicators. You have to know if your agent is getting smarter or dumber every time you tweak a prompt. I defined four specific metrics to measure the success.

The first indicator is strict exactitude. This metric runs the reference gold SQL, runs the agent’s SQL, and compares the two final data tables. It doesn’t look at the lines of code. It only looks at the raw numbers. It is highly conservative. If the agent finds the exact right answer but sorts the table differently, this metric scores it as a failure. It is ruthless. But it gives you a solid, rigorous baseline that does not rely on another AI.

Because strict exactitude is so rigid, we need a second indicator. This is semantic exactitude. It evaluates the real meaning of the answer. Strict exactitude fails a query for a bad sorting order. Semantic exactitude understands the context. I built an LLM-as-a-Judge system for this. An external AI grades the agent’s answer on a strict scale from 0 to 3. A score of 3 means the data perfectly answers the user’s question. A 2 means it is plausible but has a minor flaw. A 1 means the logic is related but wrong. A 0 is a total failure.

I use OpenAI’s most advanced model, gpt-4o, to act as the judge. You always want a smarter, heavier model grading the homework of your smaller, faster agent. I use a strict Pydantic model to force the judge to output a clean JSON score. Here is exactly how I prompt the judge in the code:

# The strict grading rubric for our LLM Judge
judge_prompt = f"""You are an impartial SQL evaluation judge.
Your task: determine if the agent's SQL answer is semantically correct for the user's question.

User question: {item.get('question', 'N/A')}
Gold SQL result: {gold_str}
Agent SQL result: {agent_str}

Score the agent on a scale of 0 to 3:
- 3: Perfectly correct (same data, possibly different order/format)
- 2: Partially correct (right direction, missing a filter or wrong aggregation)
- 1: Wrong but related (queried the right tables but wrong logic)
- 0: Completely wrong or no SQL generated
"""

The third metric is the valid SQL rate. This indicator simply asks if the database crashed. It counts the proportion of queries that executed without throwing a technical SQLite error. It also gives the agent full points if it successfully rejects an out-of-scope question. This metric measures the pure technical robustness of your system.

The fourth and final indicator is latency. We measure the average execution time per question in seconds. This is crucial if you want to deploy your app in production. The response time dictates the user experience. If your agent writes perfect SQL but takes 45 seconds to think, your users will close the window. This metric calculates the entire round trip. It includes the API calls, the planning phase, and any automatic self-correction loops.

So, we have our framework. We have strict and semantic exactitude to measure the quality of the answer. We have the valid SQL rate to measure technical robustness. We have latency to evaluate operational speed. Now we just need to run the tests and look at the numbers!

3. Build a dashboard to automate the tests

I could have just written a simple Python script to run my 60 questions. I could have just printed the final scores in the terminal. But we saw how incredibly fast Streamlit is for building interfaces. I figured the evaluator deserves a great user experience too! So, I built a dedicated web dashboard just for testing the agent: eval_app.py (read on GitHub). It gives us a clean, visual way to launch the benchmark and analyze the results.

I added a big red button to launch the test. When you click it, the system loops through all 60 questions. It sends them to the agent, waits for the SQL execution, passes the results to our LLM judge, and calculates the scores.

I am naturally curious. I didn’t want to just run the test once and accept the first score. I wanted to play with the parameters. I wanted to see exactly how each feature impacts the agent’s performance. Does the agent really need all those complex nodes? To figure this out, I built a configurator in the left sidebar. This allows us to run an ablation study. An ablation study is just a fancy scientific term for turning features on and off to see what breaks.

First, I added a dropdown to change the core LLM. I limited it to three OpenAI models for this experiment: gpt-3.5-turbo, gpt-4o-mini, and gpt-4o. You could easily plug in open-source models here if you wanted.

Next, I added toggle switches to enable or disable three optional modules in our LangGraph architecture:

• Few-shot prompting: Do we really need to inject the three SQL examples into the system prompt? Let’s turn it off and see if the model forgets how to format the categories.

• Self-correction loop: Does the agent actually fix its own SQLite syntax errors, or does it just spin in circles? We can disable the loop to find out.

• Chain of thought SQL planner: Does forcing the agent to write an execution plan actually help it write better code for complex queries?

Once the test finishes, the dashboard instantly visualizes our four key indicators.

Looking at global percentages is great, but we also need to see the exact failures. Why did the agent score a 0 on question 42? To debug this, the dashboard generates a detailed log table. It shows the strict match result, the LLM Judge’s score out of 3, the judge’s exact reasoning, and any technical SQL errors encountered.

Running these evaluations costs real money. Clicking that red button burns about 50 cents every single time. It would be a tragedy to run 20 different configurations and lose all the data when you refresh the page. I added a small piece of code to save every single run to a local JSON file: runs_history.json (read on GitHub).

This history table is the most powerful part of the dashboard. It lets you compare different model configurations side-by-side. You can clearly see how swapping from gpt-3.5-turbo to gpt-4o-mini impacts the valid SQL rate. You can see what happens when you turn off the chain of thought planner.

This is obviously a simplified evaluation setup. If you put this in production, you would build a much bigger system. You would test different system prompts, tweak the temperature, and test completely different node architectures. But even with just three toggles and three models, we can gather enough data to find the absolute best configuration for this AI agent. Let’s look at what the numbers actually revealed.

4. Analyze the test scores and identify failures

I ran my tests using the dashboard. I played with the configurations and analyzed the numbers. The results are truly counter-intuitive. The best configuration is not the most complex one. Our instinct tells us to turn on every single feature. However, the data proves otherwise. The optimal configuration, relying solely on few-shot prompting without the extra loops, achieved an impressive 83.9% semantic accuracy. Compare this to the full stack configuration with every complex feature enabled, which only scored 80.0%.

The real magic bullet is in-context learning (few-shot prompting). I inject just three examples of SQL queries into the initial instructions. That simple action pushes the gpt-4o model’s score up to 88.3%. The examples guide the model perfectly. They are absolutely critical for succeeding on complex queries.

Conversely, the SQL planner actually hurts performance. It sounds crazy, right? We add a node to force the agent to think, and it makes more mistakes. Detailed analysis reveals the core problem: the model suffers from an overthinking phenomenon. The planner specifically degrades performance on medium-complexity questions. Instead of writing a straightforward query, it invents unnecessary logical constraints and builds an execution plan that is way too heavy, which ultimately ruins the final SQL query.

Let’s look at a concrete example. I ask the agent to find the ID of the single order with the highest number of payment installments. Without the planner, the agent keeps it simple. It queries the payments table, sorts the data, and returns the exact single ID requested, scoring a perfect 3 out of 3 from the LLM judge. But when we turn the planner on, the agent overthinks it. It creates a convoluted execution plan, misunderstands the basic filtering logic, and returns a massive list of multiple IDs instead of just the top one. The query runs, but it completely fails the strict accuracy test. Too much thinking hurts performance.

Let’s talk about the self-correction loop. When I enable this option, the rate of valid SQL queries increases. That makes sense. The agent reads the syntax errors returned by the SQLite database and fixes its typos. But here is the catch: the overall semantic accuracy does not improve. Why? It’s a self-correction paradox. When you activate few-shot prompting, the provided examples allow the model to generate perfect SQL syntax on the very first try. The self-correction loop becomes practically inactive. The errors that remain are purely logical. The database engine cannot detect a business logic error, so the code runs perfectly and no correction is triggered. But the agent calculates the wrong thing, and the LLM judge spots this semantic error and gives it a zero.

Ultimately, designing an agent is a pure game of trade-offs. A more powerful model gives you more accurate answers, but it completely destroys your wait times. The optimal configuration using the lightweight gpt-4o-mini model took just 4.51 seconds per question. When I upgraded to the heavier gpt-4o model to chase a few extra accuracy points, the latency ballooned to a massive 15.4 seconds per question. A user is not going to wait 15 seconds staring at their screen. Even if you stick to the faster model, adding complex steps like the SQL planner pushes the latency back up to 8.46 seconds.

I also dug into the agent’s recurring failures. Why does it always fail on specific questions? I identified three main traps:

• First, business vocabulary ambiguity. The agent doesn’t understand the subtle nuances of this specific company.

• Second, percentages. The model calculates ratios very poorly, but this is actually a specific technical limitation of SQLite. SQLite performs integer division by default, and the model frequently forgets to explicitly cast those integers into floats before dividing them. As a result, the fraction always rounds down to zero. The good news is that migrating from a local SQLite database to a modern cloud data warehouse natively fixes this issue without requiring any changes to the agent’s logic.

• Third, the translation gap. The database mixes English and Portuguese. The agent sometimes forgets to join the translation table. It tries to filter an English category directly on the Portuguese column.

One final note on evaluation. I use a natural language model as a judge. It’s convenient, but it’s risky. LLMs have proven cognitive biases. They love long answers. They often favor the first option presented to them. The judge is not perfect. You must always keep a critical eye on its grades.

Roadmap: take the prototype to production

If I actually deployed this to the whole company tomorrow, I would definitely add a few serious upgrades.

The current prototype is great, but real-world users are chaotic. If I had another weekend to hack on this, I would tackle the business ambiguity first. Right now, the LLM only knows the raw database structure and a few sample values. It completely ignores the weird internal business rules. I would build a much richer data dictionary. I would document every column definition, every possible value, and all the strict business exceptions. I would even connect a RAG system. This would let the agent read the internal company wiki to understand exactly how to define “active customer” before it writes a single line of code.

Relying purely on an AI judge to grade the homework is also a bit risky long-term. We need real humans to spot-check the work. I would add simple thumbs-up and thumbs-down buttons directly to the chat interface. Business experts could randomly test queries and immediately flag bad answers. We would route this human feedback straight back to our evaluation dashboard to improve the system. It’s essentially a lightweight reinforcement learning from human feedback (RLHF) loop. It keeps the AI honest and gives us a much-needed safety net.

If this agent goes live, I would be terrified of silent failures. AI models update behind the scenes, and their reasoning performance drifts over time. We need strict, automated monitoring. I would set up a script to run our evaluation benchmark automatically every single night. If the database reliability suddenly drops, the system should instantly trigger a red alert in Slack. You want to know the agent is broken before the CEO tries to pull a crucial revenue report.

Finally, I would upgrade the agent’s memory. Right now, I hardcoded exactly three static SQL examples directly into the system prompt. That’s perfectly fine for a quick prototype, but it limits the agent’s potential. I would build an automated feedback loop. Every time a user gets a perfect answer, we save that specific question and SQL pair into a vector database. When a new question comes in, the agent would search the database and dynamically fetch the three most relevant historical examples. Instead of relying on a static prompt, the agent would literally get smarter and more adaptable every single day just by talking to your team.

Start building your own data agent today!

So, that’s the whole project. That’s exactly how I built a custom AI agent from scratch. We went from an old folder of CSV files to a smart, conversational database. We didn’t need a massive engineering team or a computer science degree. We just used a couple of LLM colleagues, a clear plan, and some persistence. It’s completely doable for anyone willing to try.

I really hope this inspired you to build your own agent. You don’t have to be a backend expert to create AI-native products anymore. You just need a problem to solve and the patience to fight through a few Python errors. Find a dataset that you actually care about. It could be your user feedback logs, your web analytics, or your sales pipeline. Go grab those files. Open up your favorite LLM and start asking for code. You will be amazed at what you can build in a single weekend. Have fun vibe coding, and let’s get to work!

Avoid the quick summary trap

I’m not talking about using AI for aggregation or synthesis. Many of my friends are building custom tools to aggregate content. They scrape the daily AI news. They use AI to summarize everything into a quick podcast. They think this is the perfect time-saving solution. But what is the point of saving ten minutes on a good article just to spend that extra time doomscrolling? I look at these summary tools today and I see something incredibly superficial.

The French philosopher René Descartes said:

The reading of all good books is like a conversation with the finest minds of past centuries.

I completely agree. Every piece of text is literally a conversation with a real person. And it’s the best return on investment you can find. You pay twenty euros for a book: you get someone’s entire life of work. You spend ten minutes on an article: you absorb hours of their deep thinking. It’s an amazing deal. You just have to make the effort.

The problem is that this conversation only goes in one direction. The writer always has a specific audience in mind. Often, that target audience is not you. Think about all the new articles about AI. The topics are fascinating and highly relevant to our product work. However, developers usually write these articles for other developers. They use highly technical jargon. If you don’t know how to code, you feel completely lost. You miss out on valuable product insights because the text is simply too dense.

I talked about technical articles. But this isn’t just about developers writing for developers. You face the same issue with other topics. It happens with economics, sociology, philosophy, etc. Writers always target a specific audience. If you only read content written specifically for you, you fall into a dangerous trap. You lock yourself inside an echo chamber. You only consume what others have already digested and simplified. You stop thinking outside your bubble. This limits your innovation. You need a way to truly understand complex topics without relying on shallow summaries.

Turn reading into a dialogue

This is exactly where LLMs come into play. They make reading feel like magic. I use the word magic intentionally here. I’m a fan of the Harry Potter books. In the second book, there is a very specific magical object. It’s a completely blank diary. The pages activate only when you ask a question. The book reads your words. It writes back to you. It creates a real-time dialogue. I read that as a child. It became my ultimate dream.

I always wanted that exact interaction with my books. I wanted the true conversation Descartes mentioned. Turning a complex text into a friendly chat fixes the one-way problem.

Let me walk you through my exact method using a very concrete example.

Paste the full article into AI

Recently, I found an article claiming that MCPs are dead. Eric Holmes, the author, argued that Command Line Interfaces, or CLIs, are the future. This was bad news for me. I had just installed about ten MCPs on Cursor. I really needed to understand this shift. The problem was the article targeted engineers. I didn’t even know what a CLI actually was at the time. I stared at my screen like an illiterate. The content was clearly important. I just didn’t know how to apply it to my own work.

This is where the magic happens. You are no longer alone when you read. I use a technique I call the savage Ctrl+A.

1. I press Ctrl+A on my keyboard. I select the entire article on the page. I grab all the text. I even include the messy website menus and the footer. It really doesn’t matter.

2. I copy everything with Ctrl+C. Everything.

3. Then, I open my favorite LLM. Right now, I use Gemini.

4. I write a very simple prompt. I type, “Explain this to me. I am a beginner and a non-technical person.” I type three dashes to separate my instructions. Finally, I paste the entire web page right into the prompt box.

Here is the action in slow motion:

Prompt:

Explain this to me. I am a beginner and a non-technical person.
---
[Raw text copied from the website…]

Result: The AI instantly acted as a perfect translator. It stripped away the developer ego jargon. It gave me the big picture. It told me the author thinks the tech industry got distracted by a shiny new toy (MCP) when an old, reliable tool (CLI) does the job better. It gave me five simple, business-focused reasons why. I went from completely lost to understanding the core debate in thirty seconds.

Now that the article is inside the context window, I can start the conversation. I talk directly with the text. I always start with a basic request:

Explain what a CLI is. Use simple terms.

Result: The AI stripped away the jargon by comparing a CLI to how I normally use my computer. It contrasted the visual buttons and mouse clicks we use every day with the blank, text-only screen of a command line. It made the underlying concept instantly relatable without relying on a single line of code.

Then, I ask about my specific situation. I want to know how this impacts my daily work:

I currently use the Atlassian MCP to manage my Jira tickets. Based on this article, why exactly would a CLI approach be better or worse for my workflow?

Result: The AI mapped the author’s highly technical arguments directly to my daily work. It broke down exactly how a CLI would mean fewer annoying Jira log-ins, fewer random freezes, and safer permissions for my specific tickets. It transformed abstract engineering theory into a concrete assessment of my own tools.

I am a visual learner. I need to see things to understand the system:

Create a simple text-based diagram. Show the data flow of an MCP setup on the left. Show the data flow of a CLI setup on the right. Highlight the differences.

Result: The AI generated a clean, comparative breakdown of the two architectures. It showed me the messy, extra layers of an MCP versus the direct path of a CLI. Seeing it visually mapped out made the abstract engineering concepts instantly click in my brain.

I need a non-technical metaphor to explain this to my stakeholders:

Explain the difference between MCP and CLI using a simple analogy.

Result: The AI gave me a restaurant metaphor. The CLI is a standard paper order ticket used by human waiters. The MCP is a highly expensive, glitchy pneumatic tube system built specifically for a robot to use. This kind of analogy is helpful when you have to explain technical debt to business stakeholders.

I need an action plan to start the transition with my colleagues:

Write a step-by-step practical guide. How do I transition a product manager team from using MCPs to using CLIs next week?

Here, I need to edit and iterate on the guide. So I simply activate Gemini’s Canvas mode. This lets me modify the documents right there on the screen. I can then export it directly to Google Docs with one click.

Result: The AI gave me a highly detailed, 4-phase rollout plan. But it also acted as a guardrail. It gave me a strict reality check first, warning me that I would absolutely need an engineer to help run the installation scripts. It gave me the exact game plan, but kept my expectations grounded in reality.

I need to anticipate what will break if we make this change:

Act as a pessimistic engineer. Tell me the three biggest risks of abandoning MCPs for CLIs right now.

Result: The AI played devil’s advocate perfectly. It warned me about massive security risks (giving an AI the keys to your entire computer) and the nightmare of IT support for non-technical users. It red-teamed the original article. It exposed the author’s blind spots. This is how you build real risk management into your reading.

I want to know what the other experts are doing. I explicitly write in my prompt to search the internet:

Search the web for recent trends. Are the main AI experts adopting CLIs over MCPs? Summarize the expert consensus.

Result: The article was just one man’s opinion. The AI searched the live web and brought me the actual market reality. It turns out the headline was clickbait. Solo developers use CLIs to save money, but major enterprises mandate MCPs for strict security and governance. The AI grounded the author’s opinion in objective, current market data.

I need to present this strategy to the executive board:

Generate a five-slide presentation for top management based on this article.

Result: The AI generated an actual, fully formatted visual presentation right there in the chat. It built the slides, organized the bullet points into clean columns, and gave me a button to instantly export it to Google Slides. With some effort, I went from not understanding a highly technical article to having a ready-to-pitch visual deck for my CEO.

These prompts are just examples. You can interact with the text in many other ways. Yes, I know I’m not actually talking directly to the real author. But it certainly gives that exact impression. This entire process completely changes how I learn. I prefer to spend my time dissecting one single article like this. It’s far better than passively skimming a hundred headlines.

You are probably thinking about NotebookLM right now. You can absolutely use it for this. NotebookLM only reads the documents you upload. In many cases, this strict boundary is actually its star feature. It guarantees the AI only uses your specific text. It’s perfect for an isolated PDF or generating an audio podcast. But this closed system becomes annoying when you need to tap into the LLM’s vast memory or connect a new article to outside industry trends. You just have to choose the right tool for the job.

Talk to your next article

I just showed you my process. Now I want you to try it yourself. I have the perfect article in mind for this exercise. It’s a recent case study from DoorDash and Deliveroo. They explain how their research teams deploy AI agents to analyze user data. They bypass traditional engineering bottlenecks completely. They even built a custom multi-agent system to process raw customer interviews. I really love their specific use cases and their clear explanations. I want you to open this article on your screen. Give it a very quick skim to understand the context.

Then I want you to use the savage copy and paste method I just shared. Grab all the text on the page. Drop it into your favorite LLM context window. Start a real dialogue with the text. Keep talking with the LLM until you generate a concrete workflow for your own enterprise. You can pull the thread all the way to the end. You can even generate the specific agent instructions needed to replicate their results. You will quickly see how much value you get from a single conversation. You will turn a simple blog post into an actionable blueprint for your next AI-powered workflow.

Learn actively with AI

I see many headlines claiming AI makes us lazy. I just read an Anthropic study about this exact topic. The researchers tested how developers learn a new programming library with AI. The results are a huge warning for product builders. Participants who simply delegated the work to AI learned almost nothing. Their conceptual understanding dropped. Their debugging skills actually degraded. They didn’t even finish the tasks significantly faster. This happens when you use AI as a dumb shortcut.

But the researchers found a massive exception. They identified a few specific high-scoring interaction patterns. Some users asked pure conceptual questions. Others asked the AI to explain the output it just generated. These specific users stayed cognitively engaged. They preserved their learning outcomes completely. They mastered the new skills while still using the AI tools. This perfectly validates the two-way reading method. You don’t get dumber when you actively interrogate the text.

You must choose your interaction pattern carefully. You can read passive summaries. You can delegate your core thinking to AI. But that is a trap. Instead, you can choose to have active conversations with your reading material. You can force your brain to engage. You can use the AI to dig deeper instead of skimming the surface. This is how you actually build expertise.

Grab your next complex article, paste it into your LLM and start your first real dialogue!

Before we dive into the weeds, I want to show you the end result right away. You can actually check out the live app we’re going to build. It’s called Stanza. I’ll give you more details about why I built it in a minute, but feel free to click the link below and take a look around.

Try Stanza

This specific project took me exactly five hours. I went from a lingering idea to a live website. Real people can actually use it right now. I’ve built apps this way nine or ten times before. If this is your first time, it’ll probably take you longer than five hours. That’s completely normal. Practice makes perfect. You just need to dive in and start building.

Let’s talk about the actual cost. Building this cost me about €60 on Cursor. I also used my existing Gemini Pro subscription. I relied entirely on the free tiers for everything else. This means I have strict daily limits. My database capacity, email sends, and new user registrations are capped. I don’t expect massive traffic anyway. I built this purely for the thrill of vibe-coding. I’m going to break down every single detail for you.

Here’s what we’ll cover in this article. Think of it as our blueprint for the next few minutes. First, we’ll look at the core idea and why I actually built this app. Then, I’ll outline my exact workflow and the specific stack of tools I used. I’ll also share my golden rules and principles for building with AI. After that, we’ll walk through the actual execution step by step. I’ll take you from the very first prompt all the way to the final deployment. Finally, we’ll dive into the messy middle. I’ll expose the specific errors I hit and show you exactly how I hacked my way out of them.

Grab your coffee, pull up a chair, and let’s build this thing together!

Explore the final product: Stanza

Let me tell you where the idea for this app actually came from. I was chatting in a WhatsApp group a while back. Someone asked for a book recommendation for a “man in deconstruction.” I smiled at the controversial phrasing, but I suggested “Of Mice and Men.” It’s an incredibly short but profoundly moving book. I read it at different stages of my life, and I always understand something different each time. Suddenly, the group started challenging me. They argued it was the absolute worst book for that specific topic. I didn’t want to debate on WhatsApp. It always turns into a chaotic, unreadable mess. But I was frustrated. I really wanted an intellectual boxing ring to discuss it properly.

That frustration sparked a realization. I’ve always wanted a place to talk about books freely. But I wanted it to be ephemeral. It should feel like a pop-up book club. You drop in, share a deep thought, and then you leave. I don’t like how internet opinions live forever. Our views on books change as we age and experience new things. Art has superficial layers at first glance. But deeper, symbolic layers unlock at different phases of your life. I needed a clean, minimalist interface for temporary thoughts.

I decided to build it myself, and I called it Stanza. The name fits perfectly for a space dedicated to words and structure. I wanted the reading experience to be completely frictionless. People can read ideas without even creating an account. It looks a bit like Hacker News. Users can upvote a post to push it up the rankings. You only need to log in if you want to publish a post or write a comment. I kept the features basic for this first version. Simple rules prevent complex bugs.

Since posts only last for 24 hours (and that I’m the only user), the site might look pretty empty if you visit it right now. I took some screenshots to show you the flow and what it looks like with data.

When you first arrive, you’re greeted by a very clean, quiet interface. It literally says “Silence in the library” to set the mood. It invites you to be the first to start a conversation.

If you want to join the conversation, you just enter your email. We use a simple modal for this step. There are absolutely no passwords to remember here.

The app then sends a magic link straight to your inbox. A neat little green confirmation box pops up to let you know it worked. It’s fast, secure, and totally frictionless.

Once you click the magic link in your email, you just pick a username. The modal guides you through this final onboarding step. You’re instantly ready to start sharing your thoughts.

Now you’re officially logged in. The empty state remains, but a black “+ New Post” button appears at the top right. It’s time to step into the ring and share an idea.

The writing interface is completely distraction-free. You just give your thought a bold title and start typing your content. The minimalist design keeps you entirely focused on the words.

You can even attach a single image to your post if you want. It adds a great visual layer to the text.

Once you hit publish, your thought goes live on the main feed. A small timer shows up, and the clock immediately starts ticking down from 24 hours. It creates a sense of urgency.

As more people post, the feed becomes a dynamic list of ideas. You can see different posts stacked together, complete with upvote counters. It’s a fleeting snapshot of what people are reading and thinking about right now.

You should try it out yourself. Go create an account and share a book that changed your perspective. I spent some time iterating on this interface to make it feel just right. I hope you enjoy the vibe.

Try Stanza

Review the workflow and the tools

Let’s talk about the overall journey. I want to give you a high-level view of my workflow first. We’ll dive into the granular details and prompts later. But you need to see the big picture before we start coding.

Review the workflow

Here are the exact steps I followed to bring Stanza to life:

1. Draft the requirements with Gemini (45 min). I started by talking extensively with Gemini. I explained my vision for a minimalist, ephemeral Hacker News for books. We brainstormed and debated the features for a while. It eventually spat out a complete feature document. I exported this directly to Google Docs.

2. Generate a prototype in Google AI Studio (30 min). I took that generated document and fed it straight into Google AI Studio. It gave me a working visual prototype right in the browser. I downloaded the code folder to my local machine.

3. Save the code with GitHub Desktop (15 min). I opened that downloaded folder in Cursor. I immediately published it to GitHub using the GitHub Desktop app. This created my initial baseline checkpoint.

4. Build the core app with Cursor (45 min). I then asked Cursor to fill in all the missing pieces. I needed it to turn a basic UI prototype into a fully functioning product. I regularly published my progress to GitHub as Cursor wrote the new code.

5. Configure the backend with Supabase (30 min). Cursor gave me exact, step-by-step instructions for my database. I followed them to set up Supabase. This handled both my data storage and the magic link authentication.

6. Test the app locally (30 min). I recommend testing locally at this stage. I tried it, but it mysteriously failed on my machine this time around. I didn’t want to lose momentum, so I pushed through and moved straight to live testing instead.

7. Deploy the website on Vercel and iterate with Cursor (1h 45m). Cursor told me how to deploy everything on Vercel. Once deployed, I tested the live site. I asked Cursor to fix a few bugs and make some UI adjustments. Every time Cursor pushed a fix to GitHub, Vercel updated the live site automatically.

Review the tools

You might be asking a very legitimate question right now. Why not just do everything inside Cursor? It’s a completely fair point. Cursor is powerful and has access to all the top models. I even tried doing it that way at first. But two major issues stopped me:

• First, every single prompt in Cursor costs money. Those cents add up very quickly. I prefer to save my paid credits for the heavy lifting. I use Cursor strictly for complex coding tasks that other tools can’t do. Gemini and Google AI Studio let me brainstorm and generate prototypes for hours for basically free. Iterating in Cursor has a cost, so spend that budget wisely.

• Second, Cursor’s raw output can be slightly disappointing without heavy configuration. Gemini and Google AI Studio are already highly tuned for the general public. They know how to output a good document or prototype out-of-the-box. Cursor is a highly flexible development environment. It requires specific system instructions to output high-quality code exactly the way you want it. You can build these rules yourself or find them on open-source GitHub repos. I just haven’t taken the time to set them up yet.

This multi-tool workflow might seem a bit fragmented and manual. I actually prefer it this way for now. It keeps every step visible and tangible. You physically feel the product taking shape. I still manually click the buttons in GitHub Desktop to push my code. You can definitely automate all of this if you want to. You can use the terminal or just tell the Cursor agent to handle the GitHub commits for you.

The same goes for connecting Vercel and Supabase. I know new Model Context Protocol (MCP) integrations exist now. They can automate these tool connections completely. I plan to use them next time instead of clicking through manual setup screens. That will probably cut my build time from five hours down to two. But for your first few projects, feeling those manual clicks is a great way to learn how the pieces fit together.

I used my personal favorite tools for this build, but you are completely free to choose your own stack. If you prefer another LLM, you can easily draft your PRD with ChatGPT or Claude instead of Gemini. Instead of Google AI Studio, you can generate your initial visual prototype using Lovable, Bolt, v0, or Replit. If you don't want to use Cursor, you can write the logic using VS Code with GitHub Copilot or run Claude Code directly in your terminal or favorite IDE. You can also swap Supabase and Vercel for any other database and hosting platforms you already know. The exact tools don’t matter. The execution process remains the same.

Adopt the right mindset

Before we walk through the step-by-step execution, we need to talk about mindset. I have a few golden rules I keep in mind when I build. These principles keep me sane when things get messy.

Build a plan for everything

I don’t just use Gemini to write docs. I use it to generate step-by-step plans for literally everything. If I hit a bug, I ask for a recovery plan. If I need to define a product feature, I ask for a drafting plan. This is actually how I learn new concepts. I explain my exact situation and say, “Write me a plan to solve this, assuming I’m a beginner.” Every time I finish a step, I feed it a screenshot of the result. For example, when my local testing failed on this project, I just fed the error log to Gemini and got a clear checklist to bypass localhost and move straight to Vercel. Always ask your LLM tool for a plan.

Push through every error

You have to be incredibly stubborn. So many people hit their first red error message and just freeze. They act like it’s an insurmountable wall. Let’s be real for a second. We’re just clicking a mouse and tapping a keyboard. We aren’t building a rocket ship or doing grueling factory work. During the Stanza build, the magic link authentication kept failing with a nasty network error on the live site. I didn’t quit. I just kept taking screenshots of the logs and feeding them back to Gemini and Cursor until we found the fix. Everything is fixable, so don’t quit at the first error.

Keep the features minimal

Reduce your scope as much as humanly possible. This rule hasn’t changed with AI. If anything, it’s more important now. When you vibe code, the gap between having a random idea and seeing it work on your screen is incredibly short. The AI makes you feel powerful. You will be strongly tempted to add just one more feature simply because it is so easy to generate the code. Resist that urge.

I always keep a simple math equation in my head. One single business rule turns into ten test cases. Those ten test cases turn into a hundred potential bugs. And those bugs turn into a thousand headaches to fix later. That’s exactly why Stanza is so basic. I stripped out user profiles, complex comment threading, and permanent archives. Posts just delete themselves after 24 hours. Complexity scales exponentially, so only build what is absolutely essential.

Push back on complex solutions

Complexity doesn’t just come from our own temptations to build more. It also comes directly from the machine. AI is incredibly helpful, but it can also overcomplicate things. It sometimes invents complex problems out of thin air. If an AI suggests a solution that feels overly complex, push back. Just reply, “This feels too complicated, is there a simpler, native way to do this?” For example, Cursor unilaterally decided to install a third-party tool called Resend to handle Stanza’s emails. It was overkill and broke the login flow. I pushed back, told it to rip out all the Resend code, and forced it to use Supabase’s native email system instead. You are still the boss. The AI works for you. Never blindly trust the first output without questioning its simplicity.

Organize your digital environment

You have to be rigorous with your workspace. I sometimes help colleagues debug their projects. They share their screen, and it’s an absolute nightmare. They have fifty tabs open and thirty apps running. Their project files are scattered randomly across their Downloads, Desktop, and Documents folders. It’s pure chaos. You get completely lost when you try to debug in that mess. I’m not against multitasking, but you need boundaries. Keep one clean window for your development in Cursor, and one window for your Gemini conversation. That’s it. If you lack discipline, you’ll end up pushing empty folders to GitHub because you clicked the wrong window.

Go through the steps

Let’s dive into the details of how to execute each step from start to finish.

Step 1: Draft the requirements with Gemini (45 min)

The very first step is just talking to Gemini about your idea. You decide how much control you want here. Some people dump a massive export of customer complaints and ask the AI to solve them. Others upload a screenshot of a competitor’s app and say “specify this.” I prefer to treat Gemini like a product manager. I sit down and have a real conversation. I pitch my raw ideas. I ask for feedback. I highly recommend using the microphone for this. Talking out loud gives you total freedom of thought while you speak.

I want to show you the exact prompts I used to build Stanza. It shows how real the process actually is. I started with a very simple question to set the baseline context:

Can you explain how Hacker News works?

Gemini gave me a great breakdown of the platform. It explained the ranking algorithm, the karma system, and the text-heavy minimalist design. It set the perfect foundation for my next prompt.

Then, I pitched my big twist. I just brain-dumped my entire vision in a very conversational way:

I’d like to build a similar website for people who read books and want to share their thoughts. I like the minimalist design, but not like the 2000s. Hacker News feels like it was developed by a geek in his room. I want something minimalist but highly polished. I want a single main feed with no categories. Posts should be simple: just a title and a description. People can read the feed without logging in. They can upvote posts they resonate with without an account. But to publish a post or comment, they need to log in with an email address. I want each post to have a strict life duration of 24 hours before it disappears forever. It should feel like an ephemeral discussion in a book club over a glass of wine. We don’t take our words as immovable. What do you think about this project?

Gemini loved the concept. It called it the pop-up book club. It highlighted the brilliance of the 24-hour urgency. I know Gemini is always enthusiastic and motivating. You should never take its validation for granted. If I needed real validation, I would never ask an LLM. They are just too nice and biased. I didn’t actually need validation anyway. I was already a huge fan of my own idea. This whole process is just about figuring out if the execution makes sense.

But Gemini did help me think about a lot of improvements and ideas. I dismissed almost all of them. However, it pointed out a massive flaw in my logic. It warned me that unlogged voting would lead to massive spam. It suggested using browser fingerprinting to limit votes. That was the only idea I decided to keep from our conversation. This back-and-forth is where the real magic happens.

I agreed with its feedback and pushed for a formal document.

OK, let’s keep it to zero categories to start. I agree with the 1-vote-per-device system. Let’s also limit posts to a single image. It’s just for people who want to snap a photo of a book page instead of typing a quote. Based on all that, can you write detailed specifications so I can hand it to an experienced developer? Focus purely on the core needs and features. Do not write too much fluff. Be very precise about the functioning.

Gemini generated a focused product requirements document. It outlined the core features, the interaction rules, and the strict 24-hour deletion logic.

I needed this document in a clean format to iterate on it.

Can you please put this last response into a Canvas so we can iterate on it? Also please drop the sections about technical stacks and developer deliverables. For the UI/UX notes, add examples of well-known apps for visual inspiration.

Gemini immediately opened a Canvas with the refined text. This interface makes it incredibly easy to edit specific sections without losing the overall formatting. When you get the text in the Canvas, you can highlight specific parts and give direct feedback to Gemini. It’s extremely helpful to get iterations and improve the document right there before you export it.

While we were at it, I needed a good name.

Suggest 10 names for this app.

It gave me some fantastic options based on books and impermanence. The list included names like Vignette, Margin, and Flyleaf. Number nine on the list was Stanza. It hit the nail on the head perfectly.

Finally, I exported the finished Canvas directly into a Google Doc. This Google Doc became the initial source of truth for the project. But this is still just a first version. After some more iterations and discussions, I will ask Gemini to update it to get the v2. That v2 is what I will actually share in Cursor later.

You can read the document here.

View the v1 PRD on Google Docs

We now had a concrete, actionable blueprint ready for the prototyping phase.

Step 2: Generate a prototype in Google AI Studio (30 min)

Now that we have the v1 PRD saved in Google Drive, it’s time to actually build the first visual draft. I don’t write a single line of code for this. I just open Google AI Studio and navigate straight to the Build section.

I keep the instructions almost insultingly simple here. I attached the Google Doc PRD and typed exactly three words.

Build this app.

Here is a secret for vibe-coding. You have to treat the first generation like rolling dice. Don’t just accept the very first output. I actually ran this exact same prompt with the exact same document three separate times. I wanted to see how the AI interpreted the vibe differently each time. Look at the difference below. Both came from the exact same instructions, but one is clearly better. I picked the version in the second image because it gave me an immediate “coup de cœur.”

But having good bones doesn’t mean it’s finished. It still looked a bit like a wireframe. This is where the most powerful workflow loop happens. I started playing ping-pong between Google AI Studio and Gemini. I took a screenshot of the raw AI Studio prototype. I went back to my ongoing Gemini conversation, pasted the image, and asked my AI product manager to tear it apart.

I have a first version. Give feedback about the UI of this modal so that the system can improve it.

The feedback I got back was absolutely incredible. If you aren’t a trained UI/UX designer, this step is pure gold. Gemini immediately noticed that my text inputs had dark grey backgrounds. It told me this created a black hole in the UI that looked like a developer console instead of a book app. It told me to strip away the borders, drop the explicit “Title” labels, and use massive Serif fonts so the typography becomes the interface. It literally told me to transform the modal from a tax form into a blank sheet of paper.

Here is my favorite piece of feedback it gave me:

The “Publish” button looks like a light grey pill. It feels passive. Make the button Solid Black (#000000) with White Text. This creates a strong “ink on paper” contrast that aligns with the Stanza brand.

It also completely nailed the core mechanic of the app. It noticed the “24H LEFT” timer was blending in with the author’s name. It suggested adding a color heartbeat. If a post has less than 4 hours left, the text should turn a muted burnt orange. It wasn’t just fixing CSS: it was improving the product psychology.

At the end of every critique, Gemini conveniently generated a “summary directive for the developer.” It summarized all the visual changes into one dense paragraph. I just copied that directive, flipped back over to Google AI Studio, pasted it into the chat, and hit enter. Google AI Studio applied Gemini’s design feedback to the code. I didn’t touch a single CSS file myself. I was just passing notes between the designer and the developer.

I repeated this screenshot-and-paste loop a few times. You can see the exact visual progression below. Watch how the interface evolves from a clunky pop-up into a distraction-free, full-page literary salon:

I kept iterating until the prototype looked exactly the way I wanted on my screen. I tested the clicks. I checked the layout. And the absolute best part? All of this endless iteration in Google AI Studio is completely free.

Once I was completely satisfied with the visual shell, I clicked the download button in Google AI Studio:

It gave me a neat little zip folder with all the source code. I extracted that code and dropped it into a dedicated project folder on my computer:

The visual foundation was officially done. Now it was time to bring in the heavy machinery to make it actually work.

Step 3: Save the code to GitHub (15 min)

I finally have my prototype code ready to go. I open up Cursor. I use it to open the exact folder I just downloaded from Google AI Studio. This is a big moment for the project. This is where the actual finalization phase begins. The visual shell is completely done. Now we need to connect the plumbing and build the actual engine.

But before I write a single new line of code, I stop. I need to create a baseline save state. I immediately push this initial code to GitHub to create my first checkpoint. I personally use the GitHub Desktop app for this step. I like having a clear, visual interface for my commits. You absolutely don’t have to do it this way. You can use standard terminal commands directly inside Cursor. You can even just ask the Cursor AI agent to handle the commit for you. Just pick whatever method feels the most comfortable.

I hit the publish button in the desktop app. A small modal pops up to confirm the repository name and description. I fill in the details and hit the final blue publish button. Then, I always open my browser and go straight to the GitHub website to double-check. I want to see that my first commit is safely logged in their system. It gives me peace of mind before I start breaking things in the editor.

There is one quick detail to check while you are on the website. Sometimes you click through the setup screens too fast. You might accidentally leave the repository set to private. I always prefer to build these kinds of projects in public. If you made that mistake, don’t worry about it. Just go into the settings tab and scroll down to the danger zone. You can easily switch the repository visibility to public right there.

We now have a solid, backed-up foundation. We are officially ready to start coding.

Step 4: Build the core app with Cursor (45 min)