Dean's WhatsApp was the product. Then it became the ceiling.
A full teardown of the AI coach we built for his mentorship programme: the architecture, this week's numbers pulled live, and a dated log of everything that broke.
last 7 days
same period
it now does
kickoff to live
running cost
Pulled from the running system on 6 August 2026. Usage figures are the trailing seven days, not all-time.
- Every question in the programme arrived in one man's DMs
- Beginners and experienced operators got the same answers
- Call insights lived in recordings nobody went back to
- The course sat in a platform the coaching never touched
- Growth meant more of Dean's evenings
- 205 conversations answered in a week without him in the loop
- Answers change based on whether you own nought properties or twelve
- Every coaching call becomes a written summary in the member's portal
- All 138 lessons are searchable inside the coaching itself
- Growth costs about forty dollars a month
The thing his members paid for was the thing he couldn't scale
Dean Reilly runs Parea Living, a short-let accommodation business across London and the Midlands, and teaches other people to build the same thing through a mentorship programme called Work Worth Doing. The course behind it, The Automatic Airbnb System, runs to 138 lessons across seven modules covering housekeeping, mindset, foundations, operations, leads, sales and systems.
The course was never the product though. What members were really buying was access to Dean's judgement, and that arrived one WhatsApp message at a time. Someone would ask whether to take a mortgage or keep saving. Someone else would ask whether to move a portfolio into an SPV before the next tax year. Both questions landed in the same inbox, and both waited for the same person.
This is the shape of almost every expert business we get called into. The bottleneck isn't marketing or delivery. It's that the founder is the only instance of the thing being sold, and there's a hard ceiling on how many hours that instance has. Dean had already hit it. He could either cap the programme, hire coaches who wouldn't sound like him, or find a third option.
A coach who answers in four hours instead of four days is worth more than a coach with better answers.
The brief we took was narrow on purpose. Answer members in Dean's voice, using only what Dean actually teaches, and adjust the answer to where that member currently stands. Everything else the system does now grew out of that, and none of it was in the original scope.
A chatbot on top of his course would have been actively dangerous
The obvious build is a week's work. Take the course, chop it into chunks, put them in a vector database, and pass whatever comes back to a language model with an instruction to answer like Dean. Most agencies would ship that and call it an AI coach. It fails in three specific ways, and each failure taught us something about what the real system had to do.
It gives the same answer to two people who need opposite advice
Take a real question from the programme: should I use a limited company or buy in my own name? For someone with no properties and no income from lettings, the answer is that the question doesn't matter yet and there's something more useful to do this week. For someone with nine units and a full-time job, it's a serious tax conversation. A plain retrieval system gives both people the same paragraph, because both queries hit the same chunk of course content. That's how an AI coach ends up giving beginners advice that only makes sense for experienced operators.
So retrieval had to know who was asking. Every member has a profile that builds up over their conversations, tracking where they are in the programme and what they're working on, and that profile changes which chunks get ranked highest before the model ever sees them.
It answers the question instead of doing the job
People don't only ask questions. They ask for a message to send a landlord who's gone quiet. They ask to be prepped for a call in thirty minutes. They ask why the last eight approaches got no reply. Those are different jobs with different shapes of correct output, and a system that treats all of them as questions produces something that reads fine and helps nobody.
The running system now recognises eleven distinct jobs, which we didn't design up front. We built four and the rest came from watching what members actually sent.
It sounds confident about things it doesn't know
This is the failure that matters most and gets discussed least. When retrieval comes back weak, a language model doesn't stop. It fills the gap with whatever is plausible. In our own evaluation the system confidently named specific software and quoted specific percentages that appear nowhere in Dean's material. To a member, an invented figure in Dean's voice is indistinguishable from Dean's actual advice, which makes it worse than no answer.
That's why the build includes a gap detector that logs any answer where retrieval scored badly, and why weak retrieval on a factual question now gets intercepted before it reaches the member rather than after.
The hard part of an AI coach isn't making it answer. It's making it decline.
Five lanes, one machine
What started as a chat endpoint is now five independent lanes sharing one knowledge base. The chat lane answers members. The ingestion lane keeps the knowledge current without anyone uploading anything. The call lane turns coaching calls into written records. The onboarding lane builds a new member's entire environment from a signature. The operations lane watches the other four and tells Dean when something breaks.
and phone
11 job types
3,500+ topics
7,787 chunks
source quota
Dean's voice
draft check
the DM
course, podcast
audio and video
chunking
knowledge graph
of its own answers
meeting log
alias-scored
decisions
plus a DM
course access
group created
access roster
week
flag at 96h
a webhook missed
against backup
and key check
by step
and weekly digest
The three decisions that matter
A knowledge graph in front of the vector search. Members ask about things using their own words, and Dean teaches them using his. A graph of more than three and a half thousand topics, built from his own material, expands a query before it reaches the vector database, so a question about "getting landlords to say yes" also searches the language Dean uses when he teaches objection handling.
Retrieval that knows the member's stage. Reranking happens after the vector search and before generation, weighted by what the member's profile says about where they are. The same query from two members returns different source material. This is the single change that stopped the system giving beginners advanced answers.
A cheap model in front of an expensive one. Every message hits a fast, inexpensive model first to work out which of the eleven jobs it is and what to actually search for. Only then does the expensive model generate. That keeps the running cost at tens of dollars a month rather than hundreds, and it makes the routing decision inspectable, which matters when something goes wrong.
What we've left out here is deliberate: no prompts, no similarity thresholds, no reranking weights. The shape of the system is worth more to you than our settings, which change every few weeks anyway.
What actually happens in those forty seconds
Here's a single exchange with the pipeline decisions shown next to it. The member and the details are invented, because publishing real conversations from a live coaching programme isn't something we'd do. The routing, the retrieval behaviour and the timings are real.
Send one message today and make it about them, not about you. Something like: "Hi Mark, been thinking about the second bedroom since we spoke. If the guarantee is the bit you want to be sure about I can put the numbers in writing so you've got them in front of you." Then stop.
You've already got the verbal. Chasing it turns your position into a weaker one. What you want is to give him a reason to reply that costs him nothing.
Two lines in that trace are worth your attention. rerank 24 → 4, stage-weighted is where a beginner and an experienced operator stop getting the same answer. patterns: asked twice before is memory across conversations, which exists because one member asked us the same question more than thirty times and got a patient fresh answer every time.
Pulled from the running system, not from a slide
Everything below came out of the live system on the morning of 6 August 2026. We're showing the trailing seven days rather than all-time totals, and that decision is worth explaining, because it's the kind of thing that usually gets quietly hidden in a case study.
The all-time counter reads higher. We checked it against the weekly series and found the conversation log only holds records from late July, which is consistent with a storage eviction we've already documented and fixed. So the bigger number exists, and we can't stand behind it. The seven-day figures we can.
Adoption, week over week
trailing 7 days vs the 7 before
Conversations up 5.5x, active members up 3.2x, one week to the next. The jump follows a roster repair and the switch of the course reference described in log entry 01. 13 of 29 people with access used it this week.
Zero knowledge gaps in the logged window
The system scores its own retrieval on every message and logs a gap whenever it answers from weak material. Across the 205 conversations logged this week, it recorded none. That's a claim worth being suspicious about, since a silent detector and a working one look identical from outside, so here's the evidence it's alive: the same detector has flagged 87 queries over the life of the system, routing twenty to Dean as missing content and closing another twenty on its own once the material existed.
The honest reading is that the knowledge base finally covers what members ask, which it didn't in June.
It stopped being a question-answering bot
This is the chart we find most useful, and the one nobody publishes. In May the system did three things and 99 percent of that was answering questions. Here's the same measurement now. Ten of the eleven job types it recognises got used this week, and they add up to the same 205 conversations in the chart above.
Nearly a third of all traffic is now outreach and closing help, which is work Dean used to do on calls. Another fifteen messages were classified as pre-call coaching, someone getting ready for a conversation they're about to have. That's the moment coaching is worth the most and the moment a human coach is least likely to be free.
Our own quality scores, including the bad one
We grade the system on five dimensions against a standing set of cases, with a separate model doing the grading against a written rubric. Every time we find a new way for it to fail, that failure becomes a permanent case. Publishing this is unusual and we think it should be normal. Two things make it worth your time: the low score, and that we re-ran the whole thing against the live system this morning, so you're seeing movement rather than a snapshot.
Quality evaluation, before and after
scored 1 to 5 by an independent model · 11 June vs 6 August 2026
Usefulness up 0.73, focus up 0.69, grounding up 0.50. Intent misclassifications halved, two down to one.
One caveat you should have, because it's the first thing we'd attack if someone showed us this chart: the June run covered 13 cases and the August run covered 19. We added six, and most of them are deliberately nasty grounding tests written from the failures in log entry 06. So the case set got harder between the two columns, which makes the grounding move more meaningful and every other column less exactly comparable. We're showing it rather than quietly re-running June's 13.
Grounding is still the weakest dimension at 3.58. On one case the system produced sensible weekly activity targets that weren't the specific targets Dean teaches, which is a milder version of the same mistake. The grounding gate fired and corrected a figure on three cases in this run, so the mechanism works. It doesn't catch everything yet.
Everything that broke, and what we did about it
This section is the reason the page exists. Anyone can show you an architecture diagram. What tells you whether a system is actually being run is the record of it failing, and the record of somebody noticing. These are real entries, dated, in the format we keep them in internally.
Read them as a checklist for your own build. Five of the six are mistakes you'd make too.
01 · The course reference held 171 characters
2026-07-21
2026-07-27
02 · The access roster silently halved, three times
03 · The call webhook had been dead since launch
04 · One member asked the same question thirty times
2026-08-04
05 · Group calls routed to nobody at all
06 · It invented software and percentages
The system watched its owner work and wrote down what it learned
Every week the system reviews recent coaching call transcripts and its own answers, and proposes behavioural rules for itself. Dean's judgement is the source and the rules are additions to how it responds. Seven are currently active. Here they are in summary, in our words rather than the system's, because the originals quote his teaching material directly.
When someone asks how fees or payment terms work, give the actual structure with a worked example. Deflecting to "it depends" is what causes the follow-up question.
After stating a profit figure, stop. Let the number sit there. Adding explanation immediately afterwards weakens it.
When someone is confused about numbers, offer a short recorded walkthrough they can pause rather than explaining the same thing again in text.
Give the next physical action, not the principle. "Walk into three agencies on Thursday" beats "build relationships with agents".
When someone brings a problem that's really about confidence, name that before solving the mechanics. Solving the wrong layer sends them back in a week.
Match the length of the answer to the size of the decision. A two-line question doesn't get a framework.
Never invent a tool name, a supplier or a percentage. If it isn't in the material, say it isn't and say what would settle it.
Rule seven is the interesting one, because the system proposed it after being caught doing exactly that. Rule two came from watching Dean handle silence on a sales call, which is not something anybody wrote into a brief.
This is the part of the build we'd point at if you asked what separates a system that's running from a system that was delivered. It gets better on a schedule, without us touching it.
Fourteen days to build, tens of dollars a month to keep
The arithmetic is the part most people get wrong, in both directions. The build is real work and it isn't cheap. What surprises people is that once it exists, the running cost barely moves as usage grows, because the expensive part is thinking and the cheap part is volume.
Rebuilding the entire knowledge base from scratch, all 7,787 chunks, costs under ten cents. We mention that because it's the number that tends to change how people think about this. The knowledge base isn't a precious asset to be protected. It's cheap enough to throw away and rebuild whenever the source material changes, which is what the daily job does.
Set against that: 205 conversations in a week, including fifteen people prepped in the half hour before a landlord call. The comparison isn't between this system and a cheaper system. It's between this system and Dean's evenings.
Fourteen days from kickoff to production. Nothing here needed a research team.
What it did need was somebody willing to look at the source connections and check their size, grade the output honestly enough to find the grounding problem, and keep running the thing after the invoice cleared. That's the whole job.
If you're the bottleneck in your own business, that's a solvable problem
We start by working out what's actually constraining you, which is usually not the thing that feels most annoying. If it turns out an AI build isn't the answer, we'll tell you that on the call.
Book a diagnostic call