Customers Love the Demo but Don't Use the Product. What Should the Deck Say?
Customers Love the Demo but Don’t Use the Product. What Should the Deck Say?
Two facts are on the same slide, and they don't get along. In a room, the people who do the work responded well to what you showed them. In a database three weeks later, the number is thin or zero. The person preparing the deck usually has to choose one fact, and the room's reaction is the friendlier one.
Don't choose, and don't average. Work in this order instead. Check whether the two signals describe the same people, the same product state, the same task, and the same period — if they don't, you have two studies, not a contradiction. Verify that your usage counter could have registered the behavior at all. Then set the plausible explanations side by side and keep only the ones you could lose. Finally, put one check in the deck with a date and the decision it changes. What comes out the other end usually says less than the team hoped and more than last week's version, and it survives the follow-up question.
Here is an invented case to work through. A small software company sells a handoff tool: a procurement coordinator assembles a document set, sends it to a supplier, and the supplier acknowledges. One manufacturer's procurement group agreed to a pilot. Four coordinators attended a prepared forty-minute demo; it ran on the vendor's clean sample data because the customer's own files weren't cleared for the vendor's environment. Accounts were opened for all four the following Monday, and the pilot agreement covers four seats. The coordinators were engaged — they asked questions, and two asked to see the tool run on their own export format. Nobody in the room said the tool was wrong for the job. Over the next twenty-one days, the tool recorded two completed handoffs across the four accounts. The counted event is defined: a handoff is recorded when the coordinator dispatches the document set from inside the tool. Almost nothing else about those three weeks is known yet.
Before you explain it, check that the two signals describe the same thing
A conflict is only a conflict under matched conditions. Four things have to line up: the people, the product state, the task, and the window. Check each one before you let yourself write "customers love it but don't use it."
The people. If the demo was full of the people who sign and the quiet dashboard tracks the people who type, you have two different studies with two different subjects. That isn't a contradiction; it's a distribution problem, and it calls for a completely different slide. In the invented case the four coordinators attended the demo and the four coordinators appeared in the usage records, so this axis holds. That's the strongest version of the conflict and worth arranging if you can, because it removes the easiest way out.
The product state. Demos rarely run on the thing you shipped. They run on prepared sample data, in a rehearsed sequence, sometimes on a build that isn't the release. A demo environment arrives already populated; the first real session starts on a blank screen with the customer's own messy exports. When the demo uses different inputs than the working product, you are not comparing reactions to the same object.
The task. A demo shows a task at its best: complete inputs, one clean pass, no revisions, no one else's approval needed. Ask what the ordinary version of that task looks like — how often it comes up, how many people touch it, how messy the inputs are. Frequently the demonstrated job and the daily job are relatives, not twins.
The window. Twenty-one days is short if handoffs run on a monthly purchasing cycle. It might contain the entire opportunity for one coordinator and none at all for another. A short window after access is standard in a pilot and dangerous for interpretation.
Only after all four line up do you have a contradiction worth explaining. Before that you have a hypothesis about a contradiction, which is a different and much weaker thing to put in front of an audience.
A missing completion is not yet a behavior
"Not used" sounds like one state. It's at least four: not needed, not possible, not recorded, and done somewhere else. Your dashboard cannot pick between them, and neither can you until you go looking.
The UK Government Digital Service's guidance on measuring completion rate is a useful discipline here, within its own limits: it treats a transaction as having a defined start and a defined end, counts people who save and come back, and takes account of services that mix digital and non-digital steps (the page was updated in February 2021). Read narrowly, that's a checklist for the questions your counter isn't answering. Where does your counted event sit in the real sequence? Can someone start, leave, and finish a week later? Does part of the job happen off the tool?
In the invented case, the tool records a dispatch. But the supplier's acknowledgment arrives by email, and a quality reviewer signs the document set before it goes out — a step that exists in this customer's process and that nobody has examined. There are obvious ways the work could have happened without producing a single record: a coordinator drafts in the tool and sends the final package from their own mail client; a reviewer signs outside the system; a handoff sits half-finished past the cutoff. Each of those looks identical to disinterest in a chart.
Then there's the denominator. Two completions is a number. Two completions out of how many handoffs that were actually due is a finding. You cannot compute the second one without the customer's purchasing calendar, and until you have it, "low use" is a description of a record, not of a person.
One more thing before the arithmetic starts feeling scientific: you have four accounts and a three-week window. Even if all four had completed a handoff, you would have four observations. Don't let anyone on the team reach for a percentage, and don't let the deck carry one.
Explanations you could lose
Four explanations are worth comparing in the invented case, and each one implies different evidence. Write down, for each, what you would have to see for it to be wrong. An explanation you cannot lose isn't an explanation; it's a belief wearing a lab coat.
The demo sold something adjacent to the job. The polished sequence — bulk assembly from a folder, one clean pass — may be a capability these coordinators don't need on a routine basis. Their ordinary handoff might be three documents, two revisions, and a supplier who wants it by email. What would count against this: they use the tool readily once it's pointed at a real, ordinary handoff.
A practical barrier stops the job inside the tool. Missing roles, a sign-off step with no account, a permission that blocks the last action. What would count against it: a handoff completed end to end by the people who attended, without a workaround.
The task didn't arise in the window. The purchasing cycle may be monthly, or the coordinators may have had no orders at the handoff stage in those three weeks. What would count against it: the calendar shows handoffs were due and nothing happened.
The counted route isn't the route they use. The work happened, somewhere else. What would count against it: no trace of the handoff in email, chat, or the older system during the same window.
The explanation that everyone reaches for first — they were just being polite in the room — is the hardest to test and is usually asserted precisely because it needs no evidence. It can be made testable, but notice what it has to survive in this case. Two coordinators asked for something specific: the tool running on their own files. That's a small artifact and it doesn't prove intent, but it is the opposite of a polite no, and it is worth writing down before someone overwrites the memory of the room with a shrug.
There is a fifth possibility that stays closed for now: the product works as described and isn't worth changing an established routine for. That one becomes assessable only after opportunity, access, and measurement are ruled out. Keep it on the list. An investigation that can only end in "adoption problem" isn't an investigation.
Pick the check, not the story
Three next actions are available, and they are not interchangeable.
Revising the demonstrated promise means changing what the next session shows — their files, their ordinary handoff, the messy version. Investigating the actual workflow means going and looking at how a handoff really gets done, including the parts that never touch software. Changing the next trial's conditions means making it possible to observe the behavior at all: a window matched to the cycle, accounts that cover everyone who touches the document set, a completion definition that matches where the work ends.
What you should not do is widen the rollout to more teams. That multiplies an unreadable signal and is often the way a team avoids admitting it doesn't know what the current numbers mean.
In the invented case, the check that belongs to changing the next trial's conditions is the one that can dissolve the contradiction outright, and it's also the cheapest: reconstruct the denominator. How many handoffs were due across those four coordinators in the twenty-one days, and of those, how many could have been recorded as complete given where the counted event sits? If the honest answer is "one, and it was," then there is no adoption problem to explain yet, and the deck's usage line changes meaning entirely. If the answer is "six, and two were dispatched," you now have a real question — and the next step is a supervised handoff with one coordinator on a live order, watched end to end, so you learn where the work goes when it leaves the tool.
State what each outcome would do. If access is the problem, you fix roles and re-measure. If opportunity is the problem, you extend the window to a full cycle. If handoffs were due, access was fine, and the work happened somewhere else, the problem is product fit, not measurement. And write down the date and the owner in the deck. A test without a date reads as a stall, because it usually is one.
What the slide says while the question is open
Paired evidence, separate scopes, limits printed next to the numbers. Something like this:
Pilot read-out — handoff tool
Customer X, four procurement coordinators
Feedback (observed, demo conditions)
Four coordinators attended a prepared 40-minute demo on our sample data.
Two asked to see the tool run on their own export format.
Scope: a designed session with clean inputs and a guide.
Does not establish fit for an ordinary handoff.
Use (observed, 21 days after accounts opened)
Two handoffs dispatched in the tool, across four accounts.
Counter fires on in-tool dispatch of a document set.
Window is shorter than one purchasing cycle; sign-off step not yet reviewed.
Open question
How many handoffs were due in those 21 days?
Next step
Reconstruct the denominator from the purchasing queue. If handoffs were due,
run one supervised handoff end to end on a live order.
Owner: [name]. Check by: [date].
Decision affected: whether to revise the demo, extend the window to a full
cycle, or investigate product fit.
Compare that with the line it replaces: "Strong customer validation; usage building as teams activate." Everything in that sentence is doing illicit work. "Validation" fuses two measurements with different scopes. "Building" implies a trend, and two completions is not a trend. "Teams" implies more than four people. Someone in the audience will ask how many accounts, over what period, doing what — and you will either have the answer or lose the room's trust on the spot. Put it in first, with its boundary, and the question becomes a sign that the deck was read.
There is a real cost here and it's worth naming rather than papering over. An unresolved slide sounds weaker than a confident one. A deck that says "we don't yet know whether the task arose" is asking the audience to trust a process instead of a number. The trade is that the number was going to be interrogated eventually, and the version where you're surprised by it is worse than the version where you scheduled it. If the check comes back in three weeks and the answer is that handoffs were due and nothing happened, the deck changes then — and it changes honestly, with a finding rather than a rewrite.
So the slide keeps both facts, each with the conditions under which it was observed, and one question it can't yet answer. The deck should leave the room with the same question the team is actually going to answer, and a date on which one of the two facts will mean something different.
Frequently asked questions
Why not simply report either the positive demo feedback or the thin usage number?
The two facts may describe different conditions, and choosing one hides that. First check whether they involve the same people, product state, task, and period. In the invented case the four coordinators appear in both, but the demo ran on clean sample data and the 21-day window is shorter than a purchasing cycle.
What does 'not used' actually conflate?
At least four states: not needed, not possible, not recorded, and done somewhere else. A dashboard cannot distinguish them. In the invented case, a coordinator could draft in the tool but send from email, a reviewer could sign outside the system, or a handoff could sit unfinished past the cutoff; each looks like disinterest in a chart.
Which explanations should be compared, and what makes one testable?
The article compares four: the demo sold an adjacent job, a practical barrier stopped the work inside the tool, the task did not arise in the window, and the counted route is not the route used. For each, write what would count against it. The polite-explanation idea is hardest to test and usually asserted without evidence. A fifth possibility, that the product works but is not worth changing routine for, becomes assessable only after opportunity, access, and measurement are ruled out.
What check is recommended before changing the promise or widening the rollout?
Reconstruct the denominator: how many handoffs were due across the four coordinators in the 21 days, and how many could have been recorded as complete. If one was due and it was completed, there may be no adoption problem to explain yet; if six were due and two were dispatched, there is a real question. Then a supervised handoff on a live order can show where the work goes. Widening rollout is not recommended.
How should the slide read while the question is open?
Keep paired evidence, separate scopes, and limits next to the numbers. The sample slide shows feedback observed under demo conditions, use observed over 21 days, the open question about how many handoffs were due, and a next step with owner, date, and decision affected. It replaces lines like 'Strong customer validation; usage building as teams activate,' which fuses measurements and implies a trend and more people than the four accounts.