915 Users, 11 Power Users, 4 Paying Customers

We build a product for supervising AI coding agents. For months, our dashboard said we had eight paying customers. Four of them were us.

Not fraud, nothing dramatic — team accounts, test purchases, a founder's own subscription bought to check that the Apple rail worked end to end. Each one was a perfectly reasonable thing to do at the time. Together they inflated the single number every other number in the business was derived from, and nobody noticed for months, because from inside the data there is nothing to notice. A subscription is a subscription.

This is a post about what we actually found when we audited our own analytics, and what the honest version of our numbers looks like. We are publishing it because we found the exercise more useful than any growth article we read that quarter, and because the specific way our metrics lied is a way yours might be lying too.

The honest shape of the business

Over three months, anchored on real signup and login events rather than "people who have an email address on file":

• 1125 registered users
• 18 who ran agent tasks on four or more separate days
• 11 of those once internal accounts are removed
• 4 genuine paying customers

Every one of those numbers was smaller than the version we had been quoting internally. Not because anyone was inflating them on purpose, but because each was calculated with a filter that seemed reasonable in isolation.

The user count came from a query scoped to "has an email" — which, in a shared analytics project, quietly included several unrelated products. The power-user count included our own team, who by definition use the product every day. The paying-customer count included the four internal subscriptions.

Individually, each is a five-minute mistake. Compounded, they described a business roughly twice the size of the real one.

The number that mattered most was the one nobody had

Here is the finding that changed what we work on.

110 external users started our free trial. One converted.

That is 0.91%. And the single conversion is stranger than the zero would have been: he subscribed two minutes and fourteen seconds after starting the trial. He never evaluated it. He arrived, hit a limit, paid to remove it, used the product for one day, and cancelled.

For a while we could not even see that he had cancelled. Our system emitted no cancellation event at all — not a broken one, none. We only learned about it because he replied to an email and mentioned it in passing.

If you take one thing from this post, take that: we could measure people arriving and could not measure them leaving. With a customer base you can count on one hand, a silent cancellation is a quarter of the business disappearing without a trace. We had built careful instrumentation for acquisition and none at all for the other direction, which is a very common shape and an expensive one.

What our best users actually told us

We wrote to the eleven external power users. Not a survey — a short personal email asking one question they could answer with a single letter.

Two of the first replies reframed the entire problem.

The first was from a customer who had paid:

"I stopped using the service because I no longer needed it. I have already canceled my subscription."

Not a complaint. Not a feature request. His use was project-shaped: he had a thing to do, the product helped him do it, and then he was finished. We had been treating a monthly subscription as the obvious model for what is, for many developers, episodic work. That is not a conversion problem. It is a packaging problem, and no amount of paywall copy would have surfaced it.

The second came from our single heaviest user by volume — someone who had dispatched thousands of commands:

"I 'ran out of credits' whenever I began a task, and eventually I could not even connect to the workspace. When it functioned, I had to guess the results because I could view only my own text."

We checked her account against our logs. Two of her three complaints were exactly what our own data showed: nine failed workspace deploys across two days, and a daily limit that must have been infuriating for someone working at her pace. Her rage-clicks stopped at 01:05:07. Her last failed deploy was at 01:05:34. Same minute.

Neither of them mentioned price.

Three ways our analytics were confidently wrong

Beyond the internal accounts, the audit surfaced three measurement bugs worth describing, because they are all generic.

A metric that counted the wrong thing. We had an event called task_dispatched and treated it as "tasks." It fires for every command type — listing models, resizing a terminal, fetching a conversation. In one thirty-hour window it fired 988 times, of which 215 were actual task starts. Every "tasks" figure we had quoted was inflated by roughly a factor of five. A user showing 261 of them was not abusing a free plan; they were scrolling.

An event that fired but measured nothing. We added an event to answer "what does a trial user actually do with the paid tier?" and shipped it at what we believed was the single choke point where a paid user takes a metered action. It was not. It covered three minor limits and missed the primary one entirely. It fired zero times in thirty hours against 215 real task dispatches — while looking, from the code, completely correct.

A chart that showed a collapse that never happened. One dashboard measured conversion using an event that only exists on one of our three payment rails. It had fired for a grand total of one person in a year. The funnel it drew was not pessimistic; it was fictional.

What we changed

We built the cancellation event we did not have. We fixed the two mis-scoped metrics. We re-pointed the charts that were measuring artifacts. We created an explicit "internal accounts" segment, because the built-in test-account filter was plainly not catching ours, and keeping that list current is now somebody's job rather than nobody's.

And we changed what we work on. When your two most detailed pieces of customer feedback are "I didn't need it anymore" and "it was broken," the honest reading is that you do not have a messaging problem. We spent the following week on deploy reliability rather than on the paywall.

The part worth stealing

You do not need our numbers. You need the habit that produced them.

Pick the single metric your decisions rest on most heavily. Then try to prove it wrong. Ask who is in the denominator and whether they should be. Ask what the event actually fires on, not what its name suggests. Ask whether you can measure the inverse — not just people arriving, but people leaving.

The failure mode is never a number that is obviously broken. It is a number that is plausible, directionally flattering, and quietly overstating the exact thing you are trying to improve. Those are invisible from the inside, which is why you have to go looking on purpose.

We found ours by counting our own team as customers for months. It cost us nothing but a bad quarter of confidence, and it was the most useful week of analysis we have done.