Our retention magic number was a mirage

We found our magic number in August. Six levels.

Plot how likely a new player is to come back the next day against how many levels they cleared in their very first session, and the chart is almost too tidy. Flat and low up to about five. Somewhere between five and six it roughly doubles. Then it keeps climbing.

About half of our new players were finishing their first session down in the flat part. The brief writes itself.

So we built it

We went at first-session depth head on. The opening levels were made quicker to read and quicker to clear, we pulled out the interruptions that fired before the threshold, and the moment a brand-new player finds their first word got rebuilt so that what the game is asking of you can't be missed.

It worked. That's the annoying part. In one release, the share of new players clearing six or more levels on their first day went up by about a third.

Day-one retention didn't move. Not slightly — it sat exactly where it had been sitting for two months, and it is sitting there now.

The number was describing them, not steering them

A player who clears six levels in one sitting has told you something about themselves before your level design gets a vote. They were in the mood, they had twenty minutes, and they had nothing better to do with them. Depth was a measurement of that, and the intent is what comes back tomorrow — not the depth.

Make the levels faster and you move the measurement. You haven't been anywhere near the thing it was measuring.

Written down now it looks obvious, which is the thing about this particular trap. It is the oldest one in product analytics and it still catches everybody, because the correlation is strong, the threshold is sharp, and shipping against a proxy feels identical to shipping against the real thing right up until the results come in. "Players who do X retain better" is not a finding. It is an invitation to ask the actual question, which is: and what happens if we make them do X?

There's a cheap way to ask that before you build anything. Go back through your own release history, find a change that already moved the proxy, and check whether the outcome followed. We had one of those. We didn't look.

Two kinds of churn, and we fixed the wrong one

The deeper mistake was treating early churn as one thing.

It's two, with opposite causes, and the bigger one is invisible if you go in looking for difficulty — which we did, because difficulty is the thing a level designer knows how to fix.

The early group isn't stuck. They leave around twenty seconds into a level. Nobody is defeated at twenty seconds. Almost everyone who starts the first level finishes it, so the first level isn't hard. And they leave with their hints unspent — they don't exhaust the help we gave them and give up, they never reach for it once. There's also a slice of installs that opens the app, registers a screen view and never starts a level at all, which is the single largest hole in our funnel and the one I think about most.

None of that is difficulty. It's cold start. You can't fix it by making the puzzle easier, because they were not trying to solve the puzzle.

The second group is genuinely stuck, and they're much further in, at the point where the content first asks something real of you. Their median time on the last level they ever touch roughly doubles, and a third of them spend over two minutes on it before they go. That's difficulty churn. It responds to difficulty work. It is also a good deal smaller than the first group.

So we spent a release tuning the opening for a population that wasn't failing it.

One number that actually had a shape

Out of everything we pulled, one measurement behaved differently enough to be worth something.

Take the first time a player is ever offered a rewarded video — in our case a rescue, which fires automatically when they stall rather than being hunted for — and compare how often it's actually watched, by ordinal. The first one is far worse than the second. The second is worse than the third. It recovers from there, and keeps recovering.

The shape is the finding, not the level. A flat line would mean players simply dislike video ads, and the fix would be to show fewer of them. A line that starts low and climbs means the first one is a surprise, and the fix is something you do once per player: say what's about to happen before it happens, and then never again.

That's a far cheaper intervention than rebuilding an ad strategy, and we'd never have found it in the average.

Reasons we might still be wrong about that

This reading is correlational too, and having just been caught by one correlation we are trying hard not to do the same trick twice in one quarter. Players who were already on their way out may tap and close, which would produce the same curve from the opposite direction.

What argues against it: it's the only placement in the game that fires unprompted on a single tap; its completion rate is anomalously low against other placements running on the same ad stack; and the monotonic recovery by ordinal is hard to explain if the cause is the kind of player rather than the moment.

The high ordinals are badly survivorship-biased, obviously. A player on their eleventh rescue has self-selected as somebody who tolerates rescues. The first-attempt row is the clean one, because every player appears in it exactly once, and that row is what the whole argument rests on.

So it's being tested rather than believed. If the test comes back flat we'll write that up too.

What I'd tell someone else

• A threshold in a correlation describes your users. It isn't a lever on them. Before you build against one, go and find a release that already moved it and check whether the outcome moved as well. That takes an afternoon and it would have saved us a release.
• Split your churn before you diagnose it. Ours looked like one curve and turned out to be two populations with opposite causes, and the larger one was invisible because it leaves before it does anything we had thought to measure.

And prefer findings that have a shape to them. "This number is bad" only tells you to improve it. "This number is bad and then recovers" tells you what kind of fix it needs, which is worth a great deal more and costs one extra group-by.

We kept the six-level work, for the record. The opening is better than it was, new players get further into it, and that's worth having on its own terms. It just wasn't the retention fix, and I'd like to be able to say we only needed the data to tell us that once.

We build Wordmere, a cosy word game where the words you find assemble a small world around you. We publish the shapes of our numbers rather than the numbers themselves; the ratios and thresholds here are real, the absolute levels are ours to keep. These posts are the things that went wrong on the way.