Feature updates statistics 2027: guide and criteria

This page was supposed to carry numbers: how many changes shipped this year, what share reached everyone, how quickly people adopted them. It carries none, and the reason is not that the research was skipped. The numbers do not exist in any form that would survive being quoted, and the useful thing to explain is why. Once you see where each one breaks, you stop looking for it and start asking the question that can be answered.

What to take away

• There is no register of product changes, so any count of them is a count of announcements, which is a different thing with a different bias.
• Staged delivery makes "released" a fuzzy state rather than a date, so a rate of anything per period has no clean denominator.
• Every adoption figure in circulation is either self-reported by the party that benefits from it or drawn from a sample that selected itself.

The four failures, in order of severity

What you wanted	Where it breaks	What that does to the number
A count of changes	No complete register exists; only announced changes are visible	Counts announcements, and announcement policy varies enormously
A release date	Delivery is staged and partial for weeks	The date is a range, and the range is undisclosed
An adoption rate	The denominator is unknown from outside	The rate is a ratio with one made-up term
A satisfaction figure	Respondents chose to respond	Measures who cared enough to answer

The first row is the one people underestimate. A change register would have to include every change, including the ones nobody announced, and the ones that were rolled back before anyone noticed, and the ones that only affected a fraction of accounts. No such register is published by anyone. What is published is a stream of announcements, and the decision to announce is itself a product decision made for reasons that have nothing to do with counting.

Why a staged rollout destroys the denominator

If delivery were a switch, a release would have a date and the arithmetic would work. It is not. A change reaches a slice, gets watched, reaches a bigger slice, occasionally goes back. Ask when it shipped and there are several defensible answers, and the operator is the only party who could pick between them.

This is not a small measurement inconvenience. Almost every quantity people want here is a rate: changes per quarter, share of accounts reached, days to adoption. A rate needs a population and a period, and staged delivery makes both of them soft at once. The concept being violated is operationalization: before you can count something you have to define it in a way that two people would apply identically, and "released" cannot be defined that way from outside.

Why the survey route does not rescue it

The obvious response is to ask people. Surveys can be good, and the reason this one fails is specific rather than general.

The sample is not drawn. Anyone reporting on product change adoption is surveying people who are reachable, interested, and willing, which is a group defined by the outcome you are trying to measure. That is selection bias at the recruitment stage, and it is not fixable by weighting because you have no frame to weight toward.

The response rate is unreported, or reported and small. What a response rate tells you is how much room there is for the non-respondents to differ from the respondents, and in this field the room is usually enormous.

The question wording does the work. Asking whether someone has "adopted" a change invites a yes from anyone who has seen it. The professional literature on this is unambiguous: AAPOR's account of what a defensible survey has to disclose sets out the items that are missing from essentially every figure circulating about product changes.

What can be answered

Three questions do have answers, and they are more useful than the ones that do not.

What changed for me, and when did I see it? This is observable on your own property, needs no population, and is the input to every decision you actually face. Recording it is the whole point of keeping a change register.

What did the operator publish, and on what date? Also observable, and it is the only citable statement about the product, for the reasons set out in why the operator's own documentation is the only source with standing.

Did the change move anything of mine? Answerable by a comparison you construct, not by a figure you look up. The design that makes such a comparison readable is in how to build a test that cannot flatter you.

The rule this leaves you with

If a figure about product changes has no stated population, no stated period, and no stated source of the denominator, it is decoration. That is not a high bar and almost nothing clears it. When you meet a number that does clear it, the habit of reading a whole report in twenty minutes will tell you within a page whether the rest of the document treats its own figures that carefully.

Common questions

Are the vendors hiding these numbers?

Mostly they are not computing them either, at least not in a form that would mean anything published. Internally an operator tracks its own metrics against its own definitions, which are not comparable to anyone else's and are not meant to be.

What about counting entries in a public changelog?

You can count them, and the count is real, but it measures editorial practice. Two operators with identical change rates and different documentation habits will produce wildly different counts. Say what you counted and the number becomes honest and much less interesting.

Could a large enough panel fix the sampling problem?

Size is the wrong lever. A panel recruited the same way will be wrong the same way at any size, and a bigger sample only narrows the interval around a biased estimate. What would fix it is a frame, and there is no frame for "everyone who uses this product".

So what do I put in a report when someone asks for the number?

Say what is measurable, name the body or the document that would have to publish the rest, and give the decision the number was supposed to inform. In practice the decision usually does not need the number, which is the most useful thing this exercise turns up. The discipline for reading a claim about a system before repeating it is the same one, applied to somebody else's page.