Reviews
Part of Feature updates: methods, tools and useful context
Feature updates updates 2027: facts and context
Feature updates updates 2027 covers staged delivery: flags, rings and holdbacks, why rolling out cannot be checked, and how to measure during one.
Two people using the same product on the same day can be using two different products. That is not a glitch and it is not rare: it is how changes are delivered now, and it explains most of the arguments about whether something exists yet. This page is about the delivery machinery, what it does to your ability to observe anything, and why "it is rolling out" is a sentence that can never be wrong.
What to take away
- A change is delivered to a slice, not to a population, and the slice is chosen by the operator for reasons that are never published.
- Because delivery is partial, your own experience is a sample of one drawn from an unknown distribution. It cannot settle whether a change is real.
- The one honest measurement is a comparison over time on your own property, and even that is contaminated while a rollout is in progress.
The machinery, in plain terms
Four mechanisms do almost all of the work.
A flag decouples shipping from releasing. The code goes out to everyone; a switch decides who runs it. That is what a feature toggle is for, and it is why a change can be present and invisible at the same time.
A ring expands the audience in steps. Internal users, then a small external fraction, then a larger one, with a pause after each step to look at the numbers.
A holdback keeps a slice deliberately without the change, so the operator has a comparison group. This is the only part of the process designed to produce evidence, and it is the part you never see the output of.
An environment separates what is being tested from what is live. The distinction between a deployment environment and production is what makes a staged release possible at all, and it is why the same version number can behave differently in two places.
What this does to observation
| What you observe | What it is compatible with | What you cannot conclude |
|---|---|---|
| You have the change, a peer does not | Any ring position, any flag rule | That it is "rolling out to everyone" |
| It appears, then disappears | A rollback, a bucket reassignment, a cache | That you imagined it |
| Two accounts behave differently | Different buckets, different account attributes | That one account is penalized |
| Your numbers move the same week | The change, the holdback, or neither | That the change caused the move |
The last row is the one that costs people quarters. If a rollout is in progress, some fraction of your audience has the change and some does not, and the fractions are moving. Your aggregate numbers are then a blend of two populations in unknown and shifting proportions. Comparing this week to last week under those conditions measures the blend, not the change.
Why "it is rolling out" cannot be checked
Take the sentence seriously for a moment. It asserts that some accounts have a change and others do not, and that the proportion is increasing. Every possible observation is consistent with it. You have it: consistent. You do not have it: consistent. Your peer has it and you do not: consistent. It disappears again: consistent, because rings contract too.
A claim compatible with every observation carries no information, which is not the same as being false. It is usually true. It just cannot do the work people want it to do, which is to tell them whether to plan around something. The general shape of the problem, a claim that no evidence could count against, is the same one that keeps ranking folklore alive long after it stops being true.
The way out is to stop asking whether the change is rolling out and start asking what you would do differently under each answer. If the answer is nothing, the question was not worth the hour.
Measuring anything during a rollout
You can still learn things, with two adjustments.
Segment before you aggregate. If you can tell which of your own accounts or surfaces have the change, compare within each group rather than across the blend. If you cannot tell, say so and treat the whole period as unreadable rather than reading it badly.
Extend the window past the rollout. Once delivery is complete the blend problem goes away, and a before-and-after comparison becomes interpretable again, subject to everything else that moved in between. The design rules for that comparison are in building a test that cannot flatter you, and the reason early numbers mislead so consistently is set out in why the first data is the worst data.
Reading the announcement against the delivery
An announcement and a delivery are two different events with a gap between them that is rarely stated. The announcement is written for a wider audience than you, and who it is actually addressed to determines almost all of its vocabulary. The delivery is what changes your product. Confusing them is why people plan against a date that was never a date.
When the two disagree, the operator's published page is what you cite and your own observation is what you act on. That split is uncomfortable and it is correct, for the reasons in why only the operator's own documentation has standing.
Common questions
Can I get a change early by changing a setting?
Sometimes, and it is usually a bad trade. A flag flipped outside the intended ring gives you a version nobody has tested against your configuration, and it can be withdrawn without warning. Worse, it makes your own account unrepresentative of your audience, which is the thing you were trying to observe.
Why would an operator hold a change back from some accounts on purpose?
Because without a group that did not get it, they cannot tell whether it helped. The holdback is the control. It is the single most defensible thing in the whole apparatus, even though it is the part that feels most arbitrary from outside.
How long does a rollout take?
Nobody outside can tell you, and the honest answer inside is that it depends on what the numbers do at each step. A rollout that hits a problem can pause for months or reverse entirely, and neither event is usually announced.
Does a rollback mean the change failed?
Not necessarily. It can mean an unrelated defect, a capacity problem, or a decision to re-sequence. Treating a rollback as a verdict on the feature is reading a single event as a conclusion, which is the error the change register exists to prevent.
