Ranking signals research: practical details and examples

You cannot see inside a ranking system, but you can run an experiment against it. The system is a black box that takes your pages as input and returns positions and traffic. Change one input on part of your property, leave a comparable part alone, and compare the two. That is the only method available to an outsider that produces evidence about your own site rather than about an average of other people's sites. It is slower and narrower than reading a study, and it is worth more.

What to take away

• The unit you randomise is the thing you changed, usually a page or a template, not a visitor.
• A control group is not optional. Before and after on the treated group alone measures the season, the news, and everything else that happened that month.
• Decide the metric, the duration and the decision rule before you look at any data. Everything after that is arithmetic.

Set the question narrowly

A test answers one question about one property over one period. "Does adding a comparison table to product pages change organic entrances to those pages" is testable. "Does quality matter" is not. If you cannot write the question so that a number would settle it, you are not ready to run anything.

Narrow also means one change. Ship three edits together and a positive result tells you the bundle helped, which is fine if you intend to keep the bundle forever and useless if you wanted to know which part earned its place. Bundles are a reasonable choice when the changes are cheap and you do not care about attribution. Be honest with yourself about which situation you are in.

Build the split so it cannot flatter you

The most common mistake is splitting on something related to the outcome. New pages against old pages, or one category against another, guarantees a difference that has nothing to do with your change. Assign pages to the treated and control groups by something arbitrary and stable, such as a hash of the identifier, then check that the two groups looked alike for several weeks before you touched anything. If they did not track each other before, they will not track each other after, and the test is dead before it starts. The general design is ordinary split testing with a slow and noisy response variable.

Two constraints make this harder than the usual on-site experiment. Nothing takes effect until the change is discovered, so the clock starts at recrawl and not at deploy. And the response is a position in someone else's system, which moves for its own reasons the entire time. Both of those push the required duration up.

Decide these before you start

Decision	Set it to	Why it has to be beforehand
Primary metric	One number, chosen now	Five metrics give five chances to find a win by accident
Unit of analysis	The thing you changed	Counting sessions when you changed pages overstates certainty
Minimum effect worth acting on	A number you would act on	Without it, any result becomes interesting after the fact
Duration	Long enough to cover a full weekly cycle and recrawl	Stopping when the line looks good is how noise becomes a finding
Stopping rule	Written down, including the null case	A test you would keep running until it wins is not a test
What would make you abandon the idea	Something specific	If nothing would, you are collecting support, not evidence

The row that gets skipped most often is the minimum effect. It is also the one that decides whether the test is possible at all. If the change you expect is a few percent and your weekly variation is larger than that, no length of run will separate them. Work out the statistical power you have before spending a quarter on a question your traffic cannot answer, and if the answer is that you cannot detect it, that is a useful finding on day one rather than an ambiguous chart in month three.

Read the result honestly

Compare the change in the treated group against the change in the control group over the same window. Not treated before against treated after. That single discipline removes most of the ways a seasonal swing or a platform-wide shift gets reported as a win.

A null result means the effect was not large enough for you to detect with this much data. It does not mean the effect is zero, and it does not mean the idea was stupid. Record it anyway, with the size you could have detected, because the next person asking the same question deserves to know what has already been ruled out at what sensitivity.

A positive result is about your site, your period, and the version of the system that was running. It is not a law. Publishing it as one is how the field ends up with confident advice that stops working and nobody can say when. The wider argument about which kinds of evidence carry what weight sits in the comparison of evidence types, and the ranking signals overview covers the vocabulary.

Common questions

How long should a test run?

Long enough for the change to be discovered, plus enough full weekly cycles to average out the day of week pattern. In practice that is weeks, not days. The exact number falls out of your traffic volume and the effect size you set in advance.

Can I test on a small site?

You can run the test. You often cannot detect anything, because the noise on small numbers swamps a modest effect. On a small property, spend the effort on changes that are obviously right for a reader rather than on measuring them.

What if the platform changes mid-test?

Your control group is exposed to the same change, which is precisely why it is there. A platform-wide shift moves both groups and the difference between them survives. A shift that hits only one group means your split was not arbitrary after all.

Should I test the same thing more than once?

Yes, if the decision matters. A single positive result at a conventional threshold is weaker than people treat it. Repeating on a different section, or in a different quarter, is the cheapest way to find out whether you measured an effect or a moment. Worked cases of the same idea behaving differently in different contexts are collected in the examples of recommendation behavior, and the mechanics of how discovery reaches your pages are set out in how a feed gets assembled.