Recommendation systems research explained with examples

Research on recommenders comes in five kinds, and the kind decides what a result can mean. Most headline claims about recommendation, in the press and in vendor material, are one kind of result reported as if it were another. Learning to tell the kinds apart is more useful than any particular finding, and it needs no citation, which is why none appears here.

What to take away

• Offline evaluation, online experiment, user study, external audit and simulation each answer a different question. A result travels badly out of its kind.
• The gap between offline and online results is the central fact of the field. A method that wins on logged data often loses in front of people, for reasons that are structural.
• You can run a small audit yourself. It will not tell you how the system works. It will tell you how the system treats material like yours, which is the question you actually had.

Five kinds of evidence

Kind	What it does	What it can show	What it cannot show
Offline evaluation	Trains a method on logged behavior and scores it on held-back logs	That a method predicts the logged behavior better than another	What people would have done if shown something the log never showed them
Online experiment	Splits live traffic and measures an outcome	That a change moved the measured outcome, for that population, then	Whether the outcome was the right thing to measure
User study	Asks or observes people, usually in small numbers	Why people say they did something; how a design felt	What people do at scale when nobody is watching
External audit	Probes a live system from outside with controlled accounts or queries	What the system does with a given input, at the time of the probe	Why; and whether it still does
Simulation	Models readers and items and runs the system against them	What a mechanism would do if the model of people is right	Whether the model of people is right

The sentence to hold onto is in the first row. Logged behavior is a record of what people did with what they were shown. A method that scores well on it is a method that predicts the past distribution well. Whether it would do better at showing people something different is exactly what the log cannot contain, and this is the survivorship bias of the field: the data only contains the survivors of previous recommendation.

The offline and online gap

A method that improves offline scores by a clear margin and then fails to move anything online is the ordinary experience of the field, not an anomaly. Three structural reasons.

• The log was made by the old system, so it flatters methods that resemble the old system.
• The offline score is a proxy for the online outcome, which is itself a proxy for what anyone wanted. Two proxies deep, agreement is not guaranteed.
• People react to the whole product, not to a ranking in isolation. An improvement that changes the feel of a surface produces responses the log never had a chance to record.

This is also why claims that a change "improved recommendations" need the kind stated. Improved offline is a claim about a score. Improved online is a claim about a measured behavior. Improved for people is a claim nobody can make from either, and the proxy problem at the heart of every feed is why.

Reading a claim

When a claim about recommendation reaches you, ask, in order:

1. Which kind of evidence? If the claim does not say, it is probably offline or simulation dressed as online.
2. What was measured? The proxy, named. "Engagement" is not a measurement; a click within a session is.
3. On what population, when? A result from one product's users in one season is a result about them, then.
4. Who benefits from the claim? A vendor, a platform, a critic and an academic have different reasons to publish. None is disqualifying; all shape emphasis.
5. Does it survive the gap? If offline, assume the online effect is smaller or absent until shown otherwise.

A claim that passes all five is rare and worth keeping. Most fail at the first question, and failing there does not make them false, only unweighable.

Running your own audit

You cannot run an offline evaluation or an online experiment on a platform you do not own. You can run an audit, and an audit answers the question you had.

Set up. Two or more accounts that differ in one respect you care about: new against established, one subject against another, one region against another. Keep everything else the same, including how you use them.

Probe. Publish comparable items, or consume comparable material, on a schedule, and record what each account is shown and how each account's items are treated. Same day, same hour, same format.

Compare over time, not at a point. A single comparison is noise. What you want is whether the difference between accounts changes when something changes, which is the logic of a difference in differences design: the account that should not have been affected is the control for the one that should.

Read narrowly. The result is a statement about how this system treated these accounts on these days. It is not a statement about how the system works, and it does not transfer to another surface or another season; note the product changes during the audit window in your change register so you know what the season contained. It is still the most reliable evidence you will ever hold, because it is about you.

Any audit that involves creating accounts or automated access has to stay inside the platform's terms; check those first, and prefer manual probing with accounts you genuinely use. The point is observation, not manipulation.

Two examples, invented

A publisher believes new accounts are held back. An offline study could not test this, because the log contains no counterfactual for the new account. A user study would give opinions. An audit does the job: a new account and an established one publish matched items for a month. If the gap between them closes as the new account accumulates history, the effect is cold start, as described in why a recommender knows nothing about anything new. If the gap stays fixed, something else is operating, and the audit has narrowed the question.

A vendor claims its method lifts engagement. Ask the five questions. Usually the answers are: offline, a click proxy, the vendor's own customers, last year, and no. That is not proof the method is worthless. It is a statement that the claim, as made, weighs nothing yet.

The habit of refusing untestable explanations, which this whole method depends on, is the same one set out in what a ranking input can and cannot be.

Common questions

Is academic research on recommenders useful to a publisher?

As a source of mechanisms, yes: cold start, concentration, feedback loops all have careful treatments. As a source of tactics, no; the mechanisms are general and the tactics are not.

Why do platforms publish so little of their own research?

Some do publish, mostly offline and simulation work. Online results are commercially sensitive and go stale fast, and a published objective is a gamed objective.

Can an audit get my account penalized?

If it breaks the platform's rules on automation or multiple accounts, yes. Read the rules, keep it manual and small, and remember that the purpose is to observe, which no rule forbids.

How much should a single audit change what I do?

One decision, narrowly scoped to the surface and the period. If the result matters, repeat the audit a season later before building on it.