Industry
Part of Recommendation systems: cold starts and feedback loops
How to think about recommendation systems research
Recommendation systems research and its five kinds of evidence: the gap between offline and live results, reading a claim, and auditing your own material.
Research on recommenders comes in five kinds, and the kind decides what a result can mean. Most headline claims about recommendation, in the press and in vendor material, are one kind of result reported as if it were another. Learning to tell the kinds apart is more useful than any particular finding.
What to take away
- Offline evaluation, online experiment, user study, external audit and simulation each answer a different question. A result travels badly out of its kind.
- The gap between offline and online results is the central fact of the field. A method that wins on logged data often loses in front of people, for reasons that are structural.
- You can run a small audit yourself. It will not tell you how the system works. It will tell you how the system treats material like yours, which is the question you actually had.
Five kinds of evidence
What it does
- Offline evaluation
- Trains a method on logged behavior and scores it on held-back logs
- Online experiment
- Splits live traffic and measures an outcome
- User study
- Asks or observes people, usually in small numbers
- External audit
- Probes a live system from outside with controlled accounts or queries
- Simulation
- Models readers and items and runs the system against them
What it can show
- Offline evaluation
- That a method predicts the logged behavior better than another
- Online experiment
- That a change moved the measured outcome, for that population, then
- User study
- Why people say they did something; how a design felt
- External audit
- What the system does with a given input, at the time of the probe
- Simulation
- What a mechanism would do if the model of people is right
What it cannot show
- Offline evaluation
- What people would have done if shown something the log never showed them
- Online experiment
- Whether the outcome was the right thing to measure
- User study
- What people do at scale when nobody is watching
- External audit
- Why; and whether it still does
- Simulation
- Whether the model of people is right
The sentence to hold onto is in the first row. Logged behavior records what people did with what they were shown, and a method that scores well predicts the past distribution well.
Whether it would do better by showing people something different is what the log cannot contain. This is the field's survivorship bias: the data holds only survivors of previous recommendation.
The offline and online gap
A method that improves offline scores by a clear margin and then fails to move anything online is the ordinary experience of the field, not an anomaly. Three structural reasons.
Why offline gains vanish online
- Log made by old system flatters lookalikes
- Offline score proxies online outcome, itself a proxy
- People react to whole product, not ranking alone
This is also why claims that a change "improved recommendations" need the kind stated. Improved offline is a claim about a score. Improved online is a claim about a measured behavior. Improved for people is a claim nobody can make from either, and the proxy problem at the heart of every feed is why.
Reading a claim
When a claim about recommendation reaches you, ask, in order:
- Which kind of evidence?If the claim does not say, it is probably offline or simulation dressed as online.
- What was measured?The proxy, named. "Engagement" is not a measurement; a click within a session is.
- On what population, when?A result from one product's users in one season is a result about them, then.
- Who benefits from the claim?A vendor, a platform, a critic and an academic have different reasons to publish. None is disqualifying; all shape emphasis.
- Does it survive the gap?If offline, assume the online effect is smaller or absent until shown otherwise.
A claim that passes all five is rare and worth keeping. Most fail at the first question, and failing there does not make them false, only unweighable.
Running your own audit
You cannot run an offline evaluation or an online experiment on a platform you do not own. You can run an audit, and an audit answers the question you had. The habit of refusing untestable explanations, which the whole design depends on, is set out in what a ranking input can and cannot be.
Set up. Two or more accounts that differ in one respect you care about: new against established, one subject against another, one region against another. Keep everything else the same, including how you use them.
Probe. Publish comparable items, or consume comparable material, on a schedule, and record what each account is shown and how each account's items are treated. Same day, same hour, same format.
Compare over time, not at a point. A single comparison is noise. What you want is whether the difference between accounts changes when something changes, which is the logic of a difference in differences design: the account that should not have been affected is the control for the one that should.
Read narrowly. The result is a statement about how this system treated these accounts on these days. It is not a statement about how the system works, and it does not transfer to another surface or another season; note the product changes during the audit window in your change register so you know what the season contained. It is still the most reliable evidence you will ever hold, because it is about you.
Any audit that involves creating accounts or automated access has to stay inside the platform's terms; check those first, and prefer manual probing with accounts you genuinely use. The point is observation, not manipulation.
Two examples, invented
A publisher believes new accounts are held back. An offline study could not test this, because the log contains no counterfactual for the new account. A user study would give opinions. An audit does the job: a new account and an established one publish matched items for a month. If the gap between them closes as the new account accumulates history, the effect is cold start. If the gap stays fixed, something else is operating, and the audit has narrowed the question.
A vendor claims its method lifts engagement. Ask the five questions. Usually the answers are: offline, a click proxy, the vendor's own customers, last year, and no. That is not proof the method is worthless. It is a statement that the claim, as made, weighs nothing yet.
Cold start, concentration and feedback loops are all described from the inside in why a recommender knows nothing about anything new.
Common questions
Is academic research on recommenders useful to a publisher?
As a source of mechanisms, yes: cold start, concentration, feedback loops all have careful treatments. As a source of tactics, no; the mechanisms are general and the tactics are not.
Why do platforms publish so little of their own research?
Some do publish, mostly offline and simulation work. Online results are commercially sensitive and go stale fast, and a published objective is a gamed objective.
Can an audit get my account penalized?
If it breaks the platform's rules on automation or multiple accounts, yes. Read the rules, keep it manual and small, and remember that the purpose is to observe, which no rule forbids.
How much should a single audit change what I do?
One decision, narrowly scoped to the surface and the period. If the result matters, repeat the audit a season later before building on it.
