Seven risks in recommendation systems and what they mean for a publisher

Recommender risks are usually discussed from the view of whoever wants a recommendation. This page takes the other two seats: the person receiving recommendations and the people running the system.

Risks and controls look different from those seats. Publishers should know them because controls, when they arrive, arrive as changes in what is eligible.

What to take away

The seven

Inference

A system trained on behavior will learn whatever behavior predicts, including things nobody typed in. Patterns of viewing can predict health, belief, circumstance and intent, and the system does not need to name any of these to act on them. The reader disclosed nothing and is being treated as though they had.

The control is to stop the system from using certain categories. That is harder than it sounds: categories are not inputs but patterns across inputs, so removing a field does not remove what the field predicted.

Operators respond with rules about which recommendations may not be made. Those rules land on publishers as categories of material that are harder to distribute.

Narrowing

The system predicts what you will do from what you did, shows you that, and then learns from your response to it. The loop tightens by itself. The reader ends up in a smaller world than they started in, without having chosen it.

The control is deliberate variety, and the feedback loop described from the inside is what it works against. The tension is a version of the exploration and exploitation dilemma: showing something uncertain costs measured performance now and may pay later. Under pressure on the numbers, exploration is the first thing cut.

Amplifying what provokes

Strong reactions arrive fast and are easy to count. A system optimizing on a fast countable response will find material that provokes one, without any decision to favor it. The outcome is a feed that leans toward outrage, fear and novelty, and the lean is emergent.

Controls take the form of slower, richer objectives: what people do afterwards, whether they come back, what they say when asked. These are more expensive to measure and slower to move. Every operator that has introduced them has done so after the fast objective produced a visible harm, which is the pattern to expect.

Starving the new

A new item or person carries no evidence. A system that leans on evidence has nothing to say about them. A system under pressure will not spend traffic finding out.

So being new is penalized, not by anyone's choice but by the absence of data. Being new is also the starting condition in what a publisher can and cannot plan.

The control is an exploration budget: a fraction of exposure reserved for the uncertain. It is measurable, it costs something, and it is the budget that gets trimmed when the quarter is bad. A publisher who is new should assume the budget is small and design for the moment when the system finally does look.

Fabricated behavior

If evidence is behavior, manufacturing behavior manufactures evidence, so fake accounts, purchased reactions and coordinated early response attack the system's sense of what is popular. They work in proportion to the system's trust in raw counts.

Once a few items are inflated, real readers respond to what looks popular, a bandwagon effect that launders the fabrication into genuine evidence.

Controls are detection, discounting, and rules against coordination. They arrive as eligibility constraints and enforcement, which is why an honest publisher can be caught by a rule aimed at someone else. Reading such rules as constraints outside the score explains their blunt shape.

Loss of shared context

When everyone sees something different, there is no common reference. Two people cannot argue about what was on the feed, because there was no "the feed". This is not a harm to any individual reader; it is a harm to the possibility of a shared conversation, and it has no control inside the system, because it is what personalisation is.

Over-trust

A recommendation reads as advice, and advice implies someone judged the thing worth recommending. Nobody did. The suggestion is a prediction that the reader will do the measured thing, and readers who do not know that treat predictions as endorsements. The control is honesty about what the suggestion means, which products rarely offer because it makes the product feel worse.

What this means for a publisher

Each control above becomes a rule or constraint for a publisher, and sensitive inference rules become categories that travel less. Exploration budgets decide how much chance a new account gets, while anti-fabrication rules decide what looks suspicious.

To watch for arriving controls, keep the change register current and read policy changes as controls against one of the seven.

Common questions

Are these risks the operator's fault?

They are the operator's responsibility, which is different. Each follows from a defensible choice; the fault, where there is one, is in not paying for the control until after the harm.

Can a reader protect themselves from narrowing?

Partly: by using explicit controls where they exist, by following deliberately outside their pattern, and by knowing that the feed is not a map of what exists. None of it is convenient.

Why do the controls keep weakening?

Because they cost the measured objective and the objective is what the team is judged on. Controls survive where a harm was public enough to be remembered.

Is there a version of recommendation without these risks?

Reduce personalisation and you reduce the first, second and sixth at the cost of relevance. Reduce reliance on fast reactions and you reduce the third at the cost of responsiveness. There is no free configuration, only different bills.