Rules
Part of Recommendation systems explained for 2027
Recommendation systems risks: facts, examples and context
Recommendation systems risks in seven kinds, from inference and narrowing to fabricated behavior, with what each one means for a publisher in practice.
A recommender's risks are usually discussed from the point of view of whoever wants to be recommended. This page takes the other two seats: the person being recommended to, and the people running the system. The risks look different from there, and the controls are different too. Someone who publishes should know them because the controls, when they arrive, arrive as changes in what is eligible.
What to take away
- Seven risks, none of them exotic. Each follows from a design choice that was reasonable on its own.
- Every control costs something the system is measured on, which is why controls erode under pressure and return after harm.
- For a publisher, the risks are context: they explain why rules tighten, and where.
The seven
| Risk | Borne by | Follows from |
|---|---|---|
| Inference about sensitive things | The reader | Behavior evidence predicts more than the reader meant to reveal |
| Narrowing | The reader | Optimizing on past behavior reinforces past behavior |
| Amplifying what provokes | Everyone | Strong reactions are cheap to measure and easy to predict |
| Starving the new | Publishers, then readers | Exploration costs measured performance now and pays off later |
| Fabricated behavior | The operator, then everyone | Evidence is behavior, and behavior can be manufactured |
| Loss of shared context | The public | Personalisation means no two people see the same thing |
| Over-trust in the recommendation | The reader | A confident suggestion reads as an endorsement |
Inference
A system trained on behavior will learn whatever behavior predicts, including things nobody typed in. Patterns of viewing can predict health, belief, circumstance and intent, and the system does not need to name any of these to act on them. The reader disclosed nothing and is being treated as though they had.
The control is to stop the system using certain categories, which is harder than it sounds, because the categories are not inputs; they are patterns across inputs. Removing a field does not remove what the field predicted. Operators respond with rules about which recommendations may not be made, and those rules land on publishers as categories of material that are harder to distribute.
Narrowing
The system predicts what you will do from what you did, shows you that, and then learns from your response to it. The loop tightens by itself. The reader ends up in a smaller world than they started in, without having chosen it.
The control is deliberate variety, imposed against the objective. The tension is a version of the exploration and exploitation dilemma: showing something uncertain costs measured performance now and may pay later. Under pressure on the numbers, exploration is the first thing cut, and the feedback loop described from the inside closes a little more.
Amplifying what provokes
Strong reactions arrive fast and are easy to count. A system optimizing on a fast countable response will find material that provokes one, without any decision to favor it. The outcome is a feed that leans toward outrage, fear and novelty, and the lean is emergent.
Controls take the form of slower, richer objectives: what people do afterwards, whether they come back, what they say when asked. These are more expensive to measure and slower to move. Every operator that has introduced them has done so after the fast objective produced a visible harm, which is the pattern to expect.
Starving the new
A new item or person has no evidence attached. A system that leans on evidence has nothing to say about them, and a system under pressure will not spend traffic finding out. The result is that being new is penalized, not by anyone's choice but by the absence of data.
The control is an exploration budget: a fraction of exposure reserved for the uncertain. It is measurable, it costs something, and it is the budget that gets trimmed when the quarter is bad. A publisher who is new should assume the budget is small and design for the moment when the system finally does look, as discussed in what a publisher can plan and what it cannot.
Fabricated behavior
If evidence is behavior, then manufacturing behavior manufactures evidence. Fake accounts, purchased reactions, coordinated early response: all of these are attacks on the system's sense of what is popular, and they work in proportion to how much the system trusts raw counts. Once a few items are inflated, real readers respond to what looks popular, a bandwagon effect that launders the fabrication into genuine evidence.
Controls are detection, discounting, and rules against coordination. They arrive as eligibility constraints and enforcement, which is why an honest publisher can be caught by a rule aimed at someone else. Reading such rules as constraints outside the score explains their blunt shape.
Loss of shared context
When everyone sees something different, there is no common reference. Two people cannot argue about what was on the feed, because there was no "the feed". This is not a harm to any individual reader; it is a harm to the possibility of a shared conversation, and it has no control inside the system, because it is what personalisation is.
Over-trust
A recommendation reads as advice, and advice implies someone judged the thing worth recommending. Nobody did. The suggestion is a prediction that the reader will do the measured thing, and readers who do not know that treat predictions as endorsements. The control is honesty about what the suggestion means, which products rarely offer because it makes the product feel worse.
What this means for a publisher
Each control above becomes, for a publisher, a rule or a constraint. Rules about sensitive inference become categories that travel less. Exploration budgets decide how much chance a new account gets. Anti-fabrication rules decide what looks suspicious. Watching for these controls arriving is a matter of keeping the change register current and reading policy changes as controls against one of the seven.
Common questions
Are these risks the operator's fault?
They are the operator's responsibility, which is different. Each follows from a defensible choice; the fault, where there is one, is in not paying for the control until after the harm.
Can a reader protect themselves from narrowing?
Partly: by using explicit controls where they exist, by following deliberately outside their pattern, and by knowing that the feed is not a map of what exists. None of it is convenient.
Why do the controls keep weakening?
Because they cost the measured objective and the objective is what the team is judged on. Controls survive where a harm was public enough to be remembered.
Is there a version of recommendation without these risks?
Reduce personalisation and you reduce the first, second and sixth at the cost of relevance. Reduce reliance on fast reactions and you reduce the third at the cost of responsiveness. There is no free configuration, only different bills.