A closer look at annual trend reports research

There is a strategy question underneath every trend report, and it is not which trend to follow. It is how much of your effort should depend on the current configuration of somebody else's system at all. The arithmetic of that question is unfavourable in a way that is easy to demonstrate and hard to accept, because the alternative is slower and less exciting. This page makes the argument, then sets out what the durable half of the portfolio looks like.

What to take away

The clock argument

Consider the loop you are actually in. Something changes. You observe an effect, with a lag, through a partial view. You form a theory, with more lag. You act, which takes weeks. You measure, which takes weeks more. By the time you have a confident answer, several months have passed inside a system whose maintainers ship continuously and can measure everything directly.

You are not competing with the system. You are competing with the clock, and the clock is not close. This is not a criticism of anyone's analysis: a perfectly executed investigation still finishes after the thing it investigated has moved.

The failure mode has a name in modeling. A tactic tuned tightly to the current configuration is overfitting to a particular period, and it performs beautifully on the data you learned it from and poorly afterwards. The tighter the fit, the shorter the useful life. The configuration being fitted is the set of inputs described in the overview of ranking signals.

There is a second cost that never appears in the accounting. Optimizing against a measured target degrades the target, which is the practical content of Campbell's law. Whatever you shaped your work around stops being a good indicator once enough people shape their work around it, and the platform then has to change it, which is where a good share of updates come from in the first place.

Why nobody can give you the full explanation

It is worth being precise about this, because the assumption that platforms are hiding a simple answer drives a lot of wasted effort.

The output of a ranking system is a composition of many components: retrieval, several scoring models, policy rules, quality filters and a diversification pass, each maintained by different people.

Behavior at the level of a single item is emergent from the interaction, and the engineer who can explain one component cannot always predict what the composition does to your page. How far that composition moves from month to month is the subject of the research on how ranking drifts.

The models themselves are fitted to enormous amounts of behavioral data that nobody has inspected item by item. Their internal structure supports statements about aggregate performance and not about why one item placed where it did. And every part of it changes on a schedule faster than documentation review.

On top of that sits a genuine incentive not to be precise: publishing an exact objective invites optimization against it, which corrupts the data, which forces a change.

So the guidance is general because it must be, and it is still the best statement of intent available. It belongs at the top of your source list even though it will never answer the question you want answered.

What survives a change

The last row is the only real overlap between chasing and building. Every ranking system in this class is attempting to predict something like satisfaction, badly and through proxies.

Improving actual satisfaction is therefore aligned with the current configuration, the previous one and the next one, which no tactic can claim. That is the reading no trend report can hand you, which is why the overview of annual trend reports treats a report as a source of questions rather than of instructions. This is not a moral point. It is the only investment whose value is independent of a variable you cannot observe.

When chasing is the right call

Not never. Short-lived arbitrage is real, and taking it is rational under three conditions: the cost of switching is near zero, you can act within days rather than months, and you have written down that it is temporary so that you do not build a department around it.

What makes chasing destructive is not the chase. It is the accumulation. Each tactic leaves behind a template, a process and someone whose job depends on it, and none of that gets removed when the tactic stops working. Two years of that produces an organization optimized for a configuration that no longer exists.

A rough allocation of most effort to the durable column and a small, capped, deliberately disposable share to the chase is the version that has survived contact with reality. How that plays out over a longer horizon is set out in the record of how past shifts unfolded.

Common questions

Is this an argument against reading trend reports?

No. It is an argument against acting on them by default. Read them for the questions they raise and the vocabulary they introduce, and hold every number in them to a stated standard before you quote it.

How do I tell a durable investment from a tactic dressed as one?

Ask what happens to its value if the system changes tomorrow. If the honest answer is that you would not know for six months, it is a tactic.

Everyone in my field is chasing and it appears to work for them.

You are seeing the survivors, and you are seeing them during the window. Ask the same question in two years about the same names. Why the illusion holds up so well is taken apart in the comparison of two editions.

If I cannot know how ranking works, what am I supposed to do?

Build something people want, make it easy to find by more than one route, measure your own property well enough to notice when something breaks, and keep enough of a direct relationship that a bad quarter is survivable.

That is the whole answer, it has been the whole answer for a long time, and it is unsatisfying precisely because it does not depend on this year's report.