Ranking signals comparison: evidence types

Most published claims about ranking factors come from one kind of study: take a few thousand items that currently rank, measure some attributes, and report which attributes track position. The results get written up as advice.

A correlation study cannot support causal advice on its own, and the gap between what it measures and what people claim from it is a major source of confident nonsense in this field. This page compares evidence about social-feed ranking signals and what each kind can and cannot establish.

What to take away

What a social-feed ranking factor study actually measures

The procedure is roughly: collect posts shown in ranked feed positions, measure attributes the researcher can observe, and compute a correlation between each attribute and placement. Everything that follows depends on that sample.

The sample contains posts that surfaced in a feed. It says little about eligible posts that were not shown, so you are looking at surfaced posts and reasoning about the wider pool. Whether the measured attributes are social-feed ranking signals, proxies, or outcomes is a separate question.

Attributes arrive in clusters, so the study cannot separate them. A post with long watch time may also have strong creative, an established creator audience, a timely topic, and a trusted account.

Correlate any one with position and you will find something, because you are measuring the cluster. That is a textbook confounder, and sample size does not fix it.

Direction is the third problem. Several of the attributes people measure may be consequences of placement rather than causes of it. A visible post can collect likes, shares, and comments because it was visible. The correlation can be real while the arrow points the other way, which is the ordinary meaning of the warning that correlation does not imply causation.

Finally, the number is an average over different topics and audiences. A relationship that is positive in one segment and negative in another can average to nothing, and a relationship that holds only in one segment can average to something. A single figure across a whole corpus hides both.

Comparing what each kind of evidence can carry

The order matters more than the rows: evidence near the bottom costs more and answers a narrower question. That is why it is worth more.

Two named examples from the bottom rows

A natural experiment needs a dated change. YouTube supplied one in January 2019, when it announced a "gradual change" of "reducing recommendations of borderline content and content that could misinform users in harmful ways." A channel that tracked its recommended traffic around that date had a before-and-after with a known cause, which is as good as natural experiments get, and still could not separate that change from everything else that moved in the same weeks.

A controlled test at platform scale looks like the field experiment run on X in 2023 and published in Nature in 2026. The researchers assigned active US-based users randomly to either an algorithmic or a chronological feed for 7 weeks, and found that switching from a chronological to an algorithmic feed increased engagement. The same paper notes that earlier evidence from switching off the algorithm on Meta platforms found no political effects. Random assignment is what lets those studies say what the feed did, and it is exactly what a correlation study across ranked posts lacks.

A study covering ten thousand posts tells you about the average across those posts, none of which is yours. Which part of a recommendation pipeline a result could touch is a separate question: post selection, ranking, and distribution are distinct stages, and a study of surfaced posts alone may not distinguish them.

How a correlation study earns a place

Its one honest use: generating candidates. If an attribute tracks position across a large sample, that is a reason to put it on a list of things to test, and no more than that.

Treat the output as a queue of hypotheses, and the study becomes useful. Treat it as a set of instructions, and you will spend a quarter on the strongest confounder in the data.

There is a second honest use, which is describing a moment. A well-run study is a snapshot of what visible feeds look like now. That is genuinely informative about the current shape of a feed, and it needs no causal claim at all.

Reading one that way is closer to reading a census than reading an experiment. The record of how these claims have changed is a useful check on how quickly the snapshots go stale.

The comparison you should actually run

Against broad studies, set the one thing you control: a deliberate change to a set of posts on your own account, with a comparable set left unchanged. It answers a narrow question about that account, though differences in audience and platform behavior can still complicate the comparison.

Knowing which stage a change could plausibly touch is worth an hour before you spend a month on one that could not have mattered.

Two habits make the comparison honest. Write your expectation before you look, so you cannot fit the story to the result. Decide in advance what result would make you abandon the idea.

A test you would explain away either way is not a test. Both habits come from formal practice; how recommendation research is designed shows them at a larger scale.

Common questions

Are ranking factor studies dishonest?

Usually not. Most are careful about their method in the methodology section and get misread in the summary. The failure is in the reading far more often than in the arithmetic.

Why do two studies disagree about the same attribute?

Different samples, different measurement tools, different corrections, different periods. Any of those can flip a weak correlation. Persistent disagreement across studies is a signal that the underlying effect is small or conditional, not that one team is wrong.

Can I compare my account against another creator's account as a control?

Not really. A control has to be similar in every respect except the change, and another creator's account differs in many respects at once. The nearest usable version is comparing two sets of posts from your own account. The failure modes worth knowing before you try are worth reading first.

What about studies run by the platform itself?

They have access nobody else does and a strong interest in the conclusion. Read them for the mechanism they describe rather than the effect size they report, and note what the comparison group was.