Rules
Part of A ranking signal is not the same thing as a ranking factor
How to run ranking signals research on your social posts
Ranking signals research on your own social account: choosing a signal to test, building a split that cannot flatter you, and reading the result honestly.
You cannot see inside the ranking system behind an Instagram, TikTok or YouTube feed, but you can run an experiment against it. The feed is a black box that takes your posts as input and returns distribution: impressions, reach, watch time. Change one input on part of your posts, leave a comparable part alone, and compare the two.
On a social feed the design is always the same: one changed input on one set of your posts, one comparable set left alone, and a record of how much reach moves on its own from week to week.
That is the only ranking signals research an outsider can run that produces evidence about your own account rather than an average of other people's accounts. It is slower and narrower than reading a study, and for your own decisions it is worth more.
What to take away
- The unit you randomise is the thing you changed, usually a post or a format, not a viewer.
- A control group is not optional. Before and after on the treated posts alone measures the season, the news, and everything else that happened in the feed that month.
- Decide the metric, the duration and the decision rule before you look at any data. Everything after that is arithmetic.
Set the question narrowly
A test answers one question about one account over one period. "Does adding on-screen captions to Reels change average watch time on those Reels" is testable, but "Does quality matter" is not.
Testable vs Untestable Questions
Testable
- Question
- One property, one period
- Change
- One change
- Settles
- A number decides
- Attribution
- Which part earned it
Untestable
- Question
- Quality matters
- Change
- Three edits bundled
- Settles
- No number settles
- Attribution
- Bundle helped, unclear
If a number would not settle the question, you are not ready to run anything. Keep the input list short and concrete, the signals a feed can actually observe: watch time, completion, shares, saves, comments. The ranking signals overview sets out what counts as one.
Where to find candidate signals
Start from what the platforms have said their feeds respond to, because those are the inputs worth a test. TikTok's "For You" page is a feed of videos recommended to users based on their activity on the app, with content curated depending on what a user liked, interacted with, or searched. When Instagram moved its feed from strictly chronological order to an algorithm in April 2016, it said the algorithm was designed so that users would see more of the photos by users that they liked. Both point at the same family of signals: what viewers do with a post. A change that should alter what viewers do, such as a stronger opening or a clearer reason to save, is a testable input; a change that viewers cannot see is a weak one.
Narrow means one change. Ship three edits together, say a new hook, a new caption style and a new posting time, and a positive result says the bundle helped. That is fine if you keep the bundle forever. It is useless if you need to know which part earned its place.
Bundles are reasonable when the changes are cheap and attribution does not matter. Be honest about which situation you are in.
Build the split so it cannot flatter you
The most common mistake is splitting on something related to the outcome. New posts against old posts, or one content series against another, guarantees a difference that has nothing to do with your change.
Building a Non-Flattering Split
- Avoid splitting on outcome-related traits
- Assign by arbitrary stable hash
- Check groups tracked before change
- Start clock at recrawl, not deploy
- Account for external position shifts
- Extend duration for slow response
Assign posts to treated and control groups by something arbitrary and decided in advance, like alternating by posting order within the same series. Then check that comparable posts of both kinds looked alike for several weeks before you changed anything.
If they did not track each other before, they will not track each other after. The general design is ordinary split testing with a slow and noisy response variable.
Two constraints make this harder than a usual on-site experiment. First, a post's distribution unfolds over its first days in the feed, so each post needs the same amount of time before you read it.
Second, the response is reach inside someone else's system, which moves on its own the whole time. Distribution belongs to machinery you do not control, and how a feed gets assembled describes the stages behind it. Both constraints push the required duration up.
Some platforms offer a narrow built-in version of this, such as YouTube Studio's option to test thumbnails against each other. Use it where it exists, and keep it for the one question it answers.
Decide these before you start
| Decision | Set it to | Why it has to be beforehand |
|---|---|---|
| Primary metric | One number, chosen now | Five metrics give five chances to find a win by accident |
| Unit of analysis | The thing you changed | Counting sessions when you changed pages overstates certainty |
| Minimum effect worth acting on | A number you would act on | Without it, any result becomes interesting after the fact |
| Duration | Long enough to cover a full weekly cycle and recrawl | Stopping when the line looks good is how noise becomes a finding |
| Stopping rule | Written down, including the null case | A test you would keep running until it wins is not a test |
| What would make you abandon the idea | Something specific | If nothing would, you are collecting support, not evidence |
On a social account, read "pages" in that table as posts and "recrawl" as the days each post needs to settle in the feed. The row that gets skipped most often is the minimum effect. It is also the one that decides whether the test is possible at all. If the change you expect is a few percent and your weekly variation in reach is larger than that, no length of run will separate them.
Pre-Test Decisions Checklist
- Primary metricone number now
- Unit of analysisthe changed thing
- Minimum effect worth acting on
- Durationfull weekly cycle and recrawl
- Stopping rulewritten, includes null
- Abandon criteriasomething specific
Work out the statistical power you have before spending a quarter on a question your posting volume cannot answer. Being told on day one that you cannot detect it beats an ambiguous chart in month three.
Read the result honestly
Compare the change in the treated posts against the change in the control posts over the same window. Not treated before against treated after. That single discipline removes most of the ways a seasonal swing or a platform-wide feed change gets reported as a win.
Reading Results Honestly
Do
- Comparison
- Treated vs control change
- Null result
- Effect not detectable
- Positive result
- Site, period, system version
- Record
- Size you could detect
Don't
- Comparison
- Treated before vs after
- Null result
- Effect is zero
- Positive result
- A universal law
- Record
- Only the win
A null result means the effect was too small to detect with this much data. It does not mean the effect is zero or the idea was stupid.
Record it anyway with the size you could have detected. The next person asking deserves to know what was ruled out and at what sensitivity.
A positive result depends on your account, your period, and the version of the feed you were tested against. It is not a law. Publishing it as one gives the field confident advice that later stops working, and no one can say when.
The wider argument about which kinds of evidence carry what weight sits in the comparison of evidence types.
Common questions
How long should a test run?
Long enough for each post's distribution to settle, plus enough full weekly cycles to average out the day of week pattern. In practice that is weeks, not days. The exact number falls out of your posting volume and the effect size you set in advance.
Can I test on a small account?
You can run the test. You often cannot detect anything, because the noise on small numbers swamps a modest effect. On a small account, spend the effort on changes that are obviously right for a viewer rather than on measuring them.
What if the platform changes mid-test?
Your control group is exposed to the same change, which is precisely why it is there. A feed-wide shift moves both groups and the difference between them survives. A shift that hits only one group means your split was not arbitrary after all.
Should I test the same thing more than once?
Yes, if the decision matters, a single positive result at a conventional threshold is weaker than people treat it. Repeat on a different series or another quarter, the cheapest way to learn if you measured an effect or a moment.
Worked cases of that idea behaving differently across different contexts appear in examples of recommendation behavior.
