The method · 3 October 2026 · 2 min read
Why one viral Short proves nothing (and how many videos a test needs)
One Short can’t tell you what works. You need at least three videos on each side of a test, and four is better. Single Shorts swing wildly for reasons you can’t control, so a test with one video per side mostly measures luck.
What happened in our first batch
Our first test batch compared two kinds of topic. Side A was “AI does something strange”. Side B was “a small business made money in an odd way”. We posted two videos of each.
| Video | Side | YouTube views (first day) |
|---|---|---|
| Beer holder | B: odd small business | 393 |
| AI cheating at chess | A: strange AI | 197 |
| Stop Hiring Humans | B: odd small business | 5 |
| Owls and numbers | A: strange AI | 1–3 |
If we’d posted only the beer holder and the owls, we’d have “proved” that small-business stories win by a mile. If we’d posted only the chess video and Stop Hiring Humans, we’d have proved the opposite.
Each side had one video that took off and one that got almost nothing. The kind of topic didn’t explain the result at all.
Why single Shorts swing so much
YouTube gives every new Short a first push to a few hundred people. What happens next depends on how many of them stayed instead of swiping. That first group is small and random, so two very similar videos can get very different starts.
Lots of things also change between any two videos besides the thing you meant to test: the opening picture, the first words, the topic itself. With one video per side, any of those could be the real cause.
How many videos a test needs
- Three per side at minimum, four recommended. That’s enough for a pattern to show through the noise, without making a batch too big to produce every day.
- Compare the middle video, not the average. With four videos per side, one runaway hit can drag the average up on its own. The median (the middle value) ignores it.
- Judge on the share who stayed, not views. Since March 2025, YouTube counts a view the moment a Short starts playing, so raw views flatter everything.
- Wait for the numbers to settle. Give every video 300 views or 72 hours before calling it.
How you’d test this
If you’re unsure whether something works, don’t change it on one video and watch. Make four videos with it and four without, keep everything else the same, and post them alternately: A, B, A, B. Then compare the middle “stayed to watch” figure on each side. If the gap is small, call it no clear winner and test something else.
That’s the loop Faceless Theory runs for you every batch.