Splitting a labeled dataset into train/val/test sounds like it should be os.listdir() plus random.shuffle() — and mechanically, that's most of it. What that naive version gets wrong is reproducibility, class balance, and a specific kind of leakage that doesn't look like a bug at all: it looks like a suspiciously good validation score.
Why a naive split isn't enough
No fixed seed means no reproducible run. Reshuffle without a seed and you get a different split every time — which means a training run from last week and one from today aren't actually comparable, and a bug report ("model does badly on X") can't be reproduced against the same val set that produced it.
Near-duplicate images end up on both sides of the split. This is the one that doesn't announce itself. Burst-mode photos, consecutive frames pulled from the same video, or the same scene shot twice a second apart are, to a naive shuffle, just unrelated files — nothing about a filename says "this is 98% the same image as that other one three folders over." When one lands in train and its near-twin lands in val, the model isn't really being tested on unseen data for that pair — it's close enough to something it already trained on that the validation score quietly overstates how well the model generalizes. This is a real, common cause of a model that looks great in validation and disappoints in production.
A rare class can end up entirely in one split. With enough classes and a small enough dataset, ordinary random chance can put every example of some class in train and none in val — or the reverse — without anyone doing anything wrong. Either way, that class isn't actually being validated (or trained on) at all, and nothing about a successful-looking training run tells you that happened.
How the split works
The YOLO Dataset Splitter takes your ratios (e.g. 70/20/10) and a seed, then assigns every image deterministically — the same seed always produces the same split, so a run is reproducible and a split can be shared (just share the seed and ratios, not the file list). Counts are computed with the largest-remainder method rather than naive rounding, so a 70/20/10 split of 137 images lands on exactly 137 total, not 136 or 138 from three separate roundings.
Checking for leakage
Turn on the near-duplicate check and the splitter computes a perceptual hash (dHash) for every image and compares every pair across split boundaries — not within a split, only between train↔val, train↔test, and val↔test, since a duplicate within the same split isn't a leakage problem. Two images within a small Hamming distance of each other are flagged as a likely near-duplicate pair straddling the split, with both file paths shown so you can look at them directly. From there it's a judgment call: move one of the pair into the same split as the other, or leave it if the two are similar but genuinely distinct scenes (the check flags visual similarity, not source — it can't know they came from the same 3-second burst versus two coincidentally similar photos).
This check decodes every image to compute its hash, so it's opt-in rather than automatic — worth running once per dataset, not on every experimental re-split.
Checking class balance
After assigning splits, the tool counts how many boxes of each class landed in each split and flags two things: a class that's completely missing from a split (it can't be trained on, or can't be validated, depending on which split), and a class present but in single digits — technically there, but not enough examples to mean anything statistically. Neither is necessarily wrong for your dataset, but both are worth a deliberate decision rather than an accident of the random seed you happened to pick.
Where each step happens
- Checking dataset structure first — the YOLO Dataset Health Checker confirms every image/label pair is intact before you split it; a structural problem is easier to fix before the dataset is divided three ways.
- Splitting — the YOLO Dataset Splitter is the tool this guide is built around — seeded, ratio-based, with the leakage and class-balance checks above.
- Fixing a class imbalance the split surfaced — the YOLO Class Manager merges or removes classes across the whole dataset in one pass, useful if the splitter's class-balance warning points at a class that's simply too rare to be worth keeping on its own.