Learn · Try · Check
What you’ll learn
Keep evaluation examples independent of model development.
Listen to this lesson
Local AI narration · Quiz answers are kept out of the audio.
Read the audio transcript
ML 101. Splits, leakage and fair testing. Keep evaluation examples independent of model development. Give each split a job. Training data is used to fit model parameters. Validation data helps choose model settings, thresholds or approaches. A test set is held back for a final assessment of the selected system. If you repeatedly change the system based on test results, that test set is influencing development and no longer serves as an untouched final check. There is no universal split percentage that makes every dataset reliable. Split the real units of independence. Suppose a capture session contains many nearly identical frames. A random split of individual frames may put one frame in training and its near-duplicate in testing. The model can appear to generalize while benefiting from familiarity with the same scene. Depending on the intended use, split by session, site, person, device or time. The goal is to make the evaluation resemble the genuinely new examples the system will face. Keep future information out. Data leakage occurs when information unavailable at prediction time influences training or evaluation. Examples include using an outcome recorded after the decision as an input, fitting a preprocessing step on the entire dataset, or tuning against the final test set. Fit transformations such as feature scaling on the training split, then apply those learned transformations to validation and test inputs. Record the split so another person can reproduce it. HerbDev example. The overview emphasizes controlled experiments and validation evidence. Use that approach before comparing architectures: define the dataset and split first, then compare methods under the same conditions. The process: Group related captures. Then Create train / validation / test splits. Then Fit using training data. Then Reserve the test set. Try this. You have ten near-identical frames from each of twenty capture sessions. Should you randomly distribute all 200 frames across the splits? Pause the audio to try the exercise. A worked answer and checkpoint questions are available on the lesson page.
Give each split a job
Training data is used to fit model parameters. Validation data helps choose model settings, thresholds or approaches. A test set is held back for a final assessment of the selected system. If you repeatedly change the system based on test results, that test set is influencing development and no longer serves as an untouched final check. There is no universal split percentage that makes every dataset reliable.
Split the real units of independence
Suppose a capture session contains many nearly identical frames. A random split of individual frames may put one frame in training and its near-duplicate in testing. The model can appear to generalize while benefiting from familiarity with the same scene. Depending on the intended use, split by session, site, person, device or time. The goal is to make the evaluation resemble the genuinely new examples the system will face.
Keep future information out
Data leakage occurs when information unavailable at prediction time influences training or evaluation. Examples include using an outcome recorded after the decision as an input, fitting a preprocessing step on the entire dataset, or tuning against the final test set. Fit transformations such as feature scaling on the training split, then apply those learned transformations to validation and test inputs. Record the split so another person can reproduce it.
- Group related captures
- Create train / validation / test splits
- Fit using training data
- Reserve the test set
Try it yourself
You have ten near-identical frames from each of twenty capture sessions. Should you randomly distribute all 200 frames across the splits?
Compare with a worked answer
If you want to measure performance on new sessions, keep each session’s frames together. Randomly splitting the frames would let near-duplicates cross the boundary and make the test easier than the intended task.
Check your understanding
Choose one answer for each question, then check your score. Read the explanations and retry any question you missed.
Read the answer guide
1. The held-out test set A held-out test set assesses the selected system without being repeatedly used to make development choices.
2. Near-duplicates can appear in both training and testing Near-duplicates reduce the independence of the evaluation and can inflate apparent performance.
3. On training data only Fit preprocessing on training data, then apply the same transformation to validation and test data.
Was this explanation clear?
Help improve this lesson. With cookies accepted, HerbDev records one current rating per lesson for this browser. Server checks limit repeated submissions; accepted changes may also appear in site analytics. No written answer or account is needed.
You can also share a suggestion with HerbDev.
Review missed questions
Revisit questions missed on your latest checkpoint attempts. These short reviews help you practise; they do not change your best checkpoint scores. Return to the lesson checkpoint when you are ready.
Check a lesson’s answers to build your review list. Saved reviews are available after cookie acceptance.
Course complete. Keep the questions coming.
You have answered every checkpoint correctly. Use your worked exercises to explain the ideas to someone else, then apply them to a small project.
Apply it in the final project · Choose what to learn next →Keep exploring: What next? · AI vocabulary · AI 101 · Learning hub