Learn · Try · Check
What you’ll learn
Diagnose a gap between fitting examples and handling new ones.
Listen to this lesson
Local AI narration · Quiz answers are kept out of the audio.
Read the audio transcript
ML 101. Generalization and overfitting. Diagnose a gap between fitting examples and handling new ones. Learning the training set is not the whole goal. Generalization means useful performance on new examples from the intended setting. Overfitting occurs when a model fits details of training data that do not transfer well. High training performance alongside much weaker independent validation performance is a warning sign. It is not a complete diagnosis: different data conditions, label problems or leakage in the evaluation process also need investigation. Compare with a baseline. A baseline is a simple reference approach under the same evaluation conditions. It can be a fixed rule, a majority-class predictor or a small model. If a more complex model cannot improve the outcome that matters, complexity has not earned its place. Options for improving generalization include better data, simpler models, regularization and stopping training based on validation behavior. Apply them as controlled experiments, not as a checklist that guarantees success. Look at where the errors happen. An average can hide weak performance on a particular device, lighting condition or capture site. Inspect meaningful groups as well as the overall metric. A model evaluated on bright images may perform poorly on low-light images encountered later. That change in input conditions is one form of distribution shift. Keep monitoring after release because the deployment environment can change even when the model file stays the same. HerbDev example. HerbDev’s guide compares model choices around workflow, memory, latency and evidence. A larger model is not automatically the best deployment decision. Compare it with a smaller baseline using the same task and evaluation procedure. The process: Fit on training data. Then Compare independent results. Then Inspect groups and mistakes. Then Change one factor. Try this. A model scores 99% on training examples and 65% on validation examples. Name two investigations before buying a larger model. Pause the audio to try the exercise. A worked answer and checkpoint questions are available on the lesson page.
Learning the training set is not the whole goal
Generalization means useful performance on new examples from the intended setting. Overfitting occurs when a model fits details of training data that do not transfer well. High training performance alongside much weaker independent validation performance is a warning sign. It is not a complete diagnosis: different data conditions, label problems or leakage in the evaluation process also need investigation.
Compare with a baseline
A baseline is a simple reference approach under the same evaluation conditions. It can be a fixed rule, a majority-class predictor or a small model. If a more complex model cannot improve the outcome that matters, complexity has not earned its place. Options for improving generalization include better data, simpler models, regularization and stopping training based on validation behavior. Apply them as controlled experiments, not as a checklist that guarantees success.
Look at where the errors happen
An average can hide weak performance on a particular device, lighting condition or capture site. Inspect meaningful groups as well as the overall metric. A model evaluated on bright images may perform poorly on low-light images encountered later. That change in input conditions is one form of distribution shift. Keep monitoring after release because the deployment environment can change even when the model file stays the same.
- Fit on training data
- Compare independent results
- Inspect groups and mistakes
- Change one factor
Try it yourself
A model scores 99% on training examples and 65% on validation examples. Name two investigations before buying a larger model.
Compare with a worked answer
Check for overfitting and inspect whether validation labels or capture conditions differ from training. Compare a simpler baseline on the same split. More model capacity is not an automatic fix for the observed gap.
Check your understanding
Choose one answer for each question, then check your score. Read the explanations and retry any question you missed.
Read the answer guide
1. Much stronger training than independent validation performance A large training–validation gap can indicate overfitting, though the data and evaluation process also need checking.
2. It provides a reference under comparable conditions A baseline tells you whether additional complexity improves the chosen outcome under the same evaluation conditions.
3. Performance under the changed input conditions Deployment inputs may differ from development inputs. Evaluate that shift rather than assuming the old score still applies.
Was this explanation clear?
Help improve this lesson. With cookies accepted, HerbDev records one current rating per lesson for this browser. Server checks limit repeated submissions; accepted changes may also appear in site analytics. No written answer or account is needed.
You can also share a suggestion with HerbDev.
Review missed questions
Revisit questions missed on your latest checkpoint attempts. These short reviews help you practise; they do not change your best checkpoint scores. Return to the lesson checkpoint when you are ready.
Check a lesson’s answers to build your review list. Saved reviews are available after cookie acceptance.
Course complete. Keep the questions coming.
You have answered every checkpoint correctly. Use your worked exercises to explain the ideas to someone else, then apply them to a small project.
Apply it in the final project · Choose what to learn next →Keep exploring: What next? · AI vocabulary · AI 101 · Learning hub