Training a machine learning model rarely fails because the algorithm is wrong. It fails because of a handful of recurring, unglamorous problems — bad data, a model that's too simple or too complex, gradients that won't behave, or a leak that makes your metrics lie to you.
This guide walks the issues that bite ML developers most often, and — more importantly — the fix for each. Knowing the name of the problem you're hitting is half the battle.
Concepts and fixes verified as of June 2026.
The most common ML training problems are data quality, overfitting and underfitting, wrong model complexity, vanishing/exploding gradients, and data leakage — most of which trace back to your data and the balance between bias and variance.
- Data quality beats model choice — garbage in, garbage out.
- Overfitting = great on train, bad on test; underfitting = bad on both.
- Vanishing/exploding gradients break deep nets — fix with ReLU, normalization, residuals, clipping.
- Data leakage makes metrics lie — fit transforms on the training split only.
- Most problems are a bias-variance or a data problem in disguise.
🧹 Data Quality Issues
Every model is downstream of its data. Missing values, mislabeled examples, inconsistent formats, and class imbalance all cap how good a model can ever get — no architecture rescues bad data.
The fix: profile the dataset before modeling. Handle missing values deliberately (impute or drop), correct or remove bad labels, standardize formats, and address class imbalance with resampling or class weights. Time spent cleaning data has the highest return of anything in the pipeline.
🎯 Feature Selection & Engineering
More features is not better. Irrelevant or redundant features add noise, slow training, and invite overfitting. The goal is the smallest set of features that carries the signal for this problem.
Common selection methods:
- Correlation coefficient — drop features highly correlated with each other (redundant).
- Fisher's test / ANOVA — rank features by how well they separate classes.
- Information gain — keep features that most reduce uncertainty about the target.
The fix: apply feature engineering to construct informative features, then select down to the ones that matter. Fewer, stronger features usually beat a kitchen-sink model.
⚖️ Overfitting and Underfitting
The two most famous failure modes, and they're opposites.
| Overfitting | Underfitting | |
|---|---|---|
| Train performance | High | Low |
| Test performance | Low | Low |
| Cause | Model memorized noise (high variance) | Model too simple (high bias) |
| Fix | More data, regularization, dropout, early stopping | More complex model, better features, train longer |
Both are frequently caused by insufficient data. The art is finding the balance — enough capacity to learn the pattern, not so much that it memorizes the training set.
🏗️ Model Complexity
Picking the architecture is a balancing act. Too simple and you underfit; too complex and you overfit (and pay in compute). The fix: start simple, establish a baseline, and add complexity only when the data justifies it — validated against a held-out set, not the training score.
📉 Exploding and Vanishing Gradients
This was a major barrier to training large networks. In deep networks with many layers — deep neural networks (DNNs) and recurrent neural networks (RNNs) especially — gradients multiplied across layers either shrink toward zero (vanishing, so early layers stop learning) or blow up (exploding, so training diverges).
The fix: ReLU-family activations, careful weight initialization (He/Xavier), batch or layer normalization, residual/skip connections, and gradient clipping for the exploding case. These are why modern deep nets train at all.
🔁 Transfer Learning Challenges
Reusing a pre-trained model is powerful but not free. The two snags: choosing the right pre-trained model for your use case, and fine-tuning it without destroying the knowledge it already has.
The fix: match the pre-trained model's domain to yours, freeze early layers and fine-tune later ones, and use a low learning rate so you adapt rather than overwrite.
🚰 Data Leakage
The most insidious problem because it makes your metrics look great while the model is secretly broken. Leakage is the unintentional inclusion of information from the test set (or the future) in the training process.
The fix: do the train-test split first, then fit every transform (scaling, encoding, imputation) on the training split only and apply it to the test split. Never use a feature that won't exist at prediction time. If results look too good to be true, suspect leakage.
🚀 Deployment Challenges
A model that trains beautifully can still be impractical to ship. Large models need significant memory and compute, and serving them at low latency in production is its own problem.
The fix: compress before you deploy — quantization, pruning, or distillation to shrink the model — and pick a serving setup that fits your latency and cost budget. The smallest model that hits your accuracy bar is the right one.
Frequently Asked Questions
What are the most common problems when training an ML model?
The recurring ones are: poor data quality, weak feature selection, overfitting and underfitting, wrong model complexity, exploding/vanishing gradients in deep networks, transfer-learning mismatches, data leakage, and deployment/serving constraints. Most trace back to data quality and the bias-variance balance.
What is the difference between overfitting and underfitting?
Overfitting means the model performs well on training data but poorly on test data — it memorized noise. Underfitting means it performs poorly even on training data — it's too simple to capture the pattern. Overfitting is high variance; underfitting is high bias. Both are often caused by too little data or the wrong model complexity.
What causes exploding and vanishing gradients?
They happen in deep networks (DNNs, RNNs) when gradients are repeatedly multiplied through many layers — shrinking toward zero (vanishing) so early layers stop learning, or blowing up (exploding) so training diverges. Fixes include ReLU-family activations, careful weight initialization, batch/layer normalization, residual connections, and gradient clipping.
What is data leakage in machine learning?
Data leakage is when information from outside the training set — often from the test set or the future — sneaks into training, so the model looks great in evaluation but fails in production. Common causes are scaling/encoding before the train-test split, or using features that won't exist at prediction time. Fit all transforms on the training split only.
How do you fix overfitting?
Get more or better data, simplify the model, add regularization (L1/L2, dropout), use early stopping, and validate with cross-validation. The goal is to lower variance without pushing the model into underfitting — it's a balance, not a switch.
🚀 What's Next
- 🧹 Audit your data first — most "model problems" are data problems. Profile before you train.
- 📏 Name the failure — decide whether you're fighting bias (underfitting) or variance (overfitting); the fixes are opposite.
- 🔍 Hunt for leakage — if your validation scores look too good, check that no test-set information reached training.
- 🧠 Go deeper on modern training — see fine-tuning an open-source LLM on your codebase for transfer learning in practice.
Solid training is the foundation; serving it privately is the next step. See the Local LLM + Ollama RAG guide for running models on your own hardware.