ML Model Selection

Overview

Selecting the right machine learning model is the most consequential early decision in any ML project. A poor choice leads to wasted compute, missed accuracy targets, and delayed timelines. This skill provides a systematic framework for matching problem characteristics to algorithm families, then narrowing to specific models based on data volume, feature types, latency requirements, and interpretability needs.

Use this skill when starting a new ML project, when an existing model underperforms and you suspect a fundamentally different approach is needed, or when stakeholders ask why a particular algorithm was chosen.

Core Framework

Problem Type Classification

1. Supervised Learning - Labeled data available; predict a target variable.
   • Classification (binary, multiclass, multilabel)
   • Regression (continuous, count, ordinal)
2. Unsupervised Learning - No labels; discover structure.
   • Clustering, dimensionality reduction, anomaly detection
3. Reinforcement Learning - Sequential decision-making with rewards.
4. Self-supervised / Semi-supervised - Limited labels augmented with unlabeled data.

Decision Criteria Matrix

Criterion	Low	Medium	High
Data volume	Rule-based / linear	Tree ensembles	Deep learning
Feature interpretability need	Neural nets OK	SHAP-compatible	Linear / GAM
Latency budget	Batch OK	Sub-second	Sub-10ms
Dimensionality	Simple models	Regularized	Embeddings / PCA first

Process

1. Define the business objective and map it to a problem type (classification, regression, clustering, etc.).
2. Profile the dataset: volume, feature count, feature types (numeric, categorical, text, image), label balance, missingness rate.
3. Identify hard constraints: latency, memory, interpretability mandates, regulatory requirements.
4. Select 2-3 candidate algorithm families using the decision criteria matrix.
5. Establish a baseline with the simplest viable model (logistic regression, k-NN, or decision tree).
6. Train candidates with default hyperparameters on a consistent train/validation split.
7. Compare candidates on the primary metric plus secondary metrics (fairness, calibration, inference speed).
8. Select the best candidate and proceed to hyperparameter tuning.
9. Document the selection rationale including rejected alternatives and why.

Key Principles

• Always start with a simple baseline before reaching for complex models.
• More data often beats a better algorithm; verify data quality before model complexity.
• Match the model to the data type natively: tabular data favors tree ensembles; images favor CNNs; sequences favor transformers or RNNs.
• Gradient-boosted trees (XGBoost, LightGBM) remain the default winner for structured/tabular data.
• Deep learning requires at minimum tens of thousands of samples to outperform classical methods on tabular data.
• Interpretability is not optional in regulated domains (finance, healthcare); plan for it from the start.
• Ensemble methods can combine strengths but add deployment complexity.

Common Pitfalls

• Jumping straight to deep learning without establishing a simpler baseline.
• Ignoring inference-time constraints until deployment and discovering the model is too slow.
• Selecting a model based on benchmark performance on different data distributions.
• Confusing model flexibility with model quality; overfitting is the usual result.
• Neglecting categorical feature handling; many models need explicit encoding strategies.
• Choosing a model because it is trendy rather than because it fits the problem constraints.

Output Format

When recommending a model, deliver:

1. Problem Statement: One sentence mapping business goal to ML task type.
2. Data Profile Summary: Key stats (rows, features, types, label distribution).
3. Constraints: Latency, interpretability, compute budget.
4. Recommended Model: Algorithm name with justification.
5. Runner-up: Alternative model and the scenario where it would be preferred.
6. Baseline Plan: Simplest model to implement first for comparison.
7. Risks: Known failure modes of the selected approach.