Skip to main content
Technology & EngineeringDeep Learning95 lines

PyTorch Training and Validation Loop

Build and debug a PyTorch model training loop with separate validation data, correct train and eval modes, finite loss checks and a verified checkpoint round trip. Use for torch tensors, DataLoader batches, autograd, optimizer steps and reproducible small-model experiments.

Quick Summary13 lines
Establish a correct small experiment before scaling training. Deliver runnable
training code, a data/model contract, validation metrics and a checkpoint that
can reproduce inference. Do not mistake successful execution for a useful model.

## Key Points

- A small synthetic classification task changes parameters and produces finite
- A wrong target shape or non-finite input fails before a misleading successful
- [PyTorch optimization loop](https://docs.pytorch.org/tutorials/beginner/basics/optimization_tutorial.html)
- [PyTorch save and load](https://docs.pytorch.org/tutorials/beginner/basics/saveloadrun_tutorial.html)
- [PyTorch quickstart](https://docs.pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html)
skilldb get deep-learning-skills/pytorch-training-validation-loopFull skill: 95 lines
Paste into your CLAUDE.md or agent config

PyTorch Training and Validation Loop

Establish a correct small experiment before scaling training. Deliver runnable training code, a data/model contract, validation metrics and a checkpoint that can reproduce inference. Do not mistake successful execution for a useful model.

Specify the experiment

Record the installed PyTorch version, Python environment, device and selected backend. Check available hardware rather than assuming CUDA. Define input shape, dtype, target meaning, batch dimension, model output and loss function together. For classification, verify whether targets are class indices or distributions and whether the loss expects logits; do not add a softmax by habit.

Split the dataset before fitting preprocessing. Keep related subjects, duplicate examples or future observations out of validation when the task requires group or time separation. Record the split policy and dataset version. Fit transforms on training data only, then reuse that state for validation. Use a small local synthetic fixture for the initial mechanics check to avoid downloading private data or starting an expensive training job.

Make one batch work

Inspect a batch from the DataLoader and assert dimensions, dtype, label range and finite numeric inputs. Instantiate the model on the selected device and move inputs and targets consistently. Run one forward pass and inspect the output before enabling a long loop.

For each training batch: set training mode, clear accumulated gradients, compute predictions and loss, reject non-finite loss, backpropagate, and take the optimizer step. Gradient accumulation is an explicit configuration; scale loss and step frequency consistently when using it. Avoid retaining computation graphs in metric lists: accumulate detached scalar values instead.

Use a tiny deterministic batch to check that parameters change and loss can decrease. Failure to overfit this small fixture is a reason to inspect labels, loss, gradients and optimizer wiring before tuning architecture. It is not a claim about generalization.

Validate without learning from validation

Switch to evaluation mode and disable gradient recording for inference. These are separate controls: evaluation mode changes modules such as dropout, while the gradient context controls autograd work. Never call the optimizer during validation. Aggregate metrics by sample count, including a smaller final batch. Report the denominator and compare against a meaningful baseline.

Check class imbalance, leakage and train/validation divergence. Do not select hyperparameters using the final held-out test set. Seeds help reproduce an experiment, but identical results across devices, releases and nondeterministic operators are not guaranteed. Record the relevant reproducibility settings and measured variance rather than promising universal bitwise equality.

Save and prove recovery

Save a state dictionary with architecture/configuration, preprocessing identity, dataset split and measured metrics. For resumable training, also preserve optimizer/scheduler state, training position and relevant random state. Write to a new checkpoint file, verify it and then update any latest-pointer metadata. Do not overwrite the only known-good checkpoint before verification.

Load the checkpoint into the matching architecture with the intended device mapping and an appropriate restricted loading mode. Never load untrusted pickle objects or disable loading protections merely to silence an error. Compare evaluation outputs on a fixed batch before and after loading within an explicit tolerance. A resumed optimizer step is a separate test from inference recovery.

Return environment versions, data/split contract, train and validation curves, the saved artifact and round-trip result. Distinguish CPU fixture success from GPU execution, distributed training, accuracy claims and production deployment.

Review scenarios

  • A small synthetic classification task changes parameters and produces finite metrics; reloaded evaluation outputs match the saved model within tolerance.
  • A wrong target shape or non-finite input fails before a misleading successful training report, and validation never changes parameter values.

References

Original SkillDB-authored procedure. References checked 2026-11-13; no upstream skill body is copied. Rights: LicenseRef-SkillDB-Content.

Install this skill directly: skilldb add deep-learning-skills

Get CLI access →

Related Skills

Recommender Systems

Triggers when users need help with recommendation systems, collaborative filtering, or ranking models. Activate for questions about matrix factorization, ALS, content-based filtering, deep recommender models (NCF, Wide&Deep, DeepFM, two-tower), sequential recommendation, cold start problem, implicit vs explicit feedback, multi-objective ranking, exploration vs exploitation, and real-time recommendation serving.

Deep Learning•168L

Recurrent Architectures

Triggers when users need help with recurrent neural networks, sequence modeling with LSTMs or GRUs, or modern state-space models. Activate for questions about vanishing gradients, sequence-to-sequence models, attention mechanisms in RNNs (Bahdanau, Luong), bidirectional RNNs, Mamba, S4, and when RNNs still outperform transformers for sequential data.

Deep Learning•146L

Regularization Generalization

Triggers when users need help with preventing overfitting, improving model generalization, or applying regularization techniques. Activate for questions about dropout, weight decay, data augmentation (CutMix, MixUp, RandAugment, AugMax), label smoothing, early stopping, knowledge distillation, ensemble methods, bias-variance tradeoff in deep learning, and double descent phenomenon.

Deep Learning•160L

Self Supervised Learning

Triggers when users need help with self-supervised learning, representation learning without labels, or pretext task design. Activate for questions about contrastive learning (SimCLR, MoCo, BYOL), masked modeling (MAE, BEiT, data2vec), pretext tasks, representation evaluation (linear probing, fine-tuning), self-supervised methods for vision vs NLP vs audio, DINO and DINOv2, and curriculum learning.

Deep Learning•168L

Speech Audio ML

Triggers when users need help with speech processing, audio machine learning, or sound generation. Activate for questions about ASR architectures (CTC, attention-based, Whisper), text-to-speech (Tacotron, VITS, neural codec models), speaker verification, speaker diarization, audio classification, music generation, speech enhancement, speech separation, mel spectrograms, and audio tokenization (SoundStream, EnCodec).

Deep Learning•144L

Training Optimization

Triggers when users need help with deep learning training procedures, optimizer selection, or training efficiency. Activate for questions about SGD, Adam, AdamW, LAMB, Lion, learning rate schedules, gradient clipping, mixed precision training, FP16, BF16, gradient accumulation, weight initialization, loss landscape analysis, and hyperparameter tuning including Bayesian optimization and population-based training.

Deep Learning•170L