Validate ML Workloads on AMD ROCm
Check AMD ROCm hardware and software compatibility, prove PyTorch runs on the expected GPU, and compare correctness and performance with a reproducible fixture.
Inventory the AMD GPU model, GPU target if available, host OS and kernel, driver, ROCm release, PyTorch wheel or image, Python version, and any container device settings. Check the AMD compatibility matrix for that exact ROCm release and the AMD product family. Confirm that the compute stack and framework combination are supported together. A working display driver or visible card alone does not establish an available ROCm compute device. Keep the known-good environment pinned while investigating. ## Key Points - [AMD ROCm compatibility matrix](https://rocm.docs.amd.com/en/latest/compatibility/compatibility-matrix.html) - [PyTorch HIP semantics](https://docs.pytorch.org/docs/stable/notes/hip.html)
skilldb get mlops-infrastructure-skills/rocm-workload-validationFull skill: 30 linesValidate ML Workloads on AMD ROCm
Establish a supported baseline
Inventory the AMD GPU model, GPU target if available, host OS and kernel, driver, ROCm release, PyTorch wheel or image, Python version, and any container device settings. Check the AMD compatibility matrix for that exact ROCm release and the AMD product family. Confirm that the compute stack and framework combination are supported together. A working display driver or visible card alone does not establish an available ROCm compute device. Keep the known-good environment pinned while investigating.
Collect hardware visibility on the host using the supported vendor tools. If the workload runs in a container, collect the same evidence inside it. A device that appears only on the host indicates that container permissions or device exposure still need investigation. Read the platform's documented device access requirements, and use the minimum permission needed. Avoid replacing or upgrading drivers just to try a different version when the current run is still needed.
Verify the actual execution path
For PyTorch, record torch.__version__, torch.version.hip, and the reported device name. HIP builds use the torch.cuda interface for many operations, including availability checks. A CUDA-named interface alone cannot identify the backend. Require both a HIP build and an available expected device before collecting ROCm performance results.
Create a small deterministic input on the CPU, calculate a CPU reference, then move the inputs to the selected GPU. Run the same operation and assert that its result tensor is on that device. Compare values using an appropriate tolerance. Use a small matrix multiplication or an operation central to the actual workload. Include an operation that previously failed, with the same input shape, dtype, and tensor layout. Keep the outputs small enough to inspect after a mismatch.
Measure and report
Synchronize before timing and at the end of each measurement, or use the framework's supported device timing mechanism. Warm up the workload before measuring steady state and report first-run compilation separately when relevant. Distinguish GPU compute from host-device transfer and request-level latency. Repeat runs and record median and variability. Compare candidate and baseline at matching precision, input shape, batch size, software versions, and memory constraints.
If the smoke test works but the workload fails, narrow the failure by operation, dtype, batch size, and backend extension. Capture a minimal reproducer before changing hardware or toolchain configuration. Stop treating a CPU fallback as GPU evidence. Report the exact working and failing tuples, compatibility source, correctness error, timing protocol, and any untested supported combinations.
References
Install this skill directly: skilldb add mlops-infrastructure-skills
Related Skills
Distributed Training
Triggers when users need help with distributed ML training, including data parallelism (DDP, FSDP), model parallelism (tensor, pipeline), DeepSpeed ZeRO stages 1-3, Megatron-LM, 3D parallelism, communication backends (NCCL, Gloo), gradient compression, checkpoint strategies, fault tolerance, and elastic training.
Feature Stores
Triggers when users need help with feature store architecture and implementation, including Feast, Tecton, and Hopsworks. Activate for questions about online vs offline feature serving, feature computation pipelines, point-in-time correctness, feature reuse, feature freshness, streaming features, and feature monitoring and drift detection.
Gpu Infrastructure
Triggers when users need help with GPU infrastructure for ML workloads, including GPU cluster architecture (A100, H100, H200, B200), NVIDIA CUDA ecosystem, multi-GPU training setup, InfiniBand networking, NVLink, GPU memory management, spot instances for training, cloud GPU comparison across AWS, GCP, Azure, Lambda, and CoreWeave, and on-prem vs cloud cost analysis.
Inference Optimization
Triggers when users need help with ML inference optimization, including model quantization (INT8, INT4, GPTQ, AWQ, GGUF), pruning strategies, knowledge distillation, ONNX Runtime, TensorRT, operator fusion, batching strategies, speculative decoding, and KV cache optimization. Activate for questions about reducing model latency, improving throughput, or lowering inference costs.
ML CI CD
Triggers when users need help with CI/CD for ML systems, including training pipelines, model validation, and deployment automation. Activate for questions about GitHub Actions or GitLab CI for ML, automated retraining triggers, model validation gates, deployment strategies (blue-green, canary, shadow), infrastructure as code for ML, and environment reproducibility with Docker, conda, and pip-tools.
ML Cost Optimization
Triggers when users need help with ML cost optimization, including compute cost management for training and inference, spot instance strategies, model size vs accuracy tradeoffs, right-sizing GPU instances, caching strategies, batch inference optimization, managed vs self-hosted infrastructure decisions, FinOps for ML teams, and cost attribution and chargeback models.