Skip to main content
Technology & EngineeringPerformance Optimization39 lines

HPC Performance Measurement

Measure high-performance computing workloads with controlled MPI and OpenMP placement, strong and weak scaling, and separate compute, communication, and I/O evidence. Use when comparing scientific builds, parallel settings, or cluster runs.

Quick Summary11 lines
Produce a repeatable scaling report that compares equivalent work and ties slowdowns to observed phases. Choose the smallest benchmark that includes the performance question and validate its numerical result before timing it.

## Key Points

- [MPI Forum MPI 4.1 standard and timing semantics](https://www.mpi-forum.org/docs/mpi-4.1/mpi41-report.pdf)
- [Open MPI MPI_Wtime documentation](https://docs.open-mpi.org/en/main/man-openmpi/man3/MPI_Wtime.3.html)
- [OpenMP thread affinity](https://www.openmp.org/spec-html/5.1/openmpse61.html)
- [Slurm multicore and multithread support](https://slurm.schedmd.com/mc_support.html)
- [Slurm MPI launch guide](https://slurm.schedmd.com/mpi_guide.html)
skilldb get performance-optimization-skills/hpc-performance-measurementFull skill: 39 lines
Paste into your CLAUDE.md or agent config

HPC Performance Measurement

Produce a repeatable scaling report that compares equivalent work and ties slowdowns to observed phases. Choose the smallest benchmark that includes the performance question and validate its numerical result before timing it.

Define the experiment

State the kernel or end-to-end job, dataset shape, precision, solver tolerances, output mode and success checks. Use a fixed total problem for strong scaling. Use a fixed per-worker problem for weak scaling and state what grows, including boundary or exchange work. Pick the baseline and comparison counts before collecting times. If memory prevents the requested one-worker run, use a documented smaller parallel baseline and report the actual baseline size.

Keep software and input revisions, node type, compiler and MPI/OpenMP/runtime options constant for comparisons. Change one scheduling or tuning decision at a time. Record MPI ranks per node, threads per rank, hardware threads, socket/NUMA placement, accelerator ownership and requested versus observed affinity. Check for hidden thread pools in BLAS or other libraries. Confirm the intended process placement through supported runtime reports instead of assuming a scheduler option was honored. Use the site-approved launcher, scheduler and allocation.

Time the correct quantities

Separate queue wait, application setup, warm-up, compute, data exchange, checkpoint and final output as needed. Capture both end-to-end application time and the phase that motivated the change. Decide whether filesystem or application caches should be cold or warm and repeat that condition consistently.

For an MPI phase, calculate elapsed time on each rank with local start and stop readings from MPI_Wtime. Use the maximum rank elapsed time for a coordinated phase whose completion depends on the slowest rank, and keep the distribution to diagnose imbalance. Cross-rank timestamp subtraction is invalid without a verified synchronized-clock contract. If a barrier is needed to define a benchmark start, state where it is and whether its cost is included. Do not add barriers to the measured algorithm merely to make timing easier.

Repeat each point and report the median and spread alongside all recorded samples. Track failed or numerically invalid trials explicitly. Strong-scaling speedup relative to baseline rank count p0 is T(p0)/T(p); efficiency is that ratio divided by p/p0 when ranks are the scaling unit. If both thread and rank counts change, use the declared resource count consistently. For example, a 4-rank baseline at 12 seconds and a 16-rank point at 5 seconds yield 2.4× speedup and 60% efficiency, assuming equal threads per rank. For weak scaling with equal work per worker, compare elapsed time relative to baseline and state any changing communication workload.

Isolate a bottleneck

Compare elapsed phase times and rank variance before explaining a plateau. A split between intra-node and inter-node runs can separate topology effects. Compare compute-only versus full I/O as an explicit alternate measurement. Use only supported counters or profilers and identify instrumentation overhead; elapsed time, CPU cycles, bandwidth and allocated memory answer different questions. If shared-resource contention changes, mark the run and repeat under comparable conditions when possible.

Publish the raw samples, exact launch and affinity evidence, phase definition, correctness checks, comparison table and limits. Include a control with an intentionally different workload or invalid result to verify the report rejects unlike runs. If no cluster is available, a local fixture can test calculation and reporting; it cannot establish multi-node scaling or cluster placement.

References and rights

Original SkillDB instructions under LicenseRef-SkillDB-Content. References were checked 2026-11-13 and used for technical verification.

Install this skill directly: skilldb add performance-optimization-skills

Get CLI access →