PyTorch 2.15 MPS inference timing on Apple Silicon
A synchronized M5 microbenchmark compares PyTorch 2.14 and 2.15 with identical FP32 weights and inputs. Separate first forwards from warmed inference, and see why batch size changes the upgrade verdict.
For one small FP32 network on an Apple M5, PyTorch 2.15 was modestly faster at batch 1; at batch 64, the observed process ranges overlapped. Batch size changed how much benefit the upgrade showed.
Across three fresh processes per version, the median warmed forward took 0.389 ms on 2.14.0 and 0.336 ms on 2.15.0 at batch 1. At batch 64, the corresponding times were 0.457 and 0.449 ms. Each forward-call time includes Python and synchronization overhead.
Native MPS kernels in PyTorch 2.15
As of 2026-11-01, the official PyTorch 2.15 announcement, dated October 28, describes additional native MPS kernels, a single-row F.linear decode path and FP8 support. The announcement also reports MoE decode results for its own workload and machine. Here, the comparison uses ordinary dense layers at FP32.
Hold the network fixed
The network has three linear layers: 1024 → 2048 → 1024 → 512, with GELU after the first two. Its 4,722,176 parameters are untrained. The script measures inference time and numerical agreement. NumPy generates the same FP32 weights and 64-row input in both environments. The script hashes all weight, bias and input bytes and asserts each full MPS output is close to its CPU reference.
Tested 2026-11-01 on an Apple M5 Mac, 10 CPU cores, 16 GiB unified memory, macOS 27.0.1 arm64, Python 3.12.13, NumPy 2.5.4, PyTorch 2.14.0 versus 2.15.0. Both versions ran on MPS with CPU fallback disabled. Both versions used .eval(), torch.inference_mode(), FP32 and one CPU thread. Other desktop applications remained active during measurement.
Download mps_benchmark.py, then create two environments:
uv venv -p 3.12 .venv214
uv venv -p 3.12 .venv215
uv pip install --python .venv214/bin/python torch==2.14.0 numpy==2.5.4
uv pip install --python .venv215/bin/python torch==2.15.0 numpy==2.5.4
.venv214/bin/python mps_benchmark.py --output run_1_214.json
.venv215/bin/python mps_benchmark.py --output run_2_215.json
For the reported comparison, I ran six processes sequentially, in the order 2.14, 2.15, 2.15, 2.14, 2.14, 2.15. No other experiment from this coordinator ran alongside that final sequence. The downloadable run instructions describe the fixture and measurements.
Each process measures the first forward for a shape, warms it with 20 calls, then records 21 windows of 20 forwards each. Every forward is synchronized. This is the timing boundary used in the full script:
import time
import torch
def timed_forward(model, x, repeats):
torch.mps.synchronize()
start = time.perf_counter()
for _ in range(repeats):
result = model(x)
torch.mps.synchronize()
return result, 1000 * (time.perf_counter() - start) / repeats
MPS execution is asynchronous, so a host timer needs synchronization to include GPU completion. torch.mps.synchronize() waits for MPS kernels to finish (checked 2026-11-01). Synchronizing every call is deliberate here: it also adds overhead and serializes the measured calls.
Warm forward times by batch size
Within each process, I take the median of the 21 window means. The table then reports the median and range of those three process-level medians. Each entry therefore summarizes repeated windows within a process.
| Batch | 2.14.0 median | 2.14.0 process range | 2.15.0 median | 2.15.0 process range |
|---|---|---|---|---|
| 1 | 0.389 ms | 0.347–0.433 ms | 0.336 ms | 0.328–0.346 ms |
| 64 | 0.457 ms | 0.445–0.470 ms | 0.449 ms | 0.443–0.468 ms |
The batch-1 ratio is about 1.16×, or 14% less elapsed time using the unrounded medians. At batch 64, the roughly 2% reduction is small relative to the observed variation. Three processes on one laptop cannot establish a general performance guarantee. The cover plots these medians with whiskers spanning the observed process ranges; confidence intervals were not estimated.
All six fixture hashes matched. Full output comparisons to the respective CPU reference passed at atol=2e-5, rtol=2e-4; the saved JSON records each maximum absolute error. This checks numerical agreement for the timing fixture.
First-forward times and startup costs
| Shape, in process order for that version | 2.14.0 first-forward times | 2.15.0 first-forward times |
|---|---|---|
| Batch 1 | 242.1, 51.9, 50.2 ms | 308.9, 71.9, 53.3 ms |
| Batch 64 | 4.4, 2.7, 2.6 ms | 7.5, 7.5, 7.5 ms |
Each first forward ran in a fresh process. OS/Metal shader caches remained uncleared, and the Mac stayed running between versions. Batch 64 is first for that shape but follows the batch-1 experiment in the same process. Input construction, CPU reference calculation and device upload happen beforehand and are recorded separately as setup_ms; imports and process startup are not included.
The first-forward observations give a separate reason to measure short scripts: 2.15 took longer for every first-forward comparison shown here, even though its warmed batch-1 median was lower.
These measurements cover one M5 Mac with other desktop applications active and only three processes per version. The untrained FP32 fixture leaves trained-model accuracy, application compatibility, FP8, grouped MoE multiplication, torch.compile, attention and full language-model throughput untested. GPU kernel timings and asynchronous server throughput require different timing boundaries.
For your upgrade, keep the current environment, reproduce a representative input in a second environment, check outputs, and measure both first-use and sustained behavior. Match the batch size, dtype and execution mode you actually deploy. If the workload uses the new FP8 or MoE paths, measure those paths with representative inputs.
Raw data: summary.json and the six process reports listed in README.txt. The summary script recomputes the aggregate from those reports. For the inference-mode distinction used here, see model.eval() vs no_grad().
