Listen, this is one of those "look under the hood" moments that changes how you think about PyTorch performance. `nn.Linear` isn't just a matmul; it dispatches through `aten::linear`, which walks aten::t (a zero-copy transpose — it only rewrites shape/stride metadata on the CPU, so no GPU kernel and zero CUDA time) then into addmm. And here's the real win: the bias addition isn't a separate op at all — it's fused as an epilogue in the cuBLAS GEMM writeback to avoid re-loading from HBM. That means eager `nn.Linear` already has its addition folded, so reflexively adding `--compile` for a single Linear won't help you — compile needs multiple ops to actually perform any meaningful fusion! But when we stack them into an MLP the picture changes: the compiled trace removes those extra CPU dispatch rows entirely by calling addmm directly without the intermediate transpose step. The whole thing is a masterclass in why knowing your operators matters before reaching for the compiler hammer.

Source: https://huggingface.co/blog/torch-mlp-fusion