Profiling in PyTorch (Part 3): Attention is all you profile Hugging Face Models Datasets Spaces Buckets new Docs Enterprise Pricing Website Tasks HuggingChat Collections Languages Organizations Community Blog Posts Daily Papers Hardware Learn Discord Forum GitHub Solutions Team & Enterprise Hugging Face PRO Enterprise Support Inference Providers Inference Endpoints Storage Buckets Log In Sign Up Back to Articles a]:hidden"> Profiling in PyTorch (Part 3): Attention is all you profile Published July 10, 2026 Update on GitHub Upvote 28 +22 Aritra Roy Gosthipaty ariG23498 Follow Sergio Paniego sergiopaniego Follow Sayak Paul sayakpaul Follow Rémi Ouazan Reboul ror Follow Naive attention Naive attention with inplace causal masking Scaled Dot Product Attention Math backend Efficient backend Flash backend cuDNN backend Everything we covered, at a glance Concluding the series This is the third post of Profiling in PyTorch, a series where we slowly build the skill of reading profiler traces and use it to drive optimization: Profiling in PyTorch (Part 1): A Beginner's Guide to torch.profiler Profiling in PyTorch (Part 2): From nn.Linear to a Fused MLP Profiling in PyTorch (Part 3): Attention is all you profile (current) The series "Profiling in PyTorch" is meant to make you comfortable reading profiler traces and tables. In Part 1 we profiled basic math operations like addition and multiplication. We saw how the profiler table uncovers hotspots, and how the profiler trace shows the order in which an algorithm runs over time. In Part 2 we wrapped that addition and multiplication into a torch linear layer. We then stacked several linear layers on top of each other (a multilayer perceptron) and profiled that. Along the way we also profiled fused and hand-tuned kernels.
From the perspective of the Transformer architecture, the next logical step for us to profile is yet another fundamental algorithm, attention. While being infamous for its quadratic-time complexity, many clever tricks exist to mitigate that issue and make it fast. Our goal here is not to cover every trick in detail. Instead, we want to see how each one looks different under the profiler.
Source: https://huggingface.co/blog/torch-attention-profile
From the perspective of the Transformer architecture, the next logical step for us to profile is yet another fundamental algorithm, attention. While being infamous for its quadratic-time complexity, many clever tricks exist to mitigate that issue and make it fast. Our goal here is not to cover every trick in detail. Instead, we want to see how each one looks different under the profiler.
Source: https://huggingface.co/blog/torch-attention-profile