Profiling in PyTorch (Part 1): A Beginner’s Guide to torch.profiler

Source: https://huggingface.co/blog/torch-profiler Back to Articles

• The matrix multiplication and addition operation
• 64x64 traces- Why does the ProfilerStep#2 take so long? - Why is there an offset of ~2.5 ms between the CPU and GPU lanes? - The chain of events - Why does matmul have an extra CUDA runtime call? - Why is cudaDeviceSynchronize so big (~1.78 ms)?
• 4096x4096 traces- Why does the same kernel take more time compared to others?
• Let’s see some torch compile at work- Did we fuse the matmul and add kernels into one? - torch.compile’s runtime architecture - Do the CUDA launches go down by half? - CPU overhead went up, not down
• Trace reading cheatsheet- Profiler table - CPU lane - GPU lane - Dispatch chain - torch.compile
• Conclusion

What you cannot profile, you cannot optimize.

Whether you are trying to squeeze more tokens per second out of a Large Language Model (LLM), shave milliseconds off inference, or just understand why your training loop runs slower than the spec sheet promises, the path eventually runs through profiling.

The catch is that profiling has asteepon-ramp. The traces are dense walls of colored rectangles. The events carry intimidating names. Most tutorials assume you can already read them. So even when weknowwe should be profiling, opening a trace can feel like a chore best left for later (or for someone else). This post, and the series it kicks off, is our attempt to lower that on-ramp.

This is the opening post ofProfiling in PyTorch, a series where we slowly build the skill of reading profiler traces and use it to drive optimization. The plan:

1. **Part 1 (this post):**start with the simplest possible operation, a matrix multiplication followed by a bias add, and learn how to read what the profiler hands back.
2. **Part 2:**scale up tonn\.Linearand a small MLP, use the traces to motivate optimizations, and peek at thekernelsunderneath.
3. **Part 3:**put it all together on Large Language Models withtransformers.

We document the journey from a beginner’s point of view. No prerequisites apart from basic PyTorch. Treat this as a leisurely read with some “Aha!” moments. The structure of the post is intentionally question-led: we open a trace, ask “wait, why isthathappening?”, and chase the answer until something clicks. By the end you should know:

• how to set uptorch\.profilerand what it actually hands back,
• how to read the profiler table and the trace (CPU lane, GPU lane, and the suspicious gaps in between),
• the chain of events from a Python call all the way down to a CUDA kernel,
• what changes (and, more interestingly, what doesnotchange) when you slaptorch\.compileon top.

Before we begin, two definitions that will make everything below read better:

1. A GPUkernelis a program that runs in parallel on many threads of the GPU.
2. The CPUschedules and launchesthese kernels.

You don’t usually have to write GPU kernels yourself; when you use a PyTorch operation, it is automatically translated to one or more kernels that do the job on GPU.

With those two ideas in your back pocket, let’s start asking questions.

Here is the entire script that we use for the post:01\_matmul\_add\.py. We recommend opening this script in a separate tab and walk through the code step by step. We use theNVIDIA A100\-SXM4\-80GBGPU to run the scripts.

https://huggingface.co/blog/torch-profiler#the-matrix-multiplication-and-addition-operationThe matrix multiplication and addition operation

As correctlyquipped by Dr. Sara Hooker, just as we are primarily made up of water, Deep Neural Networks are primarily made up of matrix multiplies. As fundamental as they are, it would be a shame to start our profiling journey with anything else.

def fn(x, w, b):
  return torch.add(torch.matmul(x, w), b)

The matrix addition along with the matrix multiplication mimics how weights and biases interact in a neuron. This addition (pun intended) will help us understand how it paves the way for compilationlater in the post.

To profile, we will be using thetorch\.profilermodule. The steps involved are:

1. Have thecode to profile ready(heredef fn, which wraps the matrix multiplication and matrix addition)
2. Annotatethe algorithm. While this is completely optional, we recommend doing this. Therecord\_functionannotates our function asmatmul\_add, which will be easy to navigate in the traces (as we note later)

def step():
  with torch.profiler.record_function("matmul_add"):
    return fn(x, w, b)

1. Wrap the code with thetorch\.profiler\.profilecontext manager

with torch.profiler.profile(
    activities=[
        torch.profiler.ProfilerActivity.CPU,  # the cpu activities
        torch.profiler.ProfilerActivity.CUDA, # the gpu activities
    ],
  ) as prof:
    # it is recommended to run events multiple times to warm up the GPUs
    for _ in range(5):
      step()
      prof.step()

1. Export theprofile

# the profiler table
prof.key_averages().table(sort_by="cuda_time_total", row_limit=15)

# the profiler trace
prof.export_chrome_trace(trace_path)

The profiler exports two distinct artifacts:

1. The profiler table: Provides the statistical summary of the algorithm. It answers “What is taking the most time”. This becomes really helpful to figure out hotspots. A hotspot would be events that take the most amount of time, can be a bottleneck of the pipeline, or an event that is triggered a lot of times.
2. The profiler trace: Provides the temporal execution view. Answers “When and Why an operation happened”, depicting the activities taking place on the CPU and the GPU. This is helpful when we want to investigate the kernel(s) that were launched, any delays in launching them, any overlap between CPU and GPU activities, etc.

Let’s see the two in action with our first execution. (Here is the entire01\_matmul\_add\.pyscript)

It is recommended to run this script on a machine with a GPU.

uv run 01_matmul_add.py --size 64

If you run the above script (on a GPU machine) you will find a foldertraces/01\_matmul\_addwith the two artifacts:

64_bf16_cold_eager.json
64_bf16_cold_eager.txt

Figure 1: Profiler table for matmul add on 64 sized matrices The\.txtfile holds the profiler table. Upon opening the file, as shown in Figure 1, one would be greeted with a big table with the first column consisting of the events that were triggered inside the scope of profile.

The other columns are related to the time the event takes on the CPU or GPU or any other device(s) specified inactivitieswithintorch\.profiler\.profile. Look at which events take the most amount of time, and try to intuitively understand if that event should in fact take that time. It is also important to look at the column “# of Calls” which dictates how many times the event was triggered.

While we are at it, let’s also talk about “Self CPU/CUDA” vs “CPU/CUDA total”. The “Self” columns measure time spent only inside the event itself, excluding its children. The “total” columns include the event and all of its children together. So if you look at the “CPU total” ofmatmul\_add, it consists of the time it took on self plus the children events it triggered. This is an important nuance to note.

If you look at the last two lines out of the table you would notice that the profiler tells us that

Self CPU time total: 2.314ms
Self CUDA time total: 23.104us

The CPU time is inmswhile the GPU time is inus. To put things in perspective, the time spent on GPUs (the kernelampere\_bf16\_s16816gemm\.\.\.) is less than 1% of the time spent on the CPU (thematmul\_addoperation). The GPU stays idle most of the time, which is an immediate red flag. The reason this happens is that the GPU can compute a small matmul very quickly, so our code spends most of the time preparing the kernels, launching them on the GPU, sending the data to multiply and gathering the results. This concept is known as anoverhead-boundalgorithm.

The easiest way to move out of this regime is to use bigger matrix multiplications.

uv run 01_matmul_add.py --size 4096

Figure 2: Profiler table for matmul add on 4096 sized matrices The last two lines in Figure 2 are:

Self CPU time total: 4.908ms
Self CUDA time total: 4.495ms

Both times are in ms, which means we have materialized more GPU time just by increasing the size of the matrix multiplications. If you look at Figure 2 you would also notice that the most CUDA time is now taken by the GPU kernel (ampere\_bf16\_s16816gemm\_\.\.) and not by the CPU operation that launched it (matmul\_add). This means that we were indeed able to move from overhead bound to compute bound.

We now move into visualising the dispatch chain, which lives inside the\.jsonartifacts. You can upload them toPerfetto UIand see the traces, or you can useuvx trace\-util traces \-b tracesto generate the Perfetto links directly.

https://huggingface.co/blog/torch-profiler#64x64-traces64x64 traces

Figure 3: Profiler trace for matmul and add on 64 sized matrices In Figure 3, we see the profiler trace for the matrix multiplication and addition. Here the bar width indicates the duration of an event, the vertical nesting is the call hierarchy, the CPU lane denotes the events that happen on the CPU, while the GPU lane shows the actual kernel executions. One might also notice the empty spaces which are the waiting or idle time.

The script was run with default configurations which are:

• size 64: The inputs, weights and biases are sized (64, 64)
• dtype bf16: The data type is bfloat16
• no compile: We have not compiled the torch operations
• no warmup: We have not warmed up the GPU before profiling

With Perfetto we suggest using the keyboard for quicker access to the trace. One could use “W A S D” for navigating the trace.

Figure 4: The CPU and GPU lanes of a PyTorch profiler trace There are two lanes in Figure 4, one for the CPU activity and one for the GPU activity. In the CPU lane one would notice three profile steps (starting fromProfilerStep\#2). This comes from theschedule.

schedule = torch.profiler.schedule(wait=1, warmup=1, active=3, repeat=1)

Thewaitskips noisy initializations (ProfilerStep\#0),warmupruns through the profiler without recording (ProfilerStep\#1), andactiveis what shows up in trace. One can find the schedule being used in thescript here.

Let’s put on our detective hats and investigate the trace and ask some questions.

https://huggingface.co/blog/torch-profiler#why-does-the-profilerstep2-take-so-longWhy does the ProfilerStep#2 take so long?

Figure 5:ProfileStep\#2is visibly wider than the steps that follow it In Figure 5, we notice thatProfileStep\#2takes more time compared to the other steps, and upon looking closely you would see a similar pattern with thematmul\_addannotation as well. The smoking gun is inside the annotation, not the annotation itself:

Stepmatmul\_addstartaten::matmulstartgap#2138.736366.493227.757 µs#3517.926523.4475.521 µs#4610.039614.5274.488 µs Figure 6: The ~228 µs dead window betweenrecord\_function\("matmul\_add"\)andaten::matmul That ~228 µs shown in Figure 6 is the “dead window” between enteringrecord\_function\("matmul\_add"\)and PyTorch actually dispatchingaten::matmul. This can happen for multiple reasons, including workspace allocations,cuBLAS(NVIDIA’s proprietary, GPU-accelerated library for performing fundamental linear algebra operations) heuristics, or lazy module loading. We can either look away or runsome more warmup stepsbefore we profile (which is the standard)

In terms of profiling, warmup is when you run the events a couple of times before actually profiling it. The pre-work done by the GPU (including the above pointers) are one time efforts which we do not want to profile. In our example, we have two warmup stages, one where we actually loop over the function before entering the profiler, and two inside the profiler which is achieved by thewarmupargument. In this section, we have enabled the actual iterations along with the schedule.

uv run 01_matmul_add.py --warmup

Perfetto Trace for 64x64 with Warmup

Figure 7: After warming up, every profile step takes a similar amount of time In Figure 7 we see that each profile step takes a similar time, but this does not mean we were able to optimize the one time overheads. We warmed up the runs so that the overheads were not profiled. We think that closing this section abruptly without a hint to solving this would do injustice to the reader, so here is alinkto read about further optimizing launch overheads.

https://huggingface.co/blog/torch-profiler#why-is-there-an-offset-of-25-ms-between-the-cpu-and-gpu-lanesWhy is there an offset of ~2.5 ms between the CPU and GPU lanes?

Figure 8: The ~2.5 ms offset between the CPU and GPU lanes In Figure 8, we see that the CPU and GPU lanes have an offset of around 2.5 ms: this is the delay after the CPU submits the CUDA kernels and the time they actually start executing. One might think the warmup stage combined with the schedule’swaitandwarmupshould keep a GPU busy and would diminish the offset.

To uncover what is really happening, let’s change our schedule a little:

- schedule = torch.profiler.schedule(wait=1, warmup=1, active=3, repeat=1)
+ schedule = torch.profiler.schedule(wait=0, warmup=0, active=3, repeat=1)

Figure 9: Withwait=0andwarmup=0, the trace reveals anActivity Buffer Request Figure 9 shows us that there is anActivity Buffer Requestin the GPU lane before any operation. Let’s zoom in a little more.

Figure 10: A gap appears between the matmul and add kernels on profile step 1 Upon zooming into the GPU trace, we notice that the matmul and add kernels forProfileStep\#0(the CPU trace of which is not visible in the Figure) happen one after the other, while the kernels forProfileStep\#1have a window in between. The best explanation for this is that there was an overflow of buffers, and another buffer request (a request to allocate some memory on the GPU VRAM) was issued during the kernel execution.

The best way to rule out other possibilities is to profile for more iterations and see whether a similar window appears in other parts of the trace. To do that we run withactive=20.

Figure 11: With 20 active steps the gap only shows up once, confirming it is a buffer request As shown in Figure 11, we see a similar trend inProfileStep\#1. This is aligned with our previous findings, and we can safely conclude that it was indeed another buffer request.

https://huggingface.co/blog/torch-profiler#the-chain-of-eventsThe chain of events

Figure 12: The chain of dispatch In Figure 12, we see the nested CPU calls. This is an important visualization, where one gets to understand what a chain of dispatch really looks like.

We begin withProfileStep\#<id\>which encapsulates the profiling step. Due to us annotating the step, we see thematmul\_addrow. Thematmul\_addconsists of twoatencalls, one for matrix multiplication and one for matrix addition.

Theaten::matmulis theATen-leveldispatch that those user-facing PyTorch matmul calls land on.aten::mmis the 2D matrix-matrix multiply backend.

It is very interesting to note how PyTorch callsaten::bmm(batched matrix multiplication) if we add the batch axis to our matrices. Let’s take a detour and see theaten::bmmin action.

- x = torch.randn(args.size, args.size, device=device, dtype=dtype)
- w = torch.randn( args.size, args.size, device=device, dtype=dtype)
- b = torch.randn(args.size, args.size, device=device, dtype=dtype)

+ # adding a batch size of 8
+ x = torch.randn(8, args.size, args.size, device=device, dtype=dtype)
+ w = torch.randn(8, args.size, args.size, device=device, dtype=dtype)
+ b = torch.randn(8, args.size, args.size, device=device, dtype=dtype)

Figure 13: Batched Matrix Multiplication In Figure 13, upon adding the batch axis to the inputs,aten::matmulnow encapsulates a bunch of other prerequisite CUDA runtime calls along withaten::bmm(instead ofaten::mm). This also hints at the heuristics that cuBLAS needs to do in order to dispatch the right (most suitable) kernel for the program.

In the rest of the post, we will be working with simple 2D matrices, unless otherwise mentioned.

https://huggingface.co/blog/torch-profiler#why-does-matmul-have-an-extra-cuda-runtime-callWhy does matmul have an extra CUDA runtime call?

Figure 14: A CUDA occupancy query fires before the matmul kernel launch We notice that foraten::mmthere are two CUDA Runtime calls, namelycudaOccupancyMaxActiveBlocksPerMultiprocessor(boxed in Figure 14) andcudaLaunchKernel, while foraten::addthere is only thecudaLaunchKernel.

cudaOccupancyMaxActiveBlocksPerMultiprocessoris a planning call and is purely CPU side. It asks: “given a kernel function, a chosen block size, and a chosen dynamic shared memory size, how many blocks of this kernel can simultaneously reside on one SM (Streaming Multiprocessor)?”

This begs the question, why do we need planning for matmul and not for add?

To understand this, we have to look at the kernel’s resource footprint. If you click on the GPU kernels, you will be able to inspect the resource footprint for the respective kernel.

In Figure 15, we note that for matrix multiplication theregisters per threadandshared memoryare dynamic (based on the size of the matrix). cuBLAS ships hundreds of kernel variants, and each has a heuristic-driven launch path that needs runtime information about hardware capacity. The occupancy query is part of that heuristic. Conceptually, we can think of GPU-accelerated matmuls asworking on independent tiles: how many tiles we use and how big each tile needs to be depends on the matrices and the hardware. Modern algorithms are way more complicated than that, but this is still a good reference framework.

From Figure 16 we see that the footprint of addition says 32 registers and zero shared memory. That fits trivially. There’s nothing to query, because no hardware resource is going to limit occupancy. The kernel is, by design, resource-light.

You can use this as a quick diagnostic when reading any trace. Scan the CPU lane forcudaOccupancyMaxActiveBlocksPerMultiprocessor. Each occurrence flags a “heavyweight, adaptively launched” kernel, usually a GEMM (GEneral Matrix Multiplication), conv, or similar. The kernels without a preceding occupancy query are the elementwise/reduction crowd that PyTorch launches mechanically.

https://huggingface.co/blog/torch-profiler#why-is-cudadevicesynchronize-so-big-178-msWhy is cudaDeviceSynchronize so big (~1.78 ms)?

cudaDeviceSynchronizeblocks the CPU until all GPU work on this device finishes. The profiler emits this sync at the end of the active window to flush events. Without it, kernel timings would be missing.

A 1.78 ms sync covering 26 µs of real GPU work tells you this run was 98% idle. That’s the textbook overhead-bound symptom.

https://huggingface.co/blog/torch-profiler#4096x4096-traces4096x4096 traces

We already know from the profiler table analysis (above) that providing bigger matrices to our algorithm moves it out from the overhead-bound region to being compute-bound.

Let’s run the command and dive deeper into the traces.

uv run 01_matmul_add.py --size 4096 --warmup

https://huggingface.co/blog/torch-profiler#why-does-the-same-kernel-take-more-time-compared-to-othersWhy does the same kernel take more time compared to others?

Figure 17: One matmul kernel runs longer than the others despite identical inputs In Figure 17, we notice that the matmul kernel forProfileStep\#3takes longer on the GPU than the other steps. This is particularly interesting to note, because the other kernels launched were the exact same, which means there were no cuBLAS heuristics involved. There are no scheduling gaps, the CPU launches are normal, and it is not a profiler artifact.

This trace in Figure 17 makes a useful point that’s easy to miss in idealized examples: kernel runtimes are not constants, even on the same hardware environment running identical code on identical data.

Let’s make this more concrete by modifying the script a little. We run the iteration 20 times, capturing each of the steps.

- schedule = torch.profiler.schedule(wait=1, warmup=1, active=3, repeat=1)
+ schedule = torch.profiler.schedule(wait=0, warmup=0, active=20, repeat=1)

- for _ in range(5):
+ for _ in range(20):

Figure 18: Across 20 iterations the same matmul kernel runs at different speeds Figure 18 reveals a similar finding. While each kernel was the exact same, they time differently. The different compute times can be blamed on a bunch of reasons:

• GPU clocks on idle and boost
• GPU heating
• GPU power management
• Driver side housekeeping

A reader who only saw the average would conclude that a matmul took ~1 ms (mean of 5 = 1084 µs); a reader who looked at the trace would see that the matmul takes ~580 µs except when the GPU throws a fit. Those are very different mental models, and only one of them is correct.

https://huggingface.co/blog/torch-profiler#lets-see-some-torch-compile-at-workLet’s see some torch compile at work

Working withtorch\.compilehas always amazed us. One writes normal eager PyTorch code, but PyTorch tries to capture tensor-heavy regions, turn them into graphs, optimize them, and run generated code. The default backend is usuallyTorchInductor, and the broad pipeline is:

1. TorchDynamocaptures Python execution into an FX graph
2. AOTAutogradprepares forward/backward graphs when gradients are involved
3. Inductorlowers the graph into optimized CPU or GPU code.

In this section, we talk about compilation and look at the profiler traces.

uv run 01_matmul_add.py --size 4096 --warmup --compile

Theargs\.compileflag triggers the following code:

def fn(x, w, b):
  return torch.add(torch.matmul(x, w), b)

fn = torch.compile(fn) if args.compile else fn

Figure 19: The compiled regions show up as TorchDynamo and Inductor frames in the trace In Figure 19, we see the new CPU rows namedTorch\-Compiled Region: 0/0which points us to the compiled functions being used.

https://huggingface.co/blog/torch-profiler#did-we-fuse-the-matmul-and-add-kernels-into-oneDid we fuse the matmul and add kernels into one?

Figure 20: Compiled run dispatches a singleaten::addmm Looking at Figure 20 we ask the question, did we actually fuse the multiplication and addition operations together into one?

This is operator fusion at the graph level. Inductor took ourtorch\.add\(torch\.matmul\(x, w\), b\)and rewrote it into a singleaten::addmm\(b, x, w\)call. The important thing to note here is that it didnotproduce anewfused CUDA kernel. The actual GPU work is stillampere\_bf16\_s16816gemm\_bf16\_128x256\_ldg8\_f2f\_stages\_64x3\_nn, the same cuBLAS kernel eager mode used. So the “fusion” here is at the dispatcher level, not at the kernel level.

PyTorch provides thetorch\.addmmfunction that does what we did into two steps, that is multiply and add. We encourage the reader to look at the traces of this function and comment your observations in the comments below!

https://huggingface.co/blog/torch-profiler#torchcompiles-runtime-architecturetorch.compile’s runtime architecture

While we know in theory what happens when we compile our functions it is equally important to see it in action. Let’s look at the CPU-side hierarchy which reflectstorch\.compile’s runtime architecture.

TorchDynamo Cache Lookupis where Dynamo checks that the current call still matches what was compiled with the same input shapes, dtypes, devices, and tensor metadata. If anything mismatched, Dynamo would recompile. This cost is paid every call, even after compilation.

Torch-Compiled Regionis the wrapper that “enters” the compiled version.AOTDispatcher Runtime Wrapper Prologueis AOT Autograd’s runtime wrapper. Even though we don’t need gradients here, AOTDispatcher is always in the stack handling tensor metadata, view tracking, and would set up the backward pass ifrequires\_gradwere true.

## Call CompiledFxGraphis where the actual generated code runs. The string after “CompiledFxGraph” is the content hash of the FX graph. It’s the same across all three active steps, confirming cache hits.

You can find the generated code on disk under/tmp/torchinductor\_<user\>/fxgraphkeyed by this hash, useful when you want to read the Triton/C++ that Inductor actually produced.

https://huggingface.co/blog/torch-profiler#do-the-cuda-launches-go-down-by-halfDo the CUDA launches go down by half?

Figure 21: Each compiled step still launches two GPU kernels, a Device-to-Device memcpy and the GEMM Looking at the traces in Figure 21, we were really happy to notice only onecudaLaunchKernelper step. This observation was directly contradicting what we were seeing in the GPU trace. There were still two kernels being launched per step, namely theMemcpy DtoD \(Device \-\> Device\)and the GEMM. Going back to the CPU trace, we noticed that we had completely missed thecudaMemcpyAsyncdispatch.

addmmcomputesout = α·A·B \+ β·C, and cuBLAS’s GEMM-with-bias-add epilogue writes into a destination buffer that needs to already contain the bias. An epilogues can be thought of all the operations that happenaftera GEMM. In the world of deep-learning we constantly come up with GEMM-Epilogues like activations, bias addition, normalization and many more. This is why there are cuBLAS GEMM-with- kernel variants.

If you use differentmodes fortorch\.compileyou would notice different kernel variants being launched. You can try it for yourself and add a comment below about your observations!

So Inductor’s generated code does:

• out = copy\(C\)← that’s the DtoD memcpy (32 MB, takes ~33 µs)
• out = α·\(A·B\) \+ β·out← GEMM withα=β=1, fusing the bias add into the writeback

The result is mathematically still the same. The bias add isn’t free, as we pay a memcpy upfront plus a slightly more expensive GEMM epilogue.

The fusion one might have hoped for, wherex·w \+ b(hereout = α·A·B \+ β·C) collapses into a single kernel with no extra memory traffic, isn’t what happened. Inductor preserved the two memory-touching operations, it just relabeled the bias copy as a memcpy and the addition as a GEMM epilogue.

A truly fused implementation would skip the memcpy. That’s what FlashAttention-style hand-written kernels do, and what Inductor can do via Triton codegen, but for a4096×4096 bf16 matmul, Inductor evidently decided “use cuBLAS, do the bias via epilogue setup” was the best path.

https://huggingface.co/blog/torch-profiler#cpu-overhead-went-up-not-downCPU overhead went up, not down

This is the easiest thing to miss when comparing an eager and a compiled run:

stepeager dur (ms)compile dur (ms)#20.10.2#30.070.1#40.070.1 Compile is roughly 2× more expensive on the CPU per step. That’s because every call walks the full Dynamo > AOTAutograd > Inductor stack, on top of the sameaten::addmmdispatch we have anyway. The compile pipeline is built for ML models with dozens of ops where the per-call overhead amortizes (for a single op it’s a tax).

torch\.compilehas amodeargument. It is for the reader to take home as an assignment to read the documentation and come up with amodethat could take the CPU overhead down. 🤗

https://huggingface.co/blog/torch-profiler#trace-reading-cheatsheetTrace reading cheatsheet

A quick reference for the patterns we walked through. The idea is: if you see this in a trace, this is what it usually means.

https://huggingface.co/blog/torch-profiler#profiler-tableProfiler table

What you seeWhat it usually meansSelf CPU time total≫Self CUDA time total(CPU in ms, GPU in µs)Overhead-bound. The CPU spends more time dispatching than the GPU spends computing. Make the work bigger (larger matrices, batched ops) or fuse calls.Self CPU time total≈Self CUDA time total, both in msCompute-bound. The GPU is the bottleneck, which is usually what you want.One event dominatesCUDA totalThat’s your hotspot. Start the optimization there.One event has a huge\# of CallsA potential bottleneck even if each call is cheap. Check whether it can be fused or batched.CPU total≫Self CPUfor a rowMost of the cost lives in children. Drill into the nested events, not the parent.

https://huggingface.co/blog/torch-profiler#cpu-laneCPU lane

What you seeWhat it usually meansFirstProfileStepmuch wider than the restCold-start overhead: workspace allocation, cuBLAS heuristics, lazy module loading. Add warmup iterations and/or the schedule’swarmupargument.Big gap betweenrecord\_function\("\.\.\."\)start and the firstaten::\*inside itSame cold-start tax, just zoomed in. The annotation entered, but the dispatch hadn’t happened yet.cudaOccupancyMaxActiveBlocksPerMultiprocessorbefore acudaLaunchKernelA heavyweight, adaptively-launched kernel (GEMM, conv, etc.). cuBLAS is asking the driver how many blocks fit on an SM so it can pick a kernel variant.cudaLaunchKernelwith no preceding occupancy queryAn elementwise or reduction kernel with a fixed, resource-light footprint. Nothing to plan.A longcudaDeviceSynchronizeat the end of the active windowThe profiler flushing events. Its duration is mostly the GPU finishing pending work, not a real CPU cost. A sync covering tiny GPU work is a classic overhead-bound symptom.AcudaMemcpyAsyncyou didn’t writeOften a hidden Device-to-Device copy. Common whenaddmmseeds its destination buffer with the bias before the GEMM epilogue.

https://huggingface.co/blog/torch-profiler#gpu-laneGPU lane

What you seeWhat it usually meansActivity Buffer Requeston the GPU laneThe profiler is allocating/refilling its own event buffer. The first one usually accounts for the initial CPU↔GPU lane offset.A gap between two kernels in a single stepLikely another buffer request mid-execution. Confirm by running more iterations: if it appears only once, it’s the profiler, not your code.The same kernel timing differently across stepsGPU clocks, thermals, power management, driver housekeeping. Read the trace, not just the mean.A kernel named likeampere\_bf16\_s16816gemm\_\.\.\.The actual cuBLAS GPU work for a matmul. The kernel name is typically the same in eager and compiled mode for the same shapes/dtypes.Memcpy DtoDimmediately before a GEMMThe bias copy for anaddmmepilogue. The “fusion” is at the dispatcher level, not in the kernel.

https://huggingface.co/blog/torch-profiler#dispatch-chainDispatch chain

What you seeWhat it usually meansProfileStep\#N→<record\_function name\>→aten::\*→aten::mm/aten::bmm/aten::addThe canonical nested call hierarchy. Self time excludes children; Total time includes them.aten::matmulresolving toaten::mm2D × 2D matrix multiply.aten::matmulresolving toaten::bmm(with extra CUDA runtime calls)Batched matmul on 3D+ tensors. cuBLAS does more heuristic work to pick the variant.aten::addmm\(b, x, w\)instead of a separateaten::add+aten::mmpairOperator fusion at the dispatcher level. The GPU kernel is still the same GEMM, with the bias add folded into the epilogue.

https://huggingface.co/blog/torch-profiler#torchcompiletorch.compile

What you seeWhat it usually meansATorch\-Compiled Region: K/Mrow in the CPU laneYou’re inside a compiled function.TorchDynamo Cache Lookupon every stepDynamo is verifying shapes/dtypes/devices match the cached compile. Paid on every call, even after compilation.AOTDispatcher Runtime Wrapper Prologueeven with no gradsAOTAutograd’s runtime wrapper is always in the stack, handling tensor metadata and view tracking.\#\# Call CompiledFxGraph <hash\>with the same hash across stepsCache hits on the generated code. The generated source lives under/tmp/torchinductor\_<user\>/fxgraph/<hash\>.Per-step CPU time higher undertorch\.compilethan eager for a tiny opExpected. The Dynamo → AOTAutograd → Inductor stack is a tax that only amortizes over many ops.

https://huggingface.co/blog/torch-profiler#conclusionConclusion

We started with a tinymatmul \+ addand used it as an excuse to learn how to read a PyTorch profiler. Along the way we picked up a few mental models that travel well to bigger workloads. This was the first stop in theProfiling PyTorchseries. In the posts that follow, we will gradually leave this two-op toy behind and walk up the ladder of complexity, looking at larger building blocks and, eventually, real models.

Thanks toNoe Flandre,Suvaditya Mukherjee, andVidit Ostwalfor their reviews on the early draft of the post!