Understanding the Go Runtime: Profiling

# Profiling Source: [https://internals-for-interns.com/posts/go-runtime-profiling](https://internals-for-interns.com/posts/go-runtime-profiling) In the[previous article](https://internals-for-interns.com/posts/go-runtime-reflect/)we took apart the reflect package and found that its magic is mostly the*compiler leaving very good notes*— type descriptors frozen into read\-only data at build time, and a package that knows how to walk them\. The whole article was about reading metadata that was already sitting in memory before`main`even started\. Today we shift the perspective\. Profiling is the runtime catching your program*in motion*— sampling what it’s doing and where it’s spending its time, then accumulating that into something you can open with`go tool pprof`\. The heart of every sample is a**call stack**, produced with the same unwinder we saw in the[Stacktraces article](https://internals-for-interns.com/posts/go-runtime-stacktraces/)\. So profiling is really live*moment\-catching*sitting on top of build\-time*stack\-reading*\. But Go doesn’t have just one type of profile — it provides**five**: CPU, heap, block, mutex, and goroutine\. At first they look like five unrelated subsystems, but they all share the same skeleton, and once you see it, the whole thing collapses into one idea repeated five ways\. \(A sixth, a*goroutine\-leak*profile, is on its way in Go 1\.27 — Alex Rios has a great[series of posts](https://alexrios.me/blog/goroutine-leak-detection-orphans/)digging into it — but we’ll stick to the five that ship today\.\) Let’s start there\. The deepest similarity between all five profiles is the thing you actually walk away with:**the file**\. No matter which profile you collected, what lands on disk is the same format — and it’s surprisingly simple: the*pprof profile*, a gzip\-compressed[protocol buffer](https://github.com/google/pprof/blob/main/proto/profile.proto)\. It’s not even Go\-specific; it’s the same format Google’s C\+\+ profiler \(gperftools\) emits, and one`go tool pprof`shares with the wider pprof ecosystem\. So before we look at how each profile is*collected*, let’s look at what they’re all collected*into*\. At the top level it’s one`Profile`message, and the part that matters is just a handful of repeated fields: ``` message Profile { repeated ValueType sample_type = 1; // what each number in a sample means repeated Sample sample = 2; // the actual data repeated Mapping mapping = 3; // the loaded binary / shared libraries repeated Location location = 4; // a PC, resolved to a place in the code repeated Function function = 5; // name, file, start line repeated string string_table = 6; // every string, deduplicated } ``` The clever part is how little is stored inline\. A`Sample`is almost nothing — a list of values and a list of*location IDs*, leaf first: ``` message Sample { repeated uint64 location_id = 1; // the call stack, as references repeated int64 value = 2; // e.g. [sample count, cpu nanoseconds] repeated Label label = 3; // extra key/value tags attached to the sample } ``` Each of those pieces lives in its own table inside the`Profile`, and they all reference each other by id\. Put together, the whole thing looks like this \(a simplified visual of it\):  What the diagram makes visible is the indirection\. A`Sample`never holds a function name or even a stack frame — it holds a list of`location\_id`s, one per frame in the stack trace, leaf first\. Follow one of those into the`location`table and you reach a`Location`, which points \(via a`Line`\) at a`Function`and also records which`Mapping`it came from — the loaded binary or shared library that address lives in\. The`Function`finally holds the human\-readable bits — name, file, start line — except those aren’t strings either; they’re integer indices into the one shared`string\_table`at the bottom\. So it’s a chain of lookups: sample → location → function → string\. And nothing is duplicated: the same`Location`,`Function`, and string are referenced by every sample that touches that spot in your code, so a stack that shows up in ten thousand samples stores its function names just*once*\. This is also where the abstract “value” of each profile gets its meaning: the`sample\_type`declares what the numbers in each sample actually measure\. That’s the one part of the shape that genuinely differs between profiles — same container, different labels on the columns — so we’ll fill it in as we get to each profile type in the rest of the article\. So much for what a profile*is*\. Now let’s see how the runtime actually fills one in\. ### How the data gets there Whichever profile you’re collecting, one thing is constant: somewhere along the way the runtime**captures a call stack**with the unwinder, and at the very end it all comes out as the pprof file we just saw\.*How*a stack gets from the one to the other, though, is not the same across the five — and that difference is the real story\. The five sort into**three collection models**: - **CPU**records*asynchronously*\. A signal interrupts a running thread, the handler captures the stack and logs it into a ring buffer, and a separate background goroutine drains that buffer as it fills\. A genuine streaming pipeline\. - **Heap, block, and mutex**record*in place*\. When the event fires, the stack is hashed into a long\-lived table of per\-stack records and the matching record’s counters are bumped right there\. Nothing streams and nothing drains in the background — the data just sits and accumulates, and the profile is assembled on demand, the moment you ask for it\. - **Goroutine**doesn’t record during execution*at all*\. There’s no trigger and no buffer\. When you ask for the profile, the runtime walks every live goroutine’s stack right then, in one snapshot\. So the thing that really changes from profile to profile isn’t a single knob — it’s the whole collection model\. Let’s walk the five, grouped by the model they use\. ## CPU profiling: the signal\-driven one This is the most distinctive of the five because its trigger comes from*outside*the program entirely: the operating system interrupts a running thread with a signal, roughly 100 times a second, and asks “what were you doing right now?” When you call`pprof\.StartCPUProfile`, the runtime sets up the timers it needs to interrupt the running threads — so that time spent burning CPU gets sampled — and starts a background goroutine to collect the results \([`src/runtime/pprof/pprof\.go:888`](https://github.com/golang/go/blob/go1.26.4/src/runtime/pprof/pprof.go#L888)\)\. Hold on to that background goroutine — we’ll see in a moment why it matters\. Arming the timers is the easy part — the interesting bit is what happens when one of them fires\. ### Catching a tick On every timer tick, the program gets**interrupted**: the thread is yanked out of whatever it was doing, mid\-instruction, to run the handler, and then it’s expected to pick up exactly where it left off\. That puts the handler in a very awkward spot — it can’t allocate memory, and it can’t take an ordinary lock \([`src/runtime/proc\.go:5748`](https://github.com/golang/go/blob/go1.26.4/src/runtime/proc.go#L5748)\)\. The handler then captures the call stack with the unwinder, fills in the sample’s values and your**labels**\(the ones you can attach with[`pprof\.Do`](https://pkg.go.dev/runtime/pprof#Do)\), and stores it\. But remember, it can’t allocate or lock — so it needs somewhere to put the sample that demands neither\. That’s exactly what it has: a structure designed for this constraint, a**lock\-free, preallocated ring buffer**\([`src/runtime/profbuf\.go:91`](https://github.com/golang/go/blob/go1.26.4/src/runtime/profbuf.go#L91)\), which can be appended to without allocating and without taking a lock\. Being a ring buffer, though, it’s finite: if it ever fills up faster than it’s emptied, there’s no room for the next sample — so rather than make the interrupted thread wait for space, the buffer simply drops samples and counts how many it lost \(losing a sample is fine; freezing a thread is not\)\. Which sounds like a problem — nobody wants to drop samples on the floor\. And that’s exactly where the background goroutine we started earlier earns its keep\. Its job is to drain the ring buffer as fast as it can — not to write anything out yet, but precisely so the ring rarely fills up and samples aren’t lost\. Each batch it pulls out is folded into an in\-memory map keyed by call stack \(the`profMap`\), where identical stacks are deduplicated and consolidated on the spot, bumping the count on the matching entry\. The output file isn’t touched during the session at all\. Only when you call`StopCPUProfile`, once the last samples have drained, does the builder make a*single*pass over that map and convert it into the pprof format — emitting every sample, location, function, and string in one go — and close the file you open with`go tool pprof`\. Let’s see the whole thing visually:  As we can see here, roughly 100 times a second the interrupted threads drop call stacks into the ring buffer, the background collector drains those and aggregates them into the`profMap`, and eventually — when you stop profiling — it’s all transformed and written out to the pprof file\. We’ve followed a single stack all the way from interrupt to file\. But a stack on its own isn’t a profile — what makes it one is the numbers that ride alongside it\. ### The values each sample carries The whole point of a CPU profile is to know*how much CPU*each stack uses — and that’s where the`value\[\]`slot on each sample comes in\. A CPU sample carries**two**values: a plain**count**of how many times the timer caught this exact stack, and an estimate of**CPU time in nanoseconds**\. The neat part is that the runtime never actually measures time — it only counts ticks\. The nanosecond figure is just the count multiplied by the sampling period, and since the timer fires at 100 Hz, each tick stands for about 10 ms\. So a stack caught 50 times is credited with roughly 500 ms, and “this function used 3\.2 seconds” is really ~320 ticks times 10 ms each \([`src/runtime/pprof/proto\.go:348`](https://github.com/golang/go/blob/go1.26.4/src/runtime/pprof/proto.go#L348)\)\. CPU profiling’s trigger came from*outside*the program\. The next profile’s comes from deep inside it\. ## Heap profiling: the allocator\-sampled one Heap profiling shares its bones with CPU — the unwinder captures a stack, and the result is the same pprof file — but it belongs to the*in\-place*model, not the streaming one\. A trigger fires, the stack is folded into a long\-lived table right there, and the profile is assembled only when you ask for it\. So we’ll focus on what makes it different, and three things do:**where the samples accumulate**,**what triggers one**, and**what data each one collects**\. Let’s take them in that order, starting with the storage \(which the block and mutex profiles share too\)\. ### A table of per\-stack buckets CPU profiling needed that lock\-free ring buffer because it records from inside a signal handler, where it can neither allocate nor take a lock\. Heap profiling records from inside the allocator instead — still a delicate place, but not a signal handler — so it*can*take a lock\. And being able to lock is what lets it do something the CPU handler couldn’t: aggregate on the spot\. So instead of a streaming buffer it uses a hash table of records keyed by the call stack, called**buckets**\([`src/runtime/mprof\.go:75`](https://github.com/golang/go/blob/go1.26.4/src/runtime/mprof.go#L75)\)\. Identical stacks collapse into the same record: a million allocations at the same line in your code don’t create a million entries, they all find the one bucket for that stack and bump its counters\. Let’s see how this changes the lifecycle of collection:  As we can see in the image, allocations are aggregated directly into the bucket table as they happen — no draining goroutine and no Start/Stop window, the bucket table just lives in the runtime and is always accumulating\. Then, whenever the profile is required, it’s exported in a single pass to the pprof file\. That’s where the samples accumulate\. Now, what decides when to take one? ### The trigger: the allocator, sampled The trigger is the[**allocator**](https://internals-for-interns.com/posts/go-memory-allocator/)itself: every allocation goes through runtime code, so there’s nothing to interrupt — the runtime just notices and records the sample right there \([`profilealloc`,`src/runtime/malloc\.go:2238`](https://github.com/golang/go/blob/go1.26.4/src/runtime/malloc.go#L2238)\)\. Recording*every*allocation would be far too expensive, so it only samples some of them — on average one for every`MemProfileRate`bytes allocated, 512 KiB by default\. It doesn’t sample on a fixed 512 KiB boundary, though, because a regular allocation pattern could fall in step with it and keep sampling the exact same spot\. Instead the gap between samples is*randomized*around that average, so over time every allocation site has a fair chance of being caught\. \(`MemProfileRate = 1`samples everything;`0`turns it off\.\) We know when a heap sample gets taken\. Here’s what it records when it does\. ### The data: allocations and what’s still live Where a CPU sample carried a tick count, a heap sample carries**four**numbers, in two pairs: how many objects were allocated at this stack and how many bytes that came to \(`alloc\_objects`/`alloc\_space`\), and how many of those are*still live*— not yet freed — and the bytes they hold \(`inuse\_objects`/`inuse\_space`\)\. The bucket keeps a running tally of allocations and frees, and “in use” is just allocations minus frees\. That last part hides a subtlety\. Allocations are counted the instant they happen, but**frees aren’t**— an object is only counted as freed once the garbage collector sweeps it, much later\. If you just counted both as they arrived, any snapshot would be unbalanced: it would show lots of allocations but very few of their frees, simply because those frees hadn’t been counted yet\. So the runtime deliberately*delays*counting: it arranges for an allocation and its eventual free to fall in the same accounting cycle, and only publishes a cycle once it’s fully settled — the “don’t announce the result until all the votes are counted” trick\. There’s one more adjustment:**scaling**\. Since the profiler records only about one allocation per 512 KiB, each sample it keeps stands in for the many it didn’t\. So before showing you the numbers, it scales each one back up to estimate the real total\. The catch is that bigger objects are more likely to be caught \(they take up more of that 512 KiB budget\), so the runtime scales small, easy\-to\-miss allocations up a lot and large, almost\-always\-caught ones barely at all \([`src/runtime/pprof/protomem\.go:16`](https://github.com/golang/go/blob/go1.26.4/src/runtime/pprof/protomem.go#L16)\)\. Heap profiling watched*memory*\. The next two profiles watch something else entirely — time lost to waiting\. ## Block profiling: the waiting one Like heap profiling, block profiling reuses machinery we’ve already seen — including the same stack\-keyed bucket buffer — so again we’ll focus on what’s different\. The trigger is**contention**: it fires when a goroutine*finishes waiting*on something — a channel, a`select`, a`sync\.Cond`, a semaphore — and the runtime records how long it was blocked, keyed by the stack where it happened \([`blockevent`,`src/runtime/mprof\.go:506`](https://github.com/golang/go/blob/go1.26.4/src/runtime/mprof.go#L506)\)\. In short, it answers “where do my goroutines spend time*waiting*?”, catching the goroutine that*did*the waiting\. Each sample carries two values: a**count**of blocking events \(`contentions`\) and the total**time waited**, in nanoseconds \(`delay`\)\. The interesting part is*how*it decides which events to keep\. Block profiling samples by**duration**: long waits are always recorded, and the shorter a wait is, the more likely it is to be skipped\. So the long blocks worth chasing are never missed, while the swarm of tiny waits gets thinned out\. This selective keeping also means that we need to**scale the numbers back up**\. If a short event was kept with a 1\-in\-20 chance, then for every one we see, roughly 19 like it were thrown away — so the runtime counts the survivor as 20 events, and inflates its delay to match\. Doing that for every kept sample brings the totals back to an estimate of what you’d have measured if nothing had been dropped\. Block profiling tells you a goroutine*waited*, but not what it was stuck behind\. That’s the gap the next profile fills\. ## Mutex profiling: the lock\-contention one Mutex profiling is block profiling’s twin — same bucket buffer, same two values \(`contentions`and`delay`\)\. What differs is*which*contention it watches\. Where block profiling catches the goroutine that waited, mutex profiling catches the lock that*made*it wait: when a goroutine releases a lock that*others were waiting on*, the runtime records how long they waited, keyed by the stack \([`mutexevent`,`src/runtime/mprof\.go:838`](https://github.com/golang/go/blob/go1.26.4/src/runtime/mprof.go#L838)\)\. So it answers “where is my lock contention?” Its**sampling**is simpler than block’s, though\. Instead of weighting by duration, mutex profiling keeps a flat fraction — on average one event in every N, chosen at random regardless of how long the wait was\. The**scaling back up**then mirrors that exactly: since each kept event stands in for N events, the runtime multiplies both its count and its delay by N, so 1\-in\-100 sampling reports totals about 100× the raw captured figures — again, an estimate of the true contention\. All four profiles so far do their recording*while your program runs*, each set off by an event — a timer signal, an allocation, a goroutine finishing its wait\. The fifth records nothing until you ask\. ## Goroutine profiling: the on\-demand snapshot It breaks the mold completely: there’s no background trigger and no sampling\. A goroutine profile is a**snapshot**— a census of every live goroutine and its current stack, taken the instant you ask for it\. The hard part is getting that snapshot without freezing the program\. The obvious approach — stop the world, walk every goroutine, start it again — works, but with hundreds of thousands of goroutines that pause is brutal\. So Go captures almost all the stacks*while the program keeps running*\. During a brief stop\-the\-world it marks every goroutine “not yet recorded,” then lets the program resume while a collector walks the goroutines one by one\. The clever bit: if the scheduler wants to run a goroutine the collector hasn’t reached yet, that goroutine records its own stack first\. So the two cooperate instead of fighting — whoever reaches a goroutine first records it, nobody records it twice, and the one giant pause becomes two tiny ones with the work done concurrently in between \([`src/runtime/mprof\.go:1356`](https://github.com/golang/go/blob/go1.26.4/src/runtime/mprof.go#L1356)\)\. The value is the simplest of all five: each sample carries a single**count**\(`goroutine`\) of how many goroutines are sitting at that exact stack right now\. Since it’s a one\-shot census there’s nothing to sample or scale — identical stacks are just tallied together, so 10,000 goroutines blocked on the same channel collapse into one entry with a count of 10,000\. That’s all five\. Step back, and the shared skeleton is hard to miss\. ## Summary So that’s the five\. They all end up as the same pprof file, and every sample is just a call stack — what really changes is*how*the stack gets recorded: - **CPU**streams: a timer interrupts your threads, and from inside the signal handler \(no allocating, no locking\) each stack goes into a ring buffer that a background goroutine keeps draining\. - **Heap**aggregates in place: the allocator folds each sampled stack straight into a bucket table that lives in the runtime, and the file is written only when you ask\. - **Block**and**mutex**ride on that same bucket table, but fire on waiting instead of allocating — one catches the goroutine that waited, the other the lock that made it wait\. - **Goroutine**doesn’t record anything until you ask; then it just snapshots every live goroutine’s stack at once\. And the lesson underneath it all: profiling looks effortless because the runtime does the awkward work for you — capturing data in places where it can’t even allocate or lock, and lining up events that arrive at completely different moments into one consistent picture\. And with that, I’m going to close this series on the Go runtime — at least for now\. We’ve gone from the scheduler and the memory allocator all the way down to how the runtime watches itself with profiling, and that feels like a good place to pause\. Thanks for reading along; if there’s a corner of the runtime you’d still like to see taken apart, let me know — there’s always more down there to explore\.