Data races and the memory model in Go

# Data races and the memory model in Go Source: [https://func25.dev/posts/go-memory-visibility/](https://func25.dev/posts/go-memory-visibility/) We write a value in one goroutine and read it in another\. Nothing crashes, and the value is there\. It looks like our code works\. We actually got lucky\. Go does not guarantee that one goroutine will see a write made by another unless the program explicitly coordinates their operations\. This post explains what happens on a real machine, why it happens, and what we should use instead\. ## 1\. The program that works Suppose we want to pass a value from one goroutine to another without using a channel\. A simple approach is to store the value in one variable and use a boolean to report when the write is complete\. The reader waits for that boolean before accessing the value: go ``` func main() { var done bool var msg string go func() { msg = "hello" done = true }() for !done { } fmt.Println(msg) } ``` You can run this example[in the Go Playground](https://go.dev/play/p/Q4ti8oRCNCm)\. The Playground prints`hello`and exits, so this execution produces exactly the result we expected\. But when we run this snippet with`go run \-race`, the race detector reports one race involving`done`and another involving`msg`: ``` ================== WARNING: DATA RACE Write at 0x00c0000121cf by goroutine 7: main.main.func1() main.go:11 +0x68 Previous read at 0x00c0000121cf by main goroutine: main.main() main.go:14 +0x110 ================== ================== WARNING: DATA RACE Read at 0x00c000014040 by main goroutine: main.main() main.go:16 +0x128 Previous write at 0x00c000014040 by goroutine 7: main.main.func1() main.go:10 +0x30 ================== hello Found 2 data race(s) exit status 66 ``` So is this snippet safe and valid because we use a`for`loop to check the`done`flag? Let’s consult the Go memory model\. ## 2\. The Go memory model The race detector does not care about the output\. It checks whether 2 goroutines access the same memory concurrently without synchronization and at least 1 access is a write\. The[Go memory model](https://go.dev/ref/mem)answers the next question: “Which write must each read use?” It tells us which behaviors Go guarantees across all runs\. I know this is not obvious, so let’s diagnose the 2 reported races\. ### Race 1: main may not read`true` Let’s put the snippet here so we don’t lose context: go ``` // goroutine A go func() { msg = "hello" done = true }() for !done { } fmt.Println(msg) ``` The first warning is for`done`: ``` WARNING: DATA RACE Write at 0x00c0000121cf by goroutine 7: main.main.func1() main.go:11 +0x68 Previous read at 0x00c0000121cf by main goroutine: main.main() main.go:14 +0x110 ``` `done = true`is a non\-atomic write, and every evaluation of`\!done`contains a non\-atomic read of the same variable\. The program does not require the write to happen before any of those reads\. Since 2 goroutines access the same variable and one access is a write,`done`has a read\-write data race\. The[Go memory model](https://go.dev/ref/mem#badsync)does not guarantee that a write in 1 goroutine becomes visible to another goroutine by itself\. This snippet does not synchronize the*write to`done`*with the*reads of`done`*in`main`, so the loop may continue reading`false`\. That may sound strange because the order looks clear in the Go source code\. In theory, the program may behave as if the generated code reused the value from its first read: SOURCE CODEfor\!done\{\}possible optimizationPOSSIBLE GENERATED FORMcached:=doneif\!cached\{for\{\}\}The source loop and a possible optimized form that reads done onceOf course, the code on the right is only for explanation\. The compiler does not generate that form for this example\. The important point is that the Go source code and generated assembly do not need a one\-to\-one relationship\. There is no guarantee that a write made by the new goroutine will become visible to`main`, so the compiler may reuse a loaded value in a register or a temporary, or arrange instructions in another order, as long as the optimization stays within the Go memory model\. Another question is what happens if the writer goroutine updates`done`while the main goroutine is reading it\. Can the`main`goroutine receive a partially written value? The answer for this specific case is no\. - On arm64, a`bool`uses one byte, - The writer stores that entire byte with one`MOVB`\(move byte\) instruction, - `main`loads the entire byte with one`MOVBU`\(move byte unsigned\) instruction\. Since each instruction accesses the complete one\-byte`bool`, the access is indivisible:`main`cannot receive half of its value\. WRITER GOROUTINEGO SOURCEdone=trueCOMPILES TO ARM64MOVBR0, \(R1\)STOREDONE1 BYTEfalse 0x00true 0x01LOADMAIN GOROUTINEGO SOURCEfor\!done \{\}COMPILES TO ARM64MOVBU\(R1\), R2WHOLE\-BYTE ACCESS, NO PARTIAL VALUEThe writer and main access the complete one\-byte bool on arm64But the same reasoning does not apply to a whole struct, array, or other value made from multiple parts\. Go may read or write a struct one field at a time, an array one element at a time, and a complex number one component at a time\. A value larger than one machine word can combine parts from separate writes\. Strings, slices, and interfaces commonly use multiword internal representations, so a race can create an inconsistent value and may corrupt memory\. For example, consider a 24\-byte struct made from three`uint64`fields: go ``` type State struct { A uint64 B uint64 C uint64 } var state State go func() { state = State{A: 2, B: 2, C: 2} }() snapshot := state ``` The assignment is 1 statement in the source code, but Go may write the 3 fields separately and in any order\. One valid execution writes`C`first\. The racing reader can then read the old values of`A`and`B`together with the new value of`C`: ASSIGNMENTstate = State\{A: 2, B: 2, C: 2\}field writesINITIAL STATEA = 0B = 0C = 0write CDURING WRITEA = 0B = 0C = 2racy readSNAPSHOTA = 0B = 0C = 2NOT ONE COMPLETE STATEA racing struct read combines old and new field valuesThe resulting`snapshot`is`\{A: 0, B: 0, C: 2\}`\. On the 64\-bit machine used for this example, each field contains a complete`uint64`value, but the struct as a whole matches neither the initial`\{A: 0, B: 0, C: 0\}`nor the value`\{A: 2, B: 2, C: 2\}`assigned by the writer\. Note You can reproduce the mixed read[in the Go Playground](https://go.dev/play/p/RZnQcmBdhF9)\. The Playground version intentionally adds a 64 KiB byte array between each pair of fields\. This makes both`state = one`and`s := state`copy 131,096 bytes instead of 24 bytes\. The larger copies take longer, so they are more likely to run at the same time before either one finishes\. The extra bytes only make the mixed result easier to reproduce\. The data race already exists without them\. ### Race 2:`done == true`does not guarantee`msg == "hello"` Assume that the loop reads`true`and exits, exactly as it does in the Playground\. Race 2 asks a separate question: “does`done == true`also guarantee that`fmt\.Println`reads`"hello"`from`msg`?” Let’s put the relevant snippet here so we can follow Race 2 without scrolling back: go ``` var done bool var msg string go func() { msg = "hello" done = true }() for !done { } fmt.Println(msg) ``` The second warning points to`msg`: ``` WARNING: DATA RACE Read at 0x00c000014040 by main goroutine: main.main() main.go:16 +0x128 Previous write at 0x00c000014040 by goroutine 7: main.main.func1() main.go:10 +0x30 ``` Since two goroutines access the same variable and one access is a write,`msg`has a second data race\. The Go source code gives us one order inside each goroutine\. The new goroutine writes`msg`before it writes`done`\.`main`reads`done`before it leaves the loop, then reads`msg`for`fmt\.Println`: new goroutinemain goroutineWRITE msg"hello"same goroutineWRITE donetrueread gets trueREAD donetruesame goroutineREAD msg?Why reading done does not order the write and read of msgIf we read the code from top to bottom, it may seem obvious that`msg`must contain`"hello"`when`main`leaves the loop\. The new goroutine writes`msg`before setting`done`to`true`, and main reads`msg`only after reading`true`from`done`\. But from Go’s point of view, the read of`done`answers only one question: which write supplied the value returned by this read? The value`true`came from`done = true`, but`msg`is a separate memory location with a separate read\. Under the Go memory model, nothing guarantees that when`main`leaves the loop,`msg`contains the value written by the other goroutine\. Go therefore allows this result: ``` read done true read msg "" ``` The Playground prints`"hello"`in this example, so this run does not show us what can go wrong\. Let’s use another snippet where the same missing cross\-goroutine order produces a result that we can reproduce on real hardware\. Two goroutines start at the same time\. Each one writes to its own variable, then reads the other one: go ``` var x, y int var r1, r2 int go func() { // goroutine A x = 1 r1 = y }() go func() { // goroutine B y = 1 r2 = x }() ``` Now let’s think about the possible results: - If goroutine A finishes before goroutine B starts, then`r2`is 1\. - If goroutine B finishes first, then`r1`is 1\. - If they interleave, at least one goroutine sees the other’s write, so either`r1`or`r2`is`1`, or both are\. But whatever order we imagine,**it should be impossible for both`r1`and`r2`to be 0**, because that would require each read to happen before the other goroutine’s write\. Running that experiment 200,000 times on an Apple M\-series machine produced: ``` both goroutines read 0: 2 out of 200000 rounds (0.0010%) ``` The program produced this “impossible” result twice out of 200,000 rounds, which is 0\.0010%\. And nothing is wrong with the hardware\. You can run the same experiment[in the Playground](https://go.dev/play/p/0JqEfwfdSTO)\. But this result needs 2 separate explanations\. First, Go guarantees the result required by the source inside one goroutine\. See this snippet: `a`must contain`1`\. The compiler may combine the 2 statements, replace them with other instructions, or arrange those instructions differently\. But any optimization must still preserve the dependency from`b`to`a`and produce the correct result\. But in our case, the 2 statements in goroutine A are independent: - `r1 = y`does not use`x`, - `x = 1`does not use`y`\. The same is true for goroutine B\. The compiler does not have to preserve their textual order in the binary as long as the generated program still follows the[Go memory model](https://go.dev/ref/mem)\. GO SOURCEx = 1r1 = ystore, then loadCOMPILERmay reorderPOSSIBLE ARM64 ASSEMBLYLDR ySTR xload, then storeThe compiler may emit the independent load before the storeThis kind of compiler reordering could explain the result described in Race 2\. But it did not happen in the experiment above\. The compiler kept the 2 memory instructions in source order in the generated arm64 binary\. That leaves a second explanation: how CPU cores make writes visible to each other\. Even when the machine instructions keep the source order, 1 core does not have to make its write available to the other core before its next read finishes\. No memory barrier enforces that order in this binary\. Core A can read the old value of`y`while core B reads the old value of`x`, so`r1`and`r2`can both be`0`\. CORE ASTR x = 1LDR y = 0NO MEMORY BARRIERA store not visible to B loadB store not visible to A loadboth loads can read 0CORE BSTR y = 1LDR x = 0The writes can reach the other core after both reads finishThis is also why the rate in our run is 0\.0010% and not 50%\. The exact rate depends on goroutine scheduling, core placement, processor memory behavior, and other runtime conditions, so a test may see the result only occasionally\. This kind of flaky and annoying bug is often the hardest to reproduce\. ## 3\. How synchronization makes earlier writes visible For this snippet to be correct, it needs 1 guarantee: the writer must write`"hello"`to`msg`before`main`reads`msg`\. The smallest change that provides this guarantee is to replace the plain`done bool`flag with`done atomic\.Bool`: go ``` var msg string var done atomic.Bool go func() { msg = "hello" done.Store(true) }() for !done.Load() {} fmt.Println(msg) ``` Why does`atomic\.Bool`create this order? `atomic\.Bool`reads and writes its value through operations defined by`sync/atomic`\. Go defines the following rule for atomic operations: ``` If the effect of an atomic operation A is observed by atomic operation B, then A "synchronizes before" B. ``` In our example: 1. `done\.Store\(true\)`is operation`A`\. 2. The`done\.Load\(\)`call that returns`true`is operation`B`\. 3. Operation`B`reads the value written by operation`A`, so`A`*synchronizes before*`B`\. In other words, every write sequenced before`done\.Store\(true\)`in the writer goroutine is guaranteed to be visible to`main`after`done\.Load\(\)`reads that`true`\. This includes`msg = "hello"`, so the later read of`msg`in`fmt\.Println`must see`"hello"`\. If you read the Go runtime and compiler source, you will see names such as`StoreRelease`and`LoadAcquire`\. - *Release*describes the store’s guarantee for writes completed before it\. - *Acquire*describes the load’s guarantee for reads that run after it\. WRITERMAINwrite msgsource orderdone\.Store\(true\)RELEASE SIDEatomic orderdone\.Load\(\) == trueACQUIRE SIDEsource orderread msgAn atomic store and load connect the write of msg to its later read### A mutex provides exclusion and visibility You are probably familiar with`sync\.Mutex`and its main job: allowing only one goroutine at a time to access protected state\. go ``` type Counter struct { mu sync.Mutex n int } func (c *Counter) Add() { c.mu.Lock() defer c.mu.Unlock() c.n++ } func (c *Counter) Value() int { c.mu.Lock() defer c.mu.Unlock() return c.n } ``` But a mutex also makes writes from one lock holder visible to the next lock holder\. This`Counter`uses both guarantees\. WRITERREADERwrite nsource orderUnlockmutex ruleLock returnssource orderread nUnlock connects a protected write to a later read after LockGo guarantees that a call to`Unlock`*synchronizes before*a later call to`Lock`returns\. In other words, after`Add`writes`n`and unlocks`mu`, a`Value`call that locks`mu`later is guaranteed to see that write\. ### WaitGroup waits for task completion The original snippet only needs`main`to wait for one task\.`sync\.WaitGroup`provides that relationship without a busy loop: go ``` func main() { var msg string var tasks sync.WaitGroup tasks.Go(func() { msg = "hello" }) tasks.Wait() fmt.Println(msg) } ``` `tasks\.Go`starts the function and tracks the task\.`tasks\.Wait\(\)`does not return until the function has completed\. Go also guarantees that writes made by the function before it returns are visible after`Wait\(\)`returns\.`fmt\.Println`therefore reads`"hello"`\. Unlike the earlier`for \!done`loop,`main`blocks inside`Wait\(\)`instead of repeatedly checking a value and using CPU while the task is still running\. ### Channel close can signal completion The same program can use a channel when one goroutine needs to announce an event: go ``` func main() { var msg string ready := make(chan struct{}) go func() { msg = "hello" close(ready) }() <-ready fmt.Println(msg) } ``` Of course, calling an arbitrary function such as`abc\(\)`does not by itself create a guarantee between goroutines\. The function would need to use a synchronization operation internally\.`close\(ready\)`provides such an operation because Go connects it to a receive that completes after`ready`is closed\. WRITERMAINmsg = "hello"source orderclose\(ready\)channel rulereceive readysource orderread msgA channel connects the write of msg to the later readSo Go guarantees that closing a channel synchronizes before a receive that returns because the channel is closed\. The goroutine writes`msg`before`close\(ready\)`, and`main`reads`msg`after`<\-ready`, so`fmt\.Println`is guaranteed to read`"hello"`\. ### sync\.Once makes initialization visible Sometimes many goroutines need the same value, but the code that initializes that value must run only once\.`sync\.Once`provides that guarantee: go ``` var once sync.Once var message string func getMessage() string { once.Do(func() { message = "hello" }) return message } ``` Only one call to`Do`runs the function\. Other calls wait for that function to return\. Go guarantees that the function’s return synchronizes before every`Do`call returns, so every caller can safely read`message`after`once\.Do`returns\. ### Atomic operations follow one global order The earlier`x`and`y`experiment can use atomic integers: go ``` var x, y atomic.Int32 var r1, r2 int32 var tasks sync.WaitGroup tasks.Go(func() { x.Store(1) r1 = y.Load() }) tasks.Go(func() { y.Store(1) r2 = x.Load() }) tasks.Wait() ``` The atomic version produces no rounds in which both reads return`0`: ``` with sync/atomic, both read 0: 0 out of 200000 rounds ``` Go requires all atomic operations to behave as if they ran in one global order\. In our example,`x\.Store`,`y\.Load`,`y\.Store`, and`x\.Load`must all belong to that same order\. Both goroutines use this order when deciding which value each`Load`returns\. Another execution may use a different order, but the two goroutines still cannot use separate orders\. In other words, only two cases are possible when we compare the two stores: x\.Store BEFORE y\.Storey\.Store BEFORE x\.Store1x\.Store\(1\)global order2y\.Store\(1\)B source order3x\.Load\(\) = 1r2 MUST BE 11y\.Store\(1\)global order2x\.Store\(1\)A source order3y\.Load\(\) = 1r1 MUST BE 1The first atomic store forces one of the later loads to read 1At least one load must therefore return`1`\. The result`r1 == 0`and`r2 == 0`is no longer possible here\. Atomic operations are useful when one shared value can be updated independently, such as a counter or a`ready`flag\. They cannot combine several updates into one operation\. For example, if`balance`and`version`must always change together, another goroutine could read between two atomic stores and see the new`balance`with the old`version`\. A mutex can protect both fields while they are updated and read\. Note The`WaitGroup`has a separate job in this snippet\.`main`reads`r1`and`r2`only after both task functions return\. ## 4\. Testing the synchronization The question in every example above is not whether the program returned the expected value\. The question is whether Go guarantees that the reader sees the writer’s work\. The race detector can report executions that lack this guarantee: The detector tracks memory accesses made by the running application\. It reports a race when concurrent goroutines access the same location, at least one access is a write, and no valid synchronization connects those accesses\. But this is a dynamic check\. An unsafe function that never runs during a test cannot produce a report\. Even a function that does run may require a particular execution path or schedule before both conflicting accesses occur\. A clean result should therefore be read as “no race was reported in these executions,” not “the program contains no races\.” The source still needs a clear explanation of why each shared read is safe\. In the examples above, that explanation comes from atomics, mutexes, task completion, channels, or one\-time initialization\. ## Source references - [The Go memory model](https://go.dev/ref/mem) - [`sync`package documentation](https://pkg.go.dev/sync) - [`sync/atomic`package documentation](https://pkg.go.dev/sync/atomic) - [`src/sync/mutex\.go`](https://go.dev/src/sync/mutex.go) - [The race detector](https://go.dev/doc/articles/race_detector) - [Memory access ordering in the Arm architecture](https://developer.arm.com/community/arm-community-blogs/b/architectures-and-processors-blog/posts/memory-access-ordering-part-3---memory-access-ordering-in-the-arm-architecture)