Why is Arrays.fill 265 times slower on G1GC?

# [Java][JVM][Tuning][Profiling][G1][JIT] Why is Arrays.fill 265 times slower on G1GC? Source: [https://krzysztofslusarski.github.io/2026/08/19/g1barrier.html](https://krzysztofslusarski.github.io/2026/08/19/g1barrier.html) ## Big fat warning This article shows some JVM tuning using JVM flags\.**You should never use any JVM flags without knowing what consequences they may produce**\. Most of the flags used here are diagnostic ones, used to understand what is going on\. Only one of them is worth considering on production, and I write about it at the very end\. ## The benchmark It started with a benchmark that I expected to be boring\. Fill two arrays with a reference, once on G1GC, once on ParallelGC: ``` package pl.ks.jmh; import org.openjdk.jmh.annotations.*; import java.util.Arrays; import java.util.concurrent.TimeUnit; @State(Scope.Benchmark) public class MyBenchmark { Object[] table = new Object[1024 * 1024]; Object[] table2 = new Object[1024 * 1024]; Object mark = new Object(); @Benchmark @Fork(value = 1, warmups = 1, jvmArgsAppend = "-XX:+UseParallelGC") @OutputTimeUnit(TimeUnit.MICROSECONDS) @Warmup(iterations = 1) @Measurement(iterations = 2) @BenchmarkMode(Mode.AverageTime) public void parallelGC() { Arrays.fill(table, mark); Arrays.fill(table2, mark); } @Benchmark @Fork(value = 1, warmups = 1, jvmArgsAppend = "-XX:+UseG1GC") @OutputTimeUnit(TimeUnit.MICROSECONDS) @Warmup(iterations = 1) @Measurement(iterations = 2) @BenchmarkMode(Mode.AverageTime) public void g1GC() { Arrays.fill(table, mark); Arrays.fill(table2, mark); } } ``` There is**no allocation**in these methods, so there are**no GC cycles at all**during the measurement\. Whatever the difference is, it cannot be “G1 collects garbage slower”\. And yet: ``` Benchmark Mode Cnt Score Units MyBenchmark.g1GC avgt 2 139019,174 us/op MyBenchmark.parallelGC avgt 2 525,537 us/op ``` **139 milliseconds versus 0\.5 millisecond\.**The same Java code, the same JDK, the same machine\. G1 is**265 times slower**\. This article is the story of chasing that number down to a single machine instruction, and then finding out that the instruction was only half of the answer\. ## Environment - **JDK 25\.0\.3**from Amazon \(Corretto\) for everything up to the summary, plus**JDK 26\.0\.2\.1**for the postscript - **Apple M4 Max**, 64 GB RAM, macOS - Default heap: max**16 GB**, which makes G1 pick**8 MB regions**\(this matters a lot later\) - Compressed oops enabled,`UseCompactObjectHeaders`disabled \(both are the defaults here\) - JMH with the`xctraceasm`profiler \(on Linux you would use`perfasm`, it works the same way\) ## Where does the time go? The first question is always the same one: which code is hot? Since there is no GC and no allocation, I ran the benchmark with an assembly\-level profiler: ``` java -jar target/benchmarks.jar -prof xctraceasm ``` On both GCs the answer is the same, 99\.8% of the samples are in the JIT\-compiled code of`Arrays\.fill`\. There is no VM overhead, no GC thread stealing the CPU, no safepoints\. The mutator thread itself is simply running slow code\. So the difference has to be in**what the JIT generated**, and the profiler gives us exactly that\. ## A minimal assembly primer This is an ARM64 machine, so the listings below are ARM64\. If you have never read assembly, here is everything you need for this article\. It is genuinely a short list\. **Registers**are the CPU’s working variables\. Think of them as around 30 pre\-declared local variables that every method shares: NotationMeaning`x0`…`x30`a register used as a full 64\-bit value`w0`…`w30`the**same**register, but only its lower 32 bits`wzr`/`xzr`the “zero register”, reading it always gives`0``x28`HotSpot reserves this one for**the current`Thread\*`**`x27`HotSpot reserves this one for**the base address of the Java heap****Instructions**, only the ones that show up here: InstructionIn Java terms`mov x1, \#5``x1 = 5``add x1, x2, \#16``x1 = x2 \+ 16``lsr x1, x2, \#9``x1 = x2 \>\>\> 9``eor x1, x2, x3``x1 = x2 ^ x3`\(XOR\)`ldr x1, \[x2\]``x1 = memory\[x2\]`\- load 8 bytes`ldrb w1, \[x2\]``w1 = memory\[x2\]`\- load**one byte**\(`b`= byte\)`str w1, \[x2\]``memory\[x2\] = w1`\- store 4 bytes`strb wzr, \[x2\]``memory\[x2\] = 0`\- store one zero byte`cbz x1, LABEL``if \(x1 == 0\) goto LABEL``cbnz x1, LABEL``if \(x1 \!= 0\) goto LABEL``cmp w1, \#2`\+`b\.ne LABEL``if \(w1 \!= 2\) goto LABEL``csel x1, x2, xzr, hs``x1 = cond ? x2 : 0`\- a ternary without a branch`dmb ish`**full memory fence**\- stall until every pending write is visible to all coresTwo addressing shorthands:`\[x5, x14\]`means`memory\[x5 \+ x14\]`, and`add x14, x1, w10, sxtw \#2`means`x14 = x1 \+ \(\(long\) w10\) \* 4`, which is array indexing in a single instruction\. That is the whole vocabulary\. Let’s read some code\. ### One caveat: all of this is ARM64 Every listing in this article is**aarch64**, because that is what I profiled on\. On**x86\-64**the same barrier looks completely different \- different registers, different mnemonics, and the memory fence is not a`dmb ish`at all\. HotSpot builds a`StoreLoad`barrier on x86 out of a locked no\-op on the stack: ``` lock addl $0x0,-0x40(%rsp) ``` Decoding the x86 version instruction by instruction is out of scope here\. What is**not**architecture specific is the logic\.`g1BarrierSetAssembler\_x86\.cpp`and`g1BarrierSetAssembler\_aarch64\.cpp`are mirror images of one another \- the same three early exits, the same young\-card check, and the same`StoreLoad`membar at the top of the slow path\.**The problem described in this article is therefore not an ARM problem\.**Only the listings are ARM\. I have not measured how large the penalty is on x86, and I would not assume it is the same number \- the cost of a fence is a very microarchitecture\-specific thing\. If you do want to reproduce these listings exactly as printed, you need an aarch64 machine\. Those stopped being exotic a while ago: - **AWS**\- Graviton2/3/4: the`m6g/c6g/r6g`,`m7g/c7g/r7g`and`m8g/c8g/r8g`families - **Google Cloud**\-`T2A`\(Ampere Altra\) and`C4A`\(Google Axion\) - **Azure**\- Ampere Altra`Dpsv5/Epsv5`, and Cobalt 100`Dpsv6/Epsv6` - **Oracle Cloud**\- Ampere`A1`and`A2`shapes - any Apple Silicon Mac \- which is what produced every listing below ## What ParallelGC generated Here is the hot loop from the ParallelGC run\. Each Java\-level`table\[i\] = mark`became this: ``` lsr x14, x11, #9 ; x14 = address(table[i]) >>> 9 strb wzr, [x5, x14, lsl #0] ; cardTable[x14] = 0 str w16, [x11] ; table[i] = mark ``` One instruction does the actual work, and two extra instructions do something with a thing called a*card table*\. That is the**write barrier**, and it is worth a short detour to explain what it is for\. ### What is a write barrier? Every generational GC has the same problem\. When it collects only the young generation, it must find all references pointing**into**the young generation, including references held by old objects\. Scanning the entire old generation to find them would defeat the whole point of a young\-only collection\. The solution is to make the application do the bookkeeping\. The heap is divided into**cards**of 512 bytes, and the JVM keeps a byte array with one byte per card \- the**card table**\. Every time your code writes a reference into an object, the JIT emits a couple of extra instructions that mark that object’s card as*dirty*\. At GC time the collector only scans the dirty cards instead of the whole old generation\. That handful of extra instructions after every reference store is the**write barrier**\. You never wrote it, you cannot see it in your code, and it runs on**every single reference assignment your application performs**\. Now the ParallelGC listing reads as plain Java: ``` table[i] = mark; // str w16, [x11] cardTable[address(table[i]) >>> 9] = 0; // 0 means "dirty" ``` The`\>\>\> 9`is dividing by 512, the card size\. Register`x5`holds the base address of the card table, and it is loaded**once before the loop**\- you will not find it recomputed anywhere inside\. There are**no branches**, no conditions, nothing to predict\. ParallelGC does not care whether the object is young or old, whether the value is null, or where it points\. It marks the card and moves on\. There is one more detail in that listing worth remembering for later: the loop counter is incremented by**8**, not by 1\. The JIT**unrolled**the loop eight times, so eight elements are filled per iteration\. ## What G1GC generated Now the same Java statement on G1\. Same JDK, same machine, same`Arrays\.fill`: ``` 0x...bff0: add x14, x1, w10, sxtw #2 ; x14 = arrayBase + i*4 0x...bff4: add x14, x14, #0x10 ; x14 = address of table[i] 0x...bff8: add w10, w10, #1 ; i++ 0x...bffc: ldrb w16, [x28, #0x48] ; load a byte from the current Thread 0x...c000: cbnz w16, #0x...c0a0 ; if it is not 0 -> jump away 0x...c004: subs x17, x2, x27 0x...c008: csel x17, x17, xzr, hs 0x...c00c: lsr x17, x17, #3 ; compress the reference into 4 bytes 0x...c010: str w17, [x14] ; table[i] = mark <-- the actual store 0x...c014: eor x16, x14, x2 0x...c018: lsr x16, x16, #0x17 0x...c01c: cbz x16, #0x...c03c ; exit 1 0x...c020: cbz x2, #0x...c03c ; exit 2 0x...c024: lsr x16, x14, #9 0x...c028: mov x15, #0xdf310000 0x...c02c: add x16, x16, x15 ; x16 = address of the card 0x...c030: ldrb w15, [x16] ; load the card byte 0x...c034: cmp w15, #2 0x...c038: b.ne #0x...c108 ; exit 3 -> the slow path 0x...c03c: cmp w10, w13 0x...c040: b.lt #0x...bff0 ; loop back ``` and somewhere further down, out of the main instruction stream: ``` 0x...c108: dmb ish ; FULL MEMORY FENCE 0x...c10c: ldrb w15, [x16] ; re-load the card byte 0x...c110: cbz w15, #0x...c03c ; already dirty -> back to the loop 0x...c114: strb wzr, [x16] ; mark the card dirty 0x...c118: ldr x15, [x28, #0x50] ; \ 0x...c11c: cbz x15, #0x...c134 ; | 0x...c120: sub x15, x15, #8 ; | push the card into this thread's 0x...c124: str x15, [x28, #0x50] ; | dirty card queue 0x...c128: ldr x8, [x28, #0x58] ; | 0x...c12c: str x16, [x8, x15] ; / 0x...c130: b #0x...c03c ``` Twenty instructions instead of three, four conditional branches, and a**full memory fence**hiding at the end\. Translated into Java, one iteration of`Arrays\.fill`on G1 is: ``` // ---- pre-barrier: is concurrent marking running right now? ---- if (currentThread.satbMarkQueueActive != 0) { slowPathForConcurrentMarking(); // rarely taken } // ---- the actual store ---- table[i] = mark; // ---- post-barrier ---- if (((address(table[i]) ^ mark) >>> 23) == 0) { goto done; } // exit 1: same region? if (mark == null) { goto done; } // exit 2: storing null? cardAddr = cardTableBase + (address(table[i]) >>> 9); if (cardTable[cardAddr] == 2) { goto done; } // exit 3: young card? // ---- slow path ---- fullMemoryFence(); // dmb ish if (cardTable[cardAddr] == 0) { goto done; } // already dirty cardTable[cardAddr] = 0; // mark it dirty enqueueIntoDirtyCardQueue(cardAddr); done: ``` G1 is doing far more work than ParallelGC, but it is also**trying much harder to do nothing**\. It has three separate escape hatches, and in normal code one of them almost always fires, so the barrier costs a handful of cheap instructions and a well\-predicted branch\. That is the design\. The interesting question is: why does none of them fire in my benchmark? ## How to map assembly back to the JDK source code Before answering that, let me show the technique, because it is reusable and it is the part I find people skip\. Those three magic numbers \-`23`,`9`,`2`\- are not arbitrary\. Every single instruction above comes from a specific line of HotSpot C\+\+, and you can walk from one to the other mechanically\. There are four layers\. `\.ad`files are HotSpot’s instruction selection rules: patterns saying “when you see this shape in the compiler’s graph, emit this machine code”\. A**debug build**of the JDK can print which rule fired: ``` java -XX:+UnlockDiagnosticVMOptions \ -XX:CompileCommand=PrintOptoAssembly,java.util.Arrays::fill \ ... ``` ``` 090 B7: # Loop( B7-B7 inner strip mined) Freq: 120,936 090 add R14, R1, R10, I2L #2 094 + add R14, R14, #16 098 + addw R10, R10, #1 09c encode_heap_oop R17, R2 strw R17, [R14] # compressed ptr 0e0 + cmpw R10, R13 0e4 blt B7 // counted loop end ``` Those strings are copied verbatim from a`format`block in`src/hotspot/cpu/aarch64/gc/g1/g1\_aarch64\.ad`, belonging to a rule called**`g1EncodePAndStoreN`**\. That name is the entry point into the source\. Also worth reading is the left column \- those are byte offsets\. The listing jumps from`09c`to`0e0`, which is**68 bytes of machine code that the rule does not print**\. That gap*is*the G1 barrier\. It is inserted after this listing is produced, which is exactly why it is invisible at this level and very much present in the real code\. ``` instruct g1EncodePAndStoreN(...) match(Set mem (StoreN mem (EncodeP src))); ins_encode %{ write_barrier_pre(masm, ...); // everything before the store __ encode_heap_oop($tmp1, $src); __ strw($tmp1$$Register, $mem$$Register); write_barrier_post(masm, ...); // everything after the store %} ``` `write\_barrier\_pre`and`write\_barrier\_post`land in`g1BarrierSetAssembler\_aarch64\.cpp`\. And here is the thing that makes this whole exercise easy:**that file is a literal, one\-line\-per\-instruction transcription**\. There is no optimizer between that C\+\+ and the machine code\. Every`\_\_ something\(\)`emits exactly one instruction, in source order: ``` static void generate_post_barrier_fast_path(MacroAssembler* masm, ...) { // Does store cross heap regions? __ eor(tmp1, store_addr, new_val); // eor x16, x14, x2 __ lsr(tmp1, tmp1, G1HeapRegion::LogOfHRGrainBytes); // lsr x16, x16, #0x17 __ cbz(tmp1, done); // cbz x16, done // Crosses regions, storing null? if (new_val_may_be_null) { __ cbz(new_val, done); // cbz x2, done } // Storing region crossing non-null, is card young? __ lsr(tmp1, store_addr, CardTable::card_shift()); // lsr x16, x14, #9 __ load_byte_map_base(tmp2); // mov x15, #0xdf310000 __ add(tmp1, tmp1, tmp2); // add x16, x16, x15 __ ldrb(tmp2, Address(tmp1)); // ldrb w15, [x16] __ cmpw(tmp2, (int)G1CardTable::g1_young_card_val()); // cmp w15, #2 } static void generate_post_barrier_slow_path(MacroAssembler* masm, ...) { __ membar(Assembler::StoreLoad); // StoreLoad membar // dmb ish __ ldrb(tmp2, Address(tmp1)); // tmp2 := card // ldrb w15, [x16] __ cbzw(tmp2, done); // cbz w15, done // Storing a region crossing, non-null oop, card is clean. // Dirty card and log. STATIC_ASSERT(CardTable::dirty_card_val() == 0); __ strb(zr, Address(tmp1)); // *(card address) := dirty_card_val generate_queue_test_and_insertion(masm, ...); __ b(done); } ``` Read it side by side with the disassembly above\. It is the same code twice\. For comparison, here is the**entire**ParallelGC barrier, from`cardTableBarrierSetAssembler\_aarch64\.cpp`: ``` void CardTableBarrierSetAssembler::store_check(MacroAssembler* masm, Register obj, Address dst) { __ lsr(obj, obj, CardTable::card_shift()); assert(CardTable::dirty_card_val() == 0, "must be"); __ load_byte_map_base(rscratch1); if (UseCondCardMark) { Label L_already_dirty; __ ldrb(rscratch2, Address(obj, rscratch1)); __ cbz(rscratch2, L_already_dirty); __ strb(zr, Address(obj, rscratch1)); __ bind(L_already_dirty); } else { __ strb(zr, Address(obj, rscratch1)); } } ``` `UseCondCardMark`is`false`by default, so we take the`else`branch: shift, and store a zero byte\. That is all of it\. ### Layer 3: the magic numbers Every constant you do not recognise is a JVM constant\. Two of them come from`\-XX:\+PrintFlagsFinal`: ``` java -XX:+UseG1GC -XX:+PrintFlagsFinal -version | grep G1HeapRegionSize ``` ``` size_t G1HeapRegionSize = 8388608 {product} {ergonomic} ``` 8388608 is 223, and`lsr x16, x16, \#0x17`is a shift by 23\. Shifting an address right by 23 turns it into a**region number**, so`eor`followed by`lsr`followed by`cbz`is asking*“are these two addresses in the same 8 MB region?”*\. If they are, G1 does not care about this store at all \- a reference that does not leave its region needs no bookkeeping\. The third constant comes from an enum in`g1CardTable\.hpp`: ``` enum G1CardValues { g1_young_gen = CT_MR_BS_last_reserved << 1, // = 2 ... }; ``` and from`cardTable\.hpp`: ``` enum CardValues { clean_card = (CardValue)-1, dirty_card = 0, CT_MR_BS_last_reserved = 1 }; ``` So`cmp w15, \#2`means*“is this card in a**young**region?”*\. G1 never tracks references pointing into the young generation \(it collects the whole young gen every time anyway\), so a young card is an instant exit\. ### Layer 4: the decisions, not the code Some differences are not in the barrier at all \- they are C2 deciding differently*because*the barrier exists\. I come back to those at the end\. ## Why none of the three exits fire Now we can answer the question\. Look at what the benchmark allocates: ``` Object[] table = new Object[1024 * 1024]; ``` With compressed oops that is 16 bytes of header plus 4 bytes per element: ``` 16 + 4 * 1 048 576 = 4 194 320 bytes ``` G1’s**humongous threshold**is half a region \(from`g1CollectedHeap\.hpp`\): ``` static size_t humongous_threshold_for(size_t region_size) { return (region_size / 2); } ``` With 8 MB regions that is**4 194 304 bytes**\. The array is over the limit**by 16 bytes**\. Two words\. If you do not know what a*humongous object*is, I wrote about them[here](https://krzysztofslusarski.github.io/2020/11/10/humongous.html)\. The part that matters today: humongous objects skip the young generation entirely and are allocated**directly into old generation regions**\. Let’s verify it rather than trust the arithmetic: ``` public class Hum { static Object[] t; public static void main(String[] a) { t = new Object[1024 * 1024]; System.gc(); } } ``` ``` java -XX:+UseG1GC -Xlog:gc+heap=info Hum.java ``` ``` [0,179s][info][gc,heap] GC(0) Humongous regions: 1->1 ``` One humongous region\. Change the array to`new Object\[1024 \* 1024 \- 8\]`and it reports`Humongous regions: 0\-\>0`\. So, walking the three exits with this in hand: 1. The array lives in an**old**region,`mark`lives in eden\. Different regions, so the XOR test fails\. 2. `mark`is not null, so the null test fails\. 3. The array’s cards belong to an old region, so they are never`2`\.**The young test fails\.** Every single one of the two million stores per operation falls through to the slow path, whose first instruction is`dmb ish`\- a full memory fence\. ## Proving it An explanation that only explains is not worth much\. Here is a plain`main`\- no JMH, and thanks to the single\-file source launcher you can run it straight from the`\.java`file, in two seconds\. One thing to explain before the code, because the name of the method is famously misleading:`Integer\.getInteger\("n", 1024\*1024\)`does**not**parse the string`"n"`\. It looks up the**system property**called`n`and returns`1024\*1024`when that property is not set\. So the two knobs of this program are ordinary`\-D`flags: - `\-Dn=<length\>`\- the length of both arrays - `\-Diters=<count\>`\- how many measured repetitions to run ``` public class Fill { static Object[] table = new Object[Integer.getInteger("n", 1024*1024)]; static Object[] table2 = new Object[Integer.getInteger("n", 1024*1024)]; static Object mark = new Object(); static void work() { java.util.Arrays.fill(table, mark); java.util.Arrays.fill(table2, mark); } public static void main(String[] a) { int iters = Integer.getInteger("iters", 40); for (int i = 0; i < 3; i++) { work(); } long t0 = System.nanoTime(); for (int i = 0; i < iters; i++) { work(); } System.out.printf("len=%d %.3f ms/op%n", table.length, (System.nanoTime() - t0) / 1e6 / iters); } } ``` The four rows of the table below are these four runs: ``` java -XX:+UseG1GC -Dn=1048576 -Diters=10 Fill.java # 1024*1024, humongous java -XX:+UseG1GC -Dn=1048568 -Diters=200 Fill.java # eight elements less java -XX:+UseG1GC -XX:G1HeapRegionSize=16m -Dn=1048576 -Diters=200 Fill.java java -XX:+UseParallelGC -Dn=1048576 -Diters=200 Fill.java ``` ConfigurationResultG1,`new Object\[1024\*1024\]`\(humongous\)**124\.651 ms/op**G1,`new Object\[1024\*1024 \- 8\]`\(**8 elements**smaller\)**1\.124 ms/op**G1, humongous array \+`\-XX:G1HeapRegionSize=16m`**0\.982 ms/op**ParallelGC,`new Object\[1024\*1024\]`**0\.501 ms/op****Removing eight array elements is a 110x speedup\.**Raising the region size, which raises the humongous threshold to 8 MB, does the same thing\. Two more runs to confirm*which*of the three exits is the one that matters\. Filling the very same humongous array with different values: Value storedResult`null`**1\.022 ms/op**a young`Object`**109\.480 ms/op**another**humongous, old**object**145\.766 ms/op**Storing`null`takes exit 2 and is fast\. Storing a reference to something in the**old**generation is just as slow as storing a reference to something young \- which rules out the obvious alternative explanation\. This has nothing to do with the young generation, remembered sets, or G1’s refinement threads\. It is purely*“the card is not marked young, so take the slow path”*\. ## It is not about humongous objects At this point it would be easy to file this under “humongous allocation strikes again” and move on\. That would be a mistake\. Nothing in the barrier mentions humongous objects\. It asks one question:**is this card young?**Being humongous is just a very fast way to get an object into old generation\. An ordinary object gets there too \- it just takes a few GC cycles\. Same array, deliberately**8 elements under**the humongous threshold, measured before and after being promoted: ``` public class Promote { static final int N = 1024*1024 - 8; // NOT humongous static Object[] table; static Object mark; static void work() { java.util.Arrays.fill(table, mark); } static void bench(String label, int iters) { for (int i = 0; i < 5; i++) { // warm up, see note below work(); } long t0 = System.nanoTime(); for (int i = 0; i < iters; i++) { work(); } System.out.printf("%-40s %9.3f ms/op%n", label, (System.nanoTime() - t0) / 1e6 / iters); } public static void main(String[] a) { table = new Object[N]; mark = new Object(); bench("array young (fresh eden allocation)", 200); for (int i = 0; i < 4; i++) { // promote it to old generation System.gc(); } mark = new Object(); // fresh young mark bench("same array, after 4x System.gc()", 30); } } ``` A word about that warm\-up loop, because five calls looks small for a JIT benchmark\. It is not the number of**calls**that gets`Arrays\.fill`compiled here, it is the number of**loop iterations inside it**\. C2 is reached through the*backedge*counter, and`Tier4BackEdgeThreshold`is 40000 \- a single call over a million\-element array passes that twenty\-six times over, so the method is already compiled during the very first call\.`\-XX:\+PrintCompilation`shows it happening: ``` 187 1664 4 java.util.Arrays::fill (21 bytes) 187 1665 % 4 java.util.Arrays::fill @ 5 (21 bytes) ``` `4`is C2, and`%`marks an*on\-stack\-replacement*compilation \- the JIT swapped the method out from under the loop while it was still running\. Had we been relying on invocation counts we would have needed thousands of calls, because`Tier4InvocationThreshold`is 5000\. The extra warm\-up calls are there for a different reason, and it is worth knowing about if you reproduce this\. The**first**fill of a freshly created array runs at about`2 ms`, some forty times faster than the steady state, and it takes another two or three calls before the timings settle\. That is not a JIT effect \- it also happens with a humongous array that was never collected, and it survives`\-XX:G1ConcRefinementThreads=0`\. I have not chased that one to the bottom\. On top of it, once we are in the slow regime the per\-call time oscillates between roughly`54 ms`and`100 ms`, so a five\-iteration measurement will cheerfully report a number 40% too low\. Hence five warm\-up calls and thirty measured ones\. ``` array young (fresh eden allocation) 0,501 ms/op same array, after 4x System.gc() 85,023 ms/op ``` **More than a hundred times slower**\- I measure between 130x and 160x across runs\. And this time we do not have to take my word for where the array went, because we can watch it move\. The same run with`\-Xlog:gc,gc\+heap=info`added: ``` [0,292s][info][gc,heap] GC(0) Eden regions: 5->0(6) [0,292s][info][gc,heap] GC(0) Survivor regions: 0->0(0) [0,292s][info][gc,heap] GC(0) Old regions: 1->6 [0,292s][info][gc,heap] GC(0) Humongous regions: 0->0 [0,292s][info][gc ] GC(0) Pause Full (System.gc()) 38M->7M(160M) 3,769ms [0,296s][info][gc,heap] GC(1) Old regions: 6->2 [0,303s][info][gc,heap] GC(2) Old regions: 2->1 [0,308s][info][gc,heap] GC(3) Eden regions: 0->0(1) [0,308s][info][gc,heap] GC(3) Survivor regions: 0->0(0) [0,308s][info][gc,heap] GC(3) Old regions: 1->1 [0,308s][info][gc,heap] GC(3) Humongous regions: 0->0 [0,308s][info][gc ] GC(3) Pause Full (System.gc()) 7M->7M(32M) 5,608ms ``` I trimmed`GC\(1\)`and`GC\(2\)`down to the single line that changes\. And if the`<type\> regions: <from\>\-\><to\>\(<max\>\)`notation is new to you, I explained how to read it in the[humongous article](https://krzysztofslusarski.github.io/2020/11/10/humongous.html)\. The first cycle is the one that does the work\.**`Eden regions: 5\-\>0`**and**`Old regions: 1\-\>6`**\- everything that was sitting in eden, our array included, has been evacuated into the**old generation**\. The remaining cycles only compact what is left, and by`GC\(3\)`the heap has settled at**`Old regions: 1\-\>1`**with an empty eden\. That single 8 MB old region is where our 4 MB array now lives, and it is why the second measurement is a hundred times slower than the first\. Note also**`Humongous regions: 0\-\>0`**in every cycle\. There is no humongous object anywhere in this test\. Nothing changed except which generation the array lives in \- and any long\-lived reference array in your application is already in this state\. ## How bad is it? It depends on the array size This is the part that surprised me, and it is the reason this article is longer than it was supposed to be\. I ran the same young\-versus\-old comparison across a range of array sizes, measuring nanoseconds per stored element: elementsarray sizeyoungoldpenalty2561 KB0\.50 ns0\.92 ns**1\.8x**1 0244 KB0\.50 ns1\.08 ns**2\.2x**4 09616 KB0\.51 ns2\.23 ns**4\.4x**16 38464 KB0\.50 ns7\.24 ns**14x**65 536256 KB0\.48 ns29\.00 ns**60x**262 1441 MB0\.48 ns50\.56 ns**105x**1 048 5684 MB0\.48 ns66\.53 ns**139x**The*code being executed is identical in every row*\. Same instructions, same branches, same fence, same one\-card\-per\-128\-elements ratio of bookkeeping\. Only the array size changes, and the penalty moves by a factor of**70**\. A long\-lived array of a few hundred references costs you essentially nothing\. A multi\-megabyte one costs you 139x\. ## Why the size matters To explain that, I needed a baseline that isolates the barrier from everything else\.`Arrays\.fill\(table, null\)`is perfect for the job: same array, same loop, same memory traffic, but it leaves the barrier through exit 2 and never reaches the fence\. Here it is, measured on old\-generation arrays from 1 KB up to 128 MB: ``` fill(null) 0.48 - 0.51 ns/store, at EVERY size ``` Dead flat\. Well past the 16 MB L2 cache on this machine\. That tells us the raw store loop is**not memory bound at any size**\- the CPU keeps dozens of stores in flight at once, merges them into whole cache lines, and pipelines them out to memory\. A 128 MB fill costs the same per element as a 1 KB fill\. Which is exactly what the fence destroys\. `dmb ish`does not have a fixed price\. It stalls the core until every write already in flight has become visible to all other cores\. With one fence after every store,**only one store can be outstanding at a time**\. All that parallelism is gone, and the per\-store cost becomes the*completion latency*of a single store to whatever level of cache owns that line: - array fits in**L1**\(128 KB on this CPU\) \- a store completes in a few cycles, the fence costs almost nothing:**1\.8x** - array in**L2**\- tens of nanoseconds per store:**60\-105x** - array**beyond cache**\- that latency is already at its maximum So the curve has the shape of the memory hierarchy\. Which also predicts that it should stop somewhere, and it does\. ## Very large arrays: it saturates Continuing past 4 MB \(all of these are humongous, so they are born in old generation \- the GC log confirms 1, 2, 4 and 16 humongous regions respectively\): elementsarray size`fill\(mark\)``fill\(null\)``arraycopy`2 097 1368 MB61\.95 ns0\.48 ns0\.10 ns4 194 27216 MB67\.02 ns0\.49 ns0\.10 ns8 388 54432 MB61\.69 ns0\.49 ns0\.11 ns33 554 176128 MB48\.63 ns0\.50 ns0\.11 nsThe penalty**plateaus at roughly 50\-70 ns per store and goes no further**\. Making the array 32 times bigger does not make each individual store any worse \- 128 MB is actually the mildest of the large sizes\. There is a ceiling on how bad this gets per store\. The total cost of course keeps growing with the array, but only linearly\. And the ParallelGC control at the same sizes: ``` ParallelGC 32 MB fill(mark) 0,24 ns/store fill(null) 0,24 ns/store ParallelGC 128 MB fill(mark) 0,24 ns/store fill(null) 0,24 ns/store ``` Perfectly flat, and identical whether or not a reference is actually being stored\. Two unconditional instructions, no branch, no fence, nothing to be slow about \- even at 128 MB\. ## Two bonus effects While reading the generated code I noticed two more differences that have nothing to do with the fence, and that are worth knowing about because they apply to**every**G1 application, not just this pathological one\. ### The barrier costs you loop unrolling Remember that the ParallelGC loop incremented its counter by 8 and the G1 loop incremented by 1? That is not a coincidence\. In`src/hotspot/share/opto/loopTransform\.cpp`, C2 decides whether a loop is small enough to unroll: ``` uint body_size = _body.size(); for (uint k = 0; k < _body.size(); k++) { Node* n = _body.at(k); if (MemNode::barrier_data(n) != 0) { body_size += BarrierSet::barrier_set()->barrier_set_c2()->estimated_barrier_size(n); } ... } ... if (body_size > (uint)_local_loop_unroll_limit) { return false; // Loop too big. } ``` C2 estimates how large the loop body will become once the barriers are expanded\. For G1 that estimate is \(`g1BarrierSetC2\.cpp`\): ``` uint G1BarrierSetC2::estimated_barrier_size(const Node* node) const { uint8_t barrier_data = MemNode::barrier_data(node); uint nodes = 0; if ((barrier_data & G1C2BarrierPre) != 0) { nodes += 6; } if ((barrier_data & G1C2BarrierPost) != 0) { nodes += 60; } return nodes; } ``` **6 \+ 60 = 66**, and`LoopUnrollLimit`defaults to**60**\. A loop containing even a single G1 reference store is over budget before anything else is counted, so it can**never**be unrolled\. Verified: ``` G1, default LoopUnrollLimit=60 1,077 ms/op G1, -XX:LoopUnrollLimit=200 0,826 ms/op ``` ### The barrier costs you inlining In the profiler output, ParallelGC inlined`Arrays\.fill`into the benchmark method, while G1 compiled it as a separate method\. The reason is in`src/hotspot/share/opto/bytecodeInfo\.cpp`: ``` if (callee_method->has_compiled_code() && callee_method->inline_instructions_size() > InlineSmallCode) { set_msg("already compiled into a big method"); return true; } ``` `InlineSmallCode`is 2500 bytes\. The barrier code bloats the standalone compilation of`Arrays\.fill`past that limit, so C2 refuses to inline it anywhere\.`\-XX:\+PrintInlining`prints that exact message\. Together these two account for the`1\.12 ms/op`versus`0\.50 ms/op`difference between G1 and ParallelGC when the array is**young**\- roughly**2x**, which is the honest, unavoidable price of the G1 barrier\. Everything above that came from the old generation accident\. ## What can we do about it? Everything in this section applies to**JDK 25 and older**\. If you are already on**JDK 26**the barrier itself has been rewritten and the problem largely disappears \- see the postscript at the end of the article\. ### Store null \- it is free ``` OLD array, Arrays.fill(mark) 55.20 ns/store OLD array, Arrays.fill(null) 0.49 ns/store ``` Clearing a long\-lived array \(`Arrays\.fill\(a, null\)`,`ArrayList\.clear\(\)`, invalidating a cache\) never enters the slow path, because of exit 2\. This one is free and you are probably already doing it\. ### Use bulk copies `System\.arraycopy`and`Arrays\.copyOf`on reference arrays**do not use the per\-element barrier at all**\. They use`G1BarrierSetAssembler::gen\_write\_ref\_array\_post\_barrier`, which dirties the whole affected card range once, after the copy is done\. One barrier for the entire array instead of a fence per element\. The classic doubling trick turns a fill into a sequence of copies: ``` static void fill(Object[] a, Object v) { int len = a.length; if (len == 0) { return; } a[0] = v; for (int i = 1; i < len; i += i) { System.arraycopy(a, 0, a, i, Math.min(i, len - i)); } } ``` On a 4 MB old\-generation array: ``` OLD Arrays.fill 55,20 ns/store OLD arraycopy doubling 0,06 ns/store ``` **900 times faster\.**Same result, same array, same GC\. ### Do not create the humongous array in the first place Everything I wrote in the[humongous article](https://krzysztofslusarski.github.io/2020/11/10/humongous.html)still applies, and this is one more reason to care\. If your array is humongous, it is in old generation from birth and it will never get the cheap barrier path, no matter how short\-lived it is\. ### Tuning flags \- carefully One flag genuinely changes this behaviour, and it comes with the usual warning\. `\-XX:G1HeapRegionSize=<size\>`raises the humongous threshold, which can move an array out of old generation\. It also changes how G1 manages the entire heap, so please read the humongous article before touching it \- and remember the maximum is 32 MB\. And one that does**not**work, in case you were about to try it the way I was\.`\-XX:\+UseCompactObjectHeaders`shrinks the array header from 16 bytes to 12, which sounds like exactly enough to get under a threshold we missed by 16 bytes\. It is not, because objects are aligned to 8 bytes and the 4 saved bytes are handed straight back as padding: headerarray sizeafter alignmenthumongous?16 bytes16 \+ 4 194 304 = 4 194 3204 194 320**yes**12 bytes12 \+ 4 194 304 = 4 194 3164 194 320**yes**Both report`Humongous regions: 1\-\>1`and both run at**109 ms/op**\. A good reminder that when a threshold is this close, you have to measure rather than reason about it\. So there is really one flag, and I would not reach for it either\.**The application\-level fixes above are better**, for exactly the reasons I gave at the end of the humongous article: you control your code, and you do not have to understand five years of G1 internals to keep it working\. ## Summary - Every reference assignment in Java runs a**write barrier**that you did not write and cannot see\. - ParallelGC’s barrier is**two unconditional instructions**\. G1’s is around twenty, with four branches and a**full memory fence**on the slow path\. - Every listing here is**aarch64**\. On x86\-64 the instructions look completely different, but the logic is identical \- the same three exits and the same`StoreLoad`membar \- so this is**not an ARM\-only problem**\. I have not measured the magnitude there\. - G1’s barrier has three escape hatches\. In normal code one of them fires\. If the target object is in**old generation**and you store a**non\-null**reference into a**different region**, all three fail and you hit the fence on**every single store**\. - Being a*humongous object*is simply the fastest way into old generation\.**Ordinary promotion does exactly the same thing**\- any long\-lived reference array is already there\. - The size of the penalty is governed by whether the array fits in cache:**under ~64 KB it is a few percent to a few times; above ~4 MB it is 100\-140x**, and then it saturates\. - On top of that, the barrier permanently costs G1 the ability to**unroll**any loop containing a reference store, and can cost**inlining**too\. That is a flat ~2x on this benchmark and it applies everywhere\. - `Arrays\.fill\(a, null\)`and`System\.arraycopy`both bypass the expensive path completely\. ### One more thing: JDK 26 fixes this Everything above was measured on**JDK 25**\. On**JDK 26**the picture changes completely, because of[JEP 522: G1 GC: Improve Throughput by Reducing Synchronization](https://openjdk.org/jeps/522)\. The JEP gives G1 a**second card table**\. Application threads now dirty their own card table with no synchronization with the GC threads, which work on the other one\. That removes the reason the`StoreLoad`fence existed in the first place, and with it goes the dirty card queue insertion, the runtime call, and the entire out\-of\-line slow path\. The barrier collapses into straight\-line code: ``` ldrb w14, [x28, #0x48] ; SATB pre-barrier flag, unchanged cbnz w14, <stub> subs x15, x19, x27 csel x15, x15, xzr, hs lsr x15, x15, #3 str w15, [x12] ; the store eor x14, x12, x19 ; cross-region? lsr x14, x14, #0x17 cbz x14, done cbz x19, done ; storing null? lsr x14, x12, #9 ldr x13, [x28, #0x50] ; card table base - now taken from the thread ldrb w8, [x14, x13] tbz w8, #0, done ; already dirty? strb wzr, [x14, x13] ; dirty it done: ``` No jump to a stub, no queue, and \- the whole point of this article \-**no`dmb`**\. I counted them in the compiled`Arrays\.fill`: one on JDK 25, zero on JDK 26\. The benchmark this article started with: JDK 25JDK 26G1, humongous array**108,811 ms/op****1,121 ms/op**G1, array promoted to old generation**56,711 ms/op****0,527 ms/op**ParallelGC0,532 ms/op0,520 ms/opAnd the size\-dependent penalty, the curve that climbed from 1\.8x to 139x, is simply gone\. On JDK 26 an old generation array costs the same per store at every size I tried \(both columns re\-run back to back for this table, which is why the JDK 25 figures differ a little from the ones earlier in the article\): array sizeJDK 25, oldJDK 26, old4 KB0\.96 ns/store**0\.49 ns/store**64 KB12\.72 ns/store**0\.50 ns/store**1 MB47\.57 ns/store**0\.51 ns/store**4 MB51\.65 ns/store**0\.52 ns/store**So if you have read this far thinking “we run G1 and we do exactly that”, your fix may be an upgrade rather than a code change\. It is on by default and there is no flag to turn on\. Thanks to**Francesco Nigro**for pointing me at JEP 522\. And the meta\-lesson, which is the one I actually care about: the JMH numbers told me*that*G1 was 265 times slower\. The assembly told me*which instruction*\. But only the JDK source told me*why that instruction was there*, and only then could I design the experiments that showed the real answer was not the one I first assumed\. All three layers were necessary, and the HotSpot source is far more readable than its reputation suggests \- the barrier assembler is literally one line of C\+\+ per machine instruction\.