Why aren't my two Cortex-A9 cores cache coherent?

# JP's Website Source: [https://thejpster.org.uk/blog/blog-2026-08-22/](https://thejpster.org.uk/blog/blog-2026-08-22/) ### [JP's Website](https://thejpster.org.uk/) --- ## Why aren't my two Cortex\-A9 cores cache coherent? Posted on**2026\-08\-22** ## Contents - [The Hardware](https://thejpster.org.uk/blog/blog-2026-08-22/#the-hardware) - [Loading Code](https://thejpster.org.uk/blog/blog-2026-08-22/#loading-code) - [The Memory Management Unit](https://thejpster.org.uk/blog/blog-2026-08-22/#the-memory-management-unit) - [Kinds of Memory](https://thejpster.org.uk/blog/blog-2026-08-22/#kinds-of-memory) - [The actual issue](https://thejpster.org.uk/blog/blog-2026-08-22/#the-actual-issue) One of my favourite things to do when I have some down time is to grab a random devkit from my pile of random devkits, and try and write some code that will run on it\. I generally try and avoid existing SDKs, or installing new tools\. I just want to see how little I can write and have something run on the board and have an observable output\. If that goes OK, I might go overboard and start writing some little drivers for a few peripherals here and there\. If it[really gets out of control](https://thejpster.org.uk/blog/blog-2018-02-20/)I might end up having written a whole new Operating System, with VGA output and SD Card support\. But usually I'm OK with`Hello, World`on a UART or a blinking LED\. I had some time off over the past few weeks and, looking for a change from[playing with ARM7](https://git.thejpster.org.uk/thejpster/mcb2130-rs), I picked up my[Terasic DE0\-Nano\-SOC](https://soc.terasic.com/)\. I've had this board for about 7 years but I hadn't really done anything with it\. It's similar to the DE10 Nano board that is used by the[MiSTER retro emulation project](https://misterfpga.org/), but not compatible\. ## The Hardware The Terasic DE0\-Nano\-SOC is a small devkit released in around 2015 from what I can tell\. It has: - An Altera Cyclone\-V System on Chip - 1 GiB DDR3 SDRAM - An RJ45 Ethernet port - A 5V power input \(barrel\-jack\) - A mini\-USB programming port - A micro\-USB port for an on\-board UART to USB Serial convertor - An SD Card slot - Arduino Uno style headers Sadly what it doesn't have is an active cooler, because when running it gets*really*hot\. Like, too hot to touch\. I've wired up a spare 80mm fan which seems to do more than enough to keep it cool\. The Altera Cyclone\-V SoC is a combination of a standard Arm SoC \(the*Hard Processor System*\) and an Altera FPGA \(with 40K logic elements\)\. In particular, I was interested in the two Arm Cortex\-A9 processors\. The SoC has a built\-in Boot ROM that can load a*preloader*from some special sectors on an SD Card\. That*preloader*will run from On\-Chip RAM \(OCRAM\) and initialise the external DDR3 SDRAM, before loading a full copy of U\-Boot from the SD Card into SDRAM and then executing it from there\. I didn't want to muck around with all that, so I just downloaded the disk image from[https://soc\.terasic\.com](https://soc.terasic.com/)and wrote it to a spare card\. It all seemed to boot OK\. By default it boots into a fairly old Linux kernel \(with a root partition stored on the SD Card\), but I deleted the kernel causing it to stop at the U\-Boot prompt\. The on\-board USB to Serial adapter means I can just hook up my PC to the micro\-USB port, and use`minicom`on`/dev/tty\.usbserial\-<something\>`and interact with the board\. The*preloader*prints this: ``` U-Boot SPL 2013.01.01 (Dec 29 2014 - 15:29:15) BOARD : Terasic DE0_Nano_SoC Version-A Board ################################################################################ ##################################= =# ########=-=#######################= -- =# ######- =#######################= ####- =###### =# ######- ###############- -=##=##= ####- ######## =# ####= =##- -### ##= -####### ###### ####- #####- - =# #### ## ### ## -##= -=--=###= ####==# ####- #### =# ######- ###- ##- -### =####= =====###= ####- #### #### =# ######- ### =##### #####= ####-=###= -=###- ####- ##### =# ######- -#= - =## #####= #### -###= #=-=#### ####- ########- ## ####### =##= ## #####= -########= ####### ####- =###### =# #########==######====####==#######= - --- -- ---- =# ##################################= =# ################################################################################ BOARD : Terasic DE0_Nano_SoC Version-A Board CLOCK: EOSC1 clock 25000 KHz CLOCK: EOSC2 clock 25000 KHz CLOCK: F2S_SDR_REF clock 0 KHz CLOCK: F2S_PER_REF clock 0 KHz CLOCK: MPU clock 925 MHz CLOCK: DDR clock 400 MHz CLOCK: UART clock 100000 KHz CLOCK: MMC clock 50000 KHz CLOCK: QSPI clock 3613 KHz SDRAM: Initializing MMR registers SDRAM: Calibrating PHY SEQ.C: Preparing to start memory calibration SEQ.C: CALIBRATION PASSED SDRAM: 1024 MiB ALTERA DWMMC: 0 ``` It's just a small copy of U\-Boot, linked to run from the small amount of On\-Chip RAM \(OCRAM\)\. The full U\-Boot then prints this: ``` U-Boot 2013.01.01 (Dec 30 2014 - 12:07:34) CPU : Altera SOCFPGA Platform BOARD : Terasic DE0_Nano_SoC Version-A Board ################################################################################ ##################################= =# ########=-=#######################= -- =# ######- =#######################= ####- =###### =# ######- ###############- -=##=##= ####- ######## =# ####= =##- -### ##= -####### ###### ####- #####- - =# #### ## ### ## -##= -=--=###= ####==# ####- #### =# ######- ###- ##- -### =####= =====###= ####- #### #### =# ######- ### =##### #####= ####-=###= -=###- ####- ##### =# ######- -#= - =## #####= #### -###= #=-=#### ####- ########- ## ####### =##= ## #####= -########= ####### ####- =###### =# #########==######====####==#######= - --- -- ---- =# ##################################= =# ################################################################################ BOARD : Terasic DE0_Nano_SoC Version-A Board I2C: ready DRAM: 1 GiB MMC: ALTERA DWMMC: 0 In: serial Out: serial Err: serial Skipped ethaddr assignment due to invalid EMAC address in EEPROM Net: mii0 Warning: failed to set MAC address Hit any key to stop autoboot: 0 SOCFPGA_CYCLONE5 # ``` ## Loading Code I've written enough[AArch32](https://github.com/rust-embedded/aarch32/tree/main/examples)[Rust](https://github.com/ferrous-systems/rust-training/tree/main/example-code/qemu-aarch32v8r)[examples](https://github.com/ferrous-systems/s32z2-rust-demo)[by now](https://github.com/ferrous-systems/rcar-v4h-experiments)that it was relatively easy to write another one\. - Start with an empty project - Bring in[`aarch32\-rt`](https://crates.io/crates/aarch32-rt) - Write a`memory\.x`linker script fragment that says where memory is - Write a very basic 16550 UART driver, and point it at the base address of UART0, assuming U\-Boot will have left it enabled and configured at a suitable baud rate The linker script fragment looked like: ``` MEMORY { RAM : ORIGIN = 0x00100000, LENGTH = 1M } REGION_ALIAS("VECTORS", RAM); REGION_ALIAS("CODE", RAM); REGION_ALIAS("DATA", RAM); REGION_ALIAS("STACKS", RAM); PROVIDE(_vector_start = ORIGIN(VECTORS)); PROVIDE(_hyp_stack_size = 16K); PROVIDE(_und_stack_size = 16K); PROVIDE(_svc_stack_size = 16K); PROVIDE(_abt_stack_size = 16K); PROVIDE(_irq_stack_size = 64); PROVIDE(_fiq_stack_size = 64); PROVIDE(_sys_stack_size = 16K); ``` We've got*a lot*of memory to play with, but 1 MiB is more that enough for what we need\. The important thing is that the start address is not`0x0`\. Instead, I chose to set the start address to`0x0010\_0000`to skip the first 1 MiB\. This was to avoid the Boot ROM \(which can be mapped in or out at address`0x0`\), and to avoid whatever RAM U\-Boot was using\. Our crappy UART driver is as simple as: ``` /// This is the same console that U-Boot uses on the DE0-Nano-SOC pub static CONSOLE: Console = Console::new(); /// Represents our standard-output console (on UART0) pub struct Console { _inner: (), } impl Console { const UART0_BASE_THR: *mut u32 = 0xFFC0_2000 as *mut u32; const UART0_BASE_LSR: *mut u32 = 0xFFC0_2014 as *mut u32; const LSR_TX_EMPTY: u32 = 1 << 6; const fn new() -> Console { Console { _inner: () } } /// Wait while the UART is busy fn waitbusy(&self) { loop { let lsr = unsafe { Self::UART0_BASE_LSR.read_volatile() }; if (lsr & Self::LSR_TX_EMPTY) != 0 { break; } } } /// Put a byte into the UART fn putc(&self, byte: u8) { // Safety: This is our UART and buffer overflows are not UB unsafe { Self::UART0_BASE_THR.write_volatile(byte as u32); } } } impl core::fmt::Write for &Console { fn write_str(&mut self, s: &str) -> core::fmt::Result { for b in s.as_bytes() { self.waitbusy(); if cfg!(feature = "console-crlf") { if *b == b'\n' { self.putc(b'\r'); self.waitbusy(); } } self.putc(*b); } Ok(()) } } ``` This is basically lifted from an earlier project I did on the*Pandaboard*\(did I mention that this is not my first Arm dev kit?\), but with the base address changed\. There are a bunch of library crates you could pull in which implement a much better driver, but I'm happy copy\-pasting these few lines because it's easier to hack on it when it's in the tree with the rest of the code\. The main function is a simple: ``` #![no_std] #![no_main] use core::fmt::Write; use hello_de0_nano_soc::CONSOLE; #[aarch32_rt::entry] fn main() -> ! { _ = writeln!(&CONSOLE, "Hello, this is a DE0-Nano-SOC!"); panic!("I am a sample panic!"); } ``` Now, to get the code onto the board we need a file format that U\-Boot likes\. I don't think you can just give it ELF files \(shame, I wrote a lovely[bare\-metal ELF parser](https://crates.io/crates/neotron-loader)so I know it's not that hard\), but you can give it[Motorola S\-Record](https://en.wikipedia.org/wiki/Motorola_S-record)files\. I know, how quaint\. Luckily, LLVM's binutils can do that, which I like to drive using the[`cargo\-binutils`](https://crates.io/crates/cargo-binutils)plugin for`cargo`: ``` cargo objcopy --release -- -O srec ``` Annoyingly this overwrites the ELF file with the hex file \- answers to the usual address if you know a way to get the`objcopy`sub\-command from`cargo\-binutils`to not do that\. But it's fine as I don't need the ELF anyway\. We get U\-Boot to load the file by running`loads`, and sending the file as ASCII through the serial terminal on my Mac\. For reasons I don't fully understand I chose to use`minicom`, and once I'd worked out that "Esc" and then "S" opened the send menu, and "double tap space" enters a directory in the file browser inside`minicom`, we were off\. ## The Memory Management Unit The program runs without issue, but we don't have: - The second core running - The L1 Instruction Cache enabled - The L1 Data Cache enabled - The L2 Cache enabled - The MMU enabled The MMU is the really important one because without it, the processor treats all memory as strongly\-ordered, and unaligned loads or atomic accesses don't work with that kind of memory\. I think technically it's*Undefined Behaviour*to execute Rust code when you're in that state but whatever, let's just get the MMU up and running\. To do this, we need an array of 4,096 Level 1 page table entries, each 32\-bits in length and each representing a 1 MiB portion of the virtual address space\. We can use a Rust const fn to generate that at compile time\. ``` /// Holds an L1 page table with appropriate alignment /// /// You should create a static variable of this type, to represent your page table. #[repr(C)] #[derive(Debug)] pub struct L1Table { /// Our mutable list of MMU table entries /// /// This table is read by the hardware. pub entries: core::cell::UnsafeCell<[L1Section; NUM_L1_PAGE_TABLE_ENTRIES]>, } unsafe impl Sync for L1Table {} /// Our MMU page table #[unsafe(no_mangle)] #[unsafe(link_section = ".pagetable")] pub static MMU_L1_PAGE_TABLE: L1Table = make_mmu_table(); const DDR_ATTRS: SectionAttributes = SectionAttributes { non_global: false, p_bit: false, shareable: true, access: AccessPermissions::FullAccess, memory_attrs: MemoryRegionAttributes::CacheableMemory { inner: CachePolicy::WriteBackWriteAlloc, outer: CachePolicy::NonCacheable, } .as_raw(), domain: u4::new(0b0), execute_never: false, }; const DEVICE_ATTRS: SectionAttributes = SectionAttributes { non_global: false, p_bit: false, shareable: true, access: AccessPermissions::FullAccess, memory_attrs: MemoryRegionAttributes::ShareableDevice.as_raw(), domain: u4::new(0b0), execute_never: false, }; /// The number of bytes in 1 MiB const ONE_MB: u32 = 1024 * 1024; const fn make_mmu_table() -> L1Table { let mut temp: [L1Section; NUM_L1_PAGE_TABLE_ENTRIES] = [L1Section::ZERO; NUM_L1_PAGE_TABLE_ENTRIES]; let mut page = 0; // Map 1024 MiB of DDR SDRAM @ 0x0000_0000 while page < 1024 { let section = L1Section::new_with_addr_and_attrs(0x0000_0000 + (page * ONE_MB), DDR_ATTRS); temp[0x000 + (page as usize)] = section; page += 1; } // Map 256 MiB of system / MPCore peripherals @ 0xF000_0000 page = 0; while page < 256 { let section = L1Section::new_with_addr_and_attrs(0xF000_0000 + (page * ONE_MB), DEVICE_ATTRS); temp[0xF00 + (page as usize)] = section; page += 1; } L1Table { entries: core::cell::UnsafeCell::new(temp), } } ``` Initially I used the[`aarch32\_cpu::mmu::L1Table`](https://docs.rs/aarch32-cpu/latest/armv7a-none-eabi/aarch32_cpu/mmu/struct.L1Table.html)type, but that is marked as requiring alignment to a 1 MiB boundary\. That's fine, except the size of an object must be a multiple of its alignment, so Rust padded the page table out from 16 KiB to 1 MiB\. Which is a problem when I'm loading it over a UART at 115,200 baud\. So instead I made my own type with no alignment requirements, and put it into a special section to ensure it was aligned appropriately\. The Motorola S\-Record format has no problem leaving out the gaps, so the load didn't take too long \(about 5 seconds or so\)\. The MMU mapping is very simple \- a flat 1:1 mapping from Virtual Address to Physical Address, with the bottom 1 GiB being*Inner Cacheable*and the top 256 MiB being*Device Memory*\. Wait, what? ## Kinds of Memory Arm processors understand there are different kinds of memory, and they do this for performance\. Some memory is the kind where if the code writes a 32\-bit value to a specific address, the hardware needs to actually do that write, to that address, exactly once, and not before or after any other write that might occur that address \(or similar addresses\)\. This is important when the address in question is the UART Transmit FIFO register, for example\. Arm call this[*Device Memory*](https://support.arm.com/documentation/ddi0406/c/Application-Level-Architecture/Application-Level-Memory-Model/Memory-types-and-attributes-and-the-memory-order-model/Device-and-Strongly-ordered-memory?lang=en), and it is non\-cacheable and strongly\-ordered\. If all RAM was treated like this, it would kill your performance\. Your RAM is much much much much slower than your processor, and so we need caches \(several levels of caches in fact\) to keep the processor fed with instructions and data as much as possible\. This has been true on desktop PCs since the early 1990s \(the Intel 486 has an on\-die 8 KiB Level 1 cache, for example\), and it's been true for Arm processors since ARM3 came out at around the same time\. Our Arm Cortex\-A9 processor has two interfaces to memory \- one for instructions and one for data \(a so\-called[*Modified Harvard Architecture*](https://en.wikipedia.org/wiki/Modified_Harvard_architecture)design\) \- and so it has two Level 1 caches built into the processor\. They are often called the I Cache and the D cache for short\. We want the processor to use them so we tell the MMU that most of our memory space is[*Normal*](https://support.arm.com/documentation/ddi0406/c/Application-Level-Architecture/Application-Level-Memory-Model/Memory-types-and-attributes-and-the-memory-order-model/Normal-memory)Memory\. That allows it to cache reads and writes, buffer writes \(so they may appear at the caches out\-of\-order or be coalesced into a single larger write\), and generally do things that make CPU go vroom but that we only get away with when address space is backed by RAM and not peripherals pretending to be RAM\. As a side note, when it comes to being cacheable, I see the terms*Inner*and*Outer*a lot\. I believe*Inner*is "other processors in the same cluster" and*Outer*is "things outside that cluster, like other processor clusters, or peripherals that are doing DMA"\. On an Armv7\-A architecture processor you turn on the L1 I Cache by setting the`SCTLR\.I`bit, and you turn on the L1 D Cache by setting the`SCTLR\.C`bit\. There are other bits too, like the`SCTLR\.M`bit to enable the MMU, or the`SCTLR\.Z`bit to turn on branch prediction\. I think technically you are supposed to invalidate the cache contents before you enable the caches too, in case your processor didn't do that automatically when it came out of reset\. With my previous examples that ran on QEMU, this was entirely sufficient\. However, I have a problem: - There's a second processor core I want to enable \(in SMP mode\), - and the two processors have different L1 caches, - and they apparently have a mechanism that allows them to 'see' into each others caches? They also share an L2 cache, which is controlled by a peripheral that is separate from that two Cortex\-A9 cores\. It's a piece of IP Altera bought from Arm called the[Corelink L2C\-310](https://support.arm.com/documentation/ddi0246/h)L2 Cache Controller\. Bringing up the L2C\-310 is a right pain, and I don't actually think I need it enabled to get SMP to work, but I've written a Rust driver for it now\. ## The actual issue Using some changes I put in the latest \(unreleased\) version of`aarch32\-rt`, we can use the library to bring up*secondary*processor cores, after the*primary*code has finished initialising all the global variables and generally decided that it's safe for those*secondary*cores to start running\. The`aarch32\-rt`library knows how to read a special register that identifies each processor core with a number, and how to ensure each processor core gets a unique allocation for each of its seven \(7\!\!\) stacks\. All we need to do is to provide a special function that can park the*secondary*cores until some event occurs \(an interrupt perhaps, or a hardware register changing value\), and a`kmain\_secondary`function for the*secondary*cores to execute\. I have a simple program where Core 0 will: - Configure and Enable the MMU - Invalidate and then enable the L1 I Cache and L1 D Cache - Invalidate and then enable the L2 Cache - Mark itself as being in "SMP" mode in the Auxilliary Control Register \(`ACTLR`\) - Enable the Snoop Control Unit and invalidate the SCU entries for Core 0 - Map the Boot ROM back in at address`0x0` - Do some UART logging whilst it does all that - Talk to the SoC's reset manager peripheral to take Core 1 out of reset - Wait for a global shared`AtomicBool`to read as`true` Core 1 will: - Configure and Enable the MMU \(each processor has its own\) - Invalidate and then enable the L1 I Cache and L1 D Cache - Mark itself as being in "SMP" mode in the Auxilliary Control Register \(`ACTLR`\) - Invalidate the Snoop Control Unit entries for Core 0 - Do some UART logging whilst it does all that - Sets the global shared`AtomicBool`to`true` This works\! Right up to that last bit \- Core 1 is running, and it sets that flag to`true`\. But Core 0 never observes the value changing from`false`to`true`\. The two cores are seeing entirely different values for the same memory address, which is not supposed to happen \(and will wreck pretty much any SMP system\)\. Whilst the two processors do have their own L1 caches, they also share a thing called the[*Snoop Control Unit*](https://support.arm.com/documentation/100486/0401/snoop-control-unit)\(or SCU\)\. This basically is a piece of hardware that watches what goes in and out of each of the processors and invalidates the other processors caches\. This means the processors share a "coherent" view of the world, even if the write from one processor hasn't fully made it out to SDRAM yet \(because it's still in L1 or L2 cache\)\. It should be very simple \- you invalidate the SCU's entries and turn it on, and then magically it all works\. OK, well it's not that simple because[this copy](https://support.arm.com/documentation/ddi0407/i/snoop-control-unit/scu-registers/scu-control-register?lang=en)and[this copy](https://support.arm.com/documentation/100486/0401/snoop-control-unit/scu-registers/scu-control-register?lang=en)of the Arm Cortex\-A9 Technical Reference Manual disagree on whether you should set the`SCU\_CTRL\.EN`bit to 1 to enable it, or to 0 to enable it\. But either way, it's not working \(Linux sets it to`1`, for what it's worth\)\. I've spent countless hours going over Arm documentation for both Armv7\-A architecture, and the Cortex\-A9 in particular\. I've looked at example C code for the Altera Cyclone\-V \(in FreeRTOS and in ThreadX, and in Altera's own driver libraries that both those RTOSes use\)\. I've carefully ported over Altera's drivers for the SCU, the L2C\-310, and the L1 I/D Caches, and the two cores are not cache coherent\. And I've got no idea what I did wrong\. The code is at[https://codeberg\.org/thejpster/hello\-de0\-nano\-soc](https://codeberg.org/thejpster/hello-de0-nano-soc)and if you can find what I did wrong and tell me how fix it, I'll happily write a follow\-up blog post about where my mistake was and how you were smart enough to find what I couldn't\.