PS3 emulation is fast on ARM now

TL;DR: RPCS3’s ARM port got a 60% performance uplift and 25% lower power draw thanks to fixing a busy-wait timer bug, replacing x86 pause with ARM ISB, and reworking LLVM code generation for dozens of SPU instructions. ## The Road to Faster PS3 Emulation on ARM Six months ago, RPCS3 was running at a crawl on ARM. Today, the emulator is able to emulate the PlayStation 3 60% faster while drawing 25% less power. The change didn’t come from a mysterious rewrite. It came from reading the 17,000-page ARM architecture reference manual, finding low-hanging fruit, and fixing the way LLVM generates code for ARM. The work was driven by an Ayn Odin 2, an Android handheld that is basically a smartphone with controls bolted on. The goal wasn’t to use it as an Android device; it was to install Linux and use it as a cheap ARM test machine for upstream RPCS3. ## The Busy Wait Bug The first major problem appeared in a tiny four-line function: `busy_wait`. The busy wait exists to improve performance in multi-threaded code. If one thread is repeatedly checking whether another thread changed a variable, a busy wait lets the monitoring thread see the change as fast as possible. But without throttling, it also produces extra heat, burns more power, and adds pressure on shared resources like cache and memory. In the past, adding a busy wait to a specific place in RPCS3 tripled performance under particular circumstances. So the function matters. But on ARM, it was far slower than intended. ### How busy wait works on x86 On x86, the code reads the hardware timer, adds 3,000 to get a target wait time, then loops until the hardware timer exceeds that target. Inside the loop it calls the `pause` instruction, which throttles the hardware and saves power. The value 3,000 works out to about a 1 microsecond wait on most x86 devices, because the hardware timer typically runs between 2 and 4 GHz. It isn’t a precise wait, just a simple way to save power. When porting to ARM, the straightforward translation was used: - x86 hardware timer read → ARM hardware timer read - x86 `pause` instruction → ARM `yield` instruction But ARM’s hardware timer on the test device runs at only 19 MHz, about 150 times slower than expected. Instead of waiting 1 microsecond, each busy wait call waited 150 microseconds. The fix was to scale the wait time by the actual timer frequency. ARM hardware timers vary wildly, typically between 10 MHz and 1 GHz, so no single constant works. With that scaling, the average performance boost was 25%, and power draw dropped by 10%. One user on an ARM MacBook went from 5–10 FPS in Skylanders to a locked 30 FPS. ## Why `yield` Was the Wrong Replacement for `pause` The busy wait also had a second issue. The ARM `yield` instruction seems like the natural replacement for x86 `pause`, but it doesn’t do what you might expect. According to the ARM manual, `yield` is a hint instruction intended for multithreading. It can indicate to the hardware that the thread is doing something like a spin lock and could be swapped out. That sounds perfect. But in another section of the manual, the behavior is more specific: `yield` only yields the current thread if the hardware supports SMT. Nearly all consumer ARM machines use traditional SMP cores, not SMT. On SMP systems, the manual says `yield` can lower the priority of the snoop bus, but if the busy wait loop isn’t reading memory, that doesn’t help either. ARM published a blog post explaining how to port x86 `pause`-style busy waits to ARM. It calls out `yield` as a complete no-op on systems without SMT. Instead, the recommended replacement is the instruction synchronization barrier instruction, `ISB`. ISB restarts instruction fetch, which burns a little more power than `pause` does, but it still saves more power in this loop than `yield`. The description of `yield` in the manual could probably be adjusted to avoid this confusion. Many large companies have run into the same issue; searching for pull requests that mention `yield` or `ISB` turns up fixes from all sorts of multi-trillion-dollar corporations. With ISB, the CPU checks the wait condition only once per several nanoseconds, rather than multiple times per nanosecond. ## The Recompiler: LLVM’s ARM Code Generation The SPU recompiler is typically the heaviest part of PS3 emulation on x86. RPCS3 uses LLVM to translate PlayStation 3 game code to x86 or ARM: PS3 assembly → LLVM IR → native code for the host machine. That’s similar to how games like Unleash Recompiled work, except Unleash translates to C, which is then compiled using LLVM’s clang. Using LLVM means RPCS3 can take advantage of LLVM’s optimizations, but LLVM isn’t perfect at optimizing. In some cases, ARM had better code generation than x86 without any intervention. ### Instructions where ARM beat x86 - **abs db**: Takes the absolute difference between two numbers. For example, the absolute difference between 3 and 7 is 4. x86 needs a three-instruction sequence because its absolute-difference instructions also sum horizontally. ARM has a single instruction that matches PS3 behavior. - **count b**: Counts the number of bits set in each byte. x86 can do it in a single instruction only with AVX-512; otherwise it needs an awful long sequence. ARM has a single instruction. - **CLZ**: Counts leading zeros in each byte until a set bit is found. Same story: ARM has one instruction, x86 needs AVX-512 for a single-instruction solution. - **CLGT**: An unsigned comparison instruction. x86 needs four instructions without AVX-512, two with AVX-512 (because the comparison writes to mask registers and must be moved back). ARM does it in one instruction. - **SELB**: A select instruction. ARM’s BSL is a single instruction. x86 needs three bitwise instructions; with AVX-512, VPTERNLOG can do the job in one. So ARM can be more optimal in some cases, but that gap is usually closed on x86 machines with AVX-512. ## Fixing Bad ARM Code Generation There were dozens of instructions where ARM code gen was worse than x86. The fixes covered a lot of ground. ### SHUFB: The Big One SHUFB is one of the most complex and most common PS3 instructions. The first bad code generation was actually due to a shortcut in the ARM port: instead of rewriting SHUFB emulation to use ARM shuffles, the port simply emulated x86’s PSHUFB. That worked, but it was inefficient. The unoptimized version took 10 ARM instructions: 1. XOR the bottom four bits of the indices, because SHUFB uses big-endian byte ordering while ARM TBL uses little-endian. 2. AND with hex 8F to mask out everything except the bottom four bits and the most significant bit. 3. Feed the indices into TBL twice, because TBL zeroes the result if any bit outside the bottom four bits is set; the masking leaves bit 7 set so that the special case value can be produced. 4. Combine the two TBL results based on bit 4 using a shift, compare, and select. 5. Generate the three special constants (zero, hex FF, hex 80) using another TBL. 6. OR the generated constants into the shuffle result. The optimization uses TBX instead of TBL. TBX, when given out-of-range indices, takes the value from the destination register instead of filling with zero. That eliminates the final OR. TBL and TBX can also take multiple input registers, so a single TBX with two input registers replaces the entire five-instruction combine sequence. That brings SHUFB down to five instructions. It could be four with the BCAX instruction, which performs both an AND and an XOR. BCAX is part of the SHA-3 crypto extension, but it’s full of bitwise instructions useful for general-purpose code. LLVM pattern-matches BCAX only when inputs aren’t constants, which wasn’t helpful here. An LLVM issue was opened. Even with five instructions, loops can be optimized further: if SHUFB is used inside a loop, all the special-case setup can be hoisted out, leaving just one TBX2 instruction inside the loop. The SHUFB optimization alone gave an 8% performance uplift. But half the games refused to boot. Using TBX/TBL with two input vectors requires the two input vectors to be adjacent in registers, which places heavy constraints on LLVM’s register allocator. With RPCS3’s LLVM configuration, it would give up and crash. The workaround: catch the crash when LLVM fails to compile a block with two-source TBX/TBL, then retry with a single-source version. It’s an ugly solution, but when 10,000 recompiled blocks succeed with the two-source version and only three need the fallback, it keeps the 8% speedup and preserves compatibility. ## ARM Is Not a Simple Instruction Set People often describe ARM as a simpler instruction set compared to x86. That’s an old story, from the days of the original ARM chips in the 1980s. Modern ARM is a massive architecture. The manual alone is 17,000 pages. As an example, three ARM instructions have separate encodings but identical behavior on ARM’s latest consumer-focused cores. The ARM instruction `US dot` behaves identically to `VPDPBUS D`. Having simpler mnemonics doesn’t make it a simpler instruction set. Modern ARM has more in common with modern x86 than with the ARM machines of the ’80s. The silver lining: many x86 optimizations can be ported directly to ARM. You don’t need a complete rewrite or a brand new emulator to take advantage of ARM hardware. It’s often just putting the square in the square hole. ## Dot Product Optimizations ### SUMB The SPU instruction `sum B` sums two vectors horizontally and packs the result. Both x86 and ARM have dot product instructions that first multiply bytes vertically, then sum horizontally. By using dot product instructions with a multiplicand of one, SUMB can be emulated the same way on both architectures. The ARM port was as simple as dropping in the dot product instruction where x86 used `VPDPBUS D`. ### Gather bits (GBB, GBH, GB) The gather bit family takes the least significant bit from each byte, packs them together, and zeroes the rest of the register. The dot product trick here uses powers of two instead of multiplying by one. Each lane multiplies by a different power of two, masks off everything but the LSB, and the dot product effectively gathers bits. Since each dot product result contains four bits, the results need to be shuffled into the correct lanes, then horizontally summed with an add-pairwise instruction. Most gather bit uses follow a comparison instruction. Comparisons produce -1 (all bits set) or 0. By multiplying with a negative power of two, you can skip isolating the least significant bit, saving another instruction. ARM’s i8mm extension goes further than x86. The `UMMLA` and `SMMLA` instructions split each half of both input vectors into bytes and multiply-accumulate them in a way that maps perfectly to the common case of GBH and GBB. With the sign trick, only two instructions are needed for GBH and GBB, and GB already had a two-instruction solution. ## Multiplications The SPU integer multiplies are all 16-bit × 16-bit producing 32-bit results. The straightforward emulation masks bits and uses 32-bit multiplies. ARM has widening multiply instructions that do 16-bit × 16-bit and produce 32-bit results, but they expect inputs packed differently than the SPU does. Still, they save instructions. For example, SPU `MPY` (signed multiplication) normally requires shifting to sign-extend the bottom 16 bits into 32 bits before multiplying. With ARM widening multiplies, you can use `XTN` to pack the low 16 bits of each lane into the low 64 bits of the vector, do that for both inputs, then use `SMLAL` for the widening multiply. That drops the cost from five instructions down to three. Similar savings apply across the other SPU multiply instructions. SVE2 versions of widening multiplies are even more useful, but the SVE instruction set has caveats that need to be addressed separately. ## Floating-Point and Other LLVM Failures ### FCGT FCGT is a floating-point comparison instruction. On the PS3’s SPUs, NaN and infinity don’t exist, so the emulation has to handle that. LLVM’s ARM output for this was bizarre. Nothing seemed to massage the code into a reasonable form except inline assembly for the BSL instruction that was expected to be there. That replaced 15 instructions with 7. Inline assembly is only a workaround because it inhibits further LLVM optimizations; an upstream LLVM issue was opened. ### FSM FSM is the reverse of the gather bits family. It takes packed bits and expands them into byte, 16-bit, or 32-bit elements of all zeros or all ones. x86 does this in three instructions: broadcast the packed value, AND with a constantto mask the appropriate bit, then compare. ARM’s LLVM output was catastrophic: it scalarized the vector operation, using SBFX to grab one bit at a time, extend it, insert it into a vector, and repeat. The workaround was to write LLVM IR in a more idiomatic way, essentially matching what the x86 code does. That produced a two-instruction ARM solution using `CM TST`, which combines AND and compare. That’s faster than the x86 version. ### SHL and ROTM SPU shift instructions mask off a certain number of bits before shifting. For example, shift-left-word masks everything except the bottom six bits; if bit 5 is set, the result is zero. x86 does the same masking via one AND instruction, then a native shift. That’s two instructions. ARM’s native shifts work the same way, with one exception: `USHL` shifts left when the shift count is positive and right when it’s negative. That doesn’t matter for emulating PS3 SHL. Yet LLVM compiled the same code to four instructions on ARM because the LLVM IR for shifts leaves part of the implementation undefined, producing “poison values” instead of zeros in the upper bits. The optimizers were supposed to recognize that native shifts on the target machine shift in zeros, but they failed to do so on ARM. The workaround uses an intrinsic for `USH`, saving two instructions per shift instruction. ROTM still needs one extra instruction over x86 because the shift count must be negated for a rotate-right. ## The Snapdragon 8 Gen 2’s Five Different Cores The Odin 2 contains a Snapdragon 8 Gen 2, a chip that’s unique in having four different core types: - **Cortex-X3**: the biggest and fastest core, high performance at the cost of die area and power. - **Two Cortex-A715s**: balanced performance, power, and size. - **Two Cortex-A710s**: essentially the previous generation of A715s, included because the X3 and A715s dropped support for 32-bit ARM instructions. - **Three Cortex-A510s**: slow, low-power small cores. If you count the clustered A510s and the unclustered A510s separately, it’s really five different core types in an eight-core machine. The A510s are ill-suited for PS3 emulation. Two of them share a single 128-bit vector unit, so even if every PS3 instruction were emulated with one host ARM instruction, they could still be significantly slower than the original PS3. The third A510 has exclusive access to a 64-bit vector unit, so 128-bit instructions run at half speed. The A715 and A710 share a lineage, crystallized around the A78. They also share an unusual behavior: they can do more vectorized loads than vectorized operations per clock. Typical CPUs have at least as many vector operations as loads per clock: - A random Intel CPU: 2 × 256-bit loads and 3 × 256-bit vector ops per clock. - A newer Intel CPU: 3 × 256-bit loads and 3 × 256-bit vector ops per clock. - A latest AMD core: 2 × 512-bit loads and 4 × 512-bit vector ops per clock. - Cortex-X3