Home made GPU escalated quickly [video]

TL;DR: This is a project to build a custom supercomputing cluster, using 65,536 low-cost RISC-V microcontrollers as "pixels". Through meticulous PCB design, immersion cooling, and automated programming, the result is a GPU-like parallel computing system with thousands of cores. ## Project Origin and Goals The author, building on a previous M.2 cluster using WCH CH32V3 chips and inspired by sponsorship from Altium, planned an even larger cluster. The idea came from the "one MCU per pixel" mad concept, aiming for QVGA resolution (320×240), requiring 65,536 MCUs. They chose the WCH CH570 chip (13 cents, 100 MHz, 12 KB SRAM, with hardware multiplication and native USB), with a total cost of about $8,500. WCH’s CTO Patrick not only guaranteed stock but also offered to provide 10,000 MCUs for free. ## Hardware Design Challenges ### Power Consumption and Cooling Each of the 65,536 MCUs draws about 10 mA, totaling 650 A, and at 3.3 V power consumption is roughly 2 kW (close to a European socket’s 3 kW limit). Adding RGB LEDs and other losses, 99.9% of the power turns into heat, so immersion cooling (in a half-meter acrylic container) was adopted. ### PCB Design and Signal Integrity Each blade contains 1,024 small MCUs (32 rows × 32 columns) and 32 row controllers. A six-layer board was used: layers 2 and 5 are ground planes, all signal lines are surrounded by ground (like a stripline / coplanar waveguide hybrid), and impedance matching was applied (Altium Designer automatically sets trace widths). Power distribution uses the bottom layer (double copper weight), with each column of LEDs and MCUs powered through copper pours. ### Clock and Crystal Issues Initially, the plan was to provide a 32 MHz clock from the row controller’s PWM pin to save on crystal costs, but the clock signal became severely distorted at high speed (peak voltage exceeded the chip’s clock pin upper limit of 1.4 V). Eventually, each MCU had to get its own crystal, requiring a PCB redesign (adding crystal packages into an already crowded layout). After test board validation, the MCUs finally worked stably. ### SPI Communication and Debugging SPI communication uses hardware SPI (no CPU overhead). Each SPI bus hosts 32 MCUs, with row controllers communicating via individual chip selects. Initially, the MOSI signal voltage halved when a chip select was active because MOSI and MISO were swapped in the top-level design (Electrical Engineering’s First Law: RX/TX will always be reversed). After fixing with series resistors, the tested maximum SPI clock was: 15 Mbps is the absolute limit (above that requires overclocking), and 7.5 Mbps is stable with no glitches. ## Manufacturing and Debugging ### Blade Design and Manufacturing The initial full-blade design (over 10,000 vias and 5,000 components) exceeded the manufacturing server’s processing capability, forcing the blade to be split in half. Using smaller PCI Express connectors and re-uploading Gerber files worked. Ordering 4 assembled boards and 1 bare PCB cost €400. When the test boards arrived, SPI communication issues were found and fixed. ### Programming Automation Each MCU has independent programming pads (on the back of the board). A Bumblebee 3D printer (with enough Y‑axis travel) was used together with a homemade probe arm and Python scripts for automated programming. OpenOCD runs as a subprocess, stepping probe positions to scan MCUs. Programming at 15 Mbps was feasible, with occasional missing MCUs requiring retries. ## Final Build and Power ### Backplane Design The backplane uses an FPGA as the bus root node, connected via USB. XT60 connectors were chosen for power (a cost‑effective solution). For the 16‑blade version (16,384 MCUs, QWGA resolution), the backplane design reuses a hierarchical structure. Power comes from a Corsair 3 kW PC PSU, stepped down to 3.3 V through Murata buck converters (95% efficiency, 30 A each), with 20 modules meeting the 80 A requirement. ### Final Assembly When the blades and backplane arrived, a minor mistake was found: programming pads under the fourth‑level MCUs were obscured, but the overall design was successful. PCB precision reached 0.1 mm traces. ## Tool Recommendations Altium Designer’s multi‑channel design (the `repeat` keyword), the PCB design tool’s rule checking, impedance matching, polar grid, and current checking were indispensable for this large project. ## Source https://www.youtube.com/watch?v=qMR3IXF2sWw