How to bring up the Linux Kernel on a new platform

# How to bring up the Linux Kernel on a new platform | WerWolv Source: [https://werwolv.net/posts/linux_bringup/](https://werwolv.net/posts/linux_bringup/) ## [Introduction](https://werwolv.net/posts/linux_bringup/#introduction) Many embedded systems these days run Linux as their operating system\. Generally because it’s a great foundation to run anything you like on top of and because many great developers and manufacturers already took care of writing drivers for all kinds of hardware components\. Even though this makes building the finished project much easier, if you’re dealing with custom hardware, you will most likely still have to bring up the Linux kernel on your own initially\. This post is about just that: Setting up the bare minimal to get Linux running on a new platform\. In this case, the new platform is not a new PCB but a minimal, emulated RISC\-V CPU\. ## [Getting Linux to run](https://werwolv.net/posts/linux_bringup/#getting-linux-to-run) Even though Linux is a very complex piece of software, it doesn’t actually have that many requirements to run\. The only thing it really needs in terms of hardware is: - A CPU with anMMU - Enough RAM to, at least, load the Kernel, a Device Tree and an initramfs\. - A periodically lapsing system clock timer ## [Writing an Emulator](https://werwolv.net/posts/linux_bringup/#writing-an-emulator) These things are what I ended up implementing in about 2000 lines of C\+\+ code\. For the CPU I implemented the[`RV32IMA`](https://werwolv.net/assets/riscv-spec.BGyOTXl_.pdf)instruction set which has all the things Linux needs to run\. For the Hardware, I implemented a simple RAM peripheral, theSBI interfacefor timers and a UART peripheral based on the[`WD8250`](https://werwolv.net/assets/WD8250_82C50_16C450.BwK-Jf5N.pdf)chip which is well supported and is really simple to implement\. Documenting the entire process of writing the emulator is a bit out of scope for this post but the full implementation can be found here:[WerWolv/riscv\-emulator](https://github.com/WerWolv/riscv-emulator)\. #### Real Hardware On a real system, none of this would be necessary of course since the hardware would already be there\. Instead of mapping the peripherals into the CPU’s address space, you would instead read the datasheet of your SoC to figure out where the peripherals are mapped and then go from there to the next step\. Also instead of simply being able to load the Kernel, Device Tree and initramfs into the RAM, you’d generally use a bootloader such as u\-boot to load everything from some non\-volatile storage medium such as a NOR Flash or an eMMC\. Using this emulator, the final hardware definition looks like this: ``` struct Hardware { Hardware() { // Map peripherals into the address space cpu.address_space().map(0x0000'0000, &ram); cpu.address_space().map(0xF400'0000, &uart8250); // Hook up a RISC-V compatible MMU as the address translator cpu.address_space().add_address_translator(&riscv_mmu); // Configure the UART peripheral to print output to the terminal uart8250.output_callback([](std::uint8_t c) { putchar(c); fflush(stdout); }); // Load the Linux Kernel to the start of the RAM ram.write(0x00, LinuxKernel); // Load the Device Tree to the end of the RAM constexpr static auto DeviceTreeBlobLoadAddress = 512_MiB - 1_MiB; ram.write(DeviceTreeBlobLoadAddress, DeviceTreeBlob); // Put the device tree address into register a1 to emulate the way // the bootloader would do it on a real system emulator.cores()[0].a1() = DeviceTreeBlobLoadAddress; // Load the Device Tree in front of the device tree // but at a fixed address we can use later constexpr static auto InitRamFsLoadAddress = 0x1F700000; ram.write(InitRamFsLoadAddress, InitRamFs); // Start up cpu cpu.power_up(); } auto step() -> void { cpu.step(); }private: riscv::Cpu<1> cpu; dev::riscv::MMU<std::uint32_t> riscv_mmu; dev::Ram ram(512_MiB); dev::UART8250 uart8250;}; ``` This, of course, currently does absolutely nothing since there’s no code for it to execute\. To get Linux running, we need to first configure and compile the Linux Kernel for our platform and then populate those`LinuxKernel`,`DeviceTreeBlob`and`InitRamFs`arrays with the correct data\. To compile anything to run on the target, the first thing we need is a toolchain\. Compiling it by hand is generally a bit of a pain since you need to get the correct version of various different tools and libraries and then configure and compile themin the correct order\(sometimes even multiple times with different configurations\!\)\. Luckily for us though, the[crosstool\-ng Project](https://github.com/crosstool-ng/crosstool-ng)exists and handles all of that for us\. With it we can simply configure the toolchain we need and then let it do all the work\. Since we’re planing on building an application that later on runs on 32\-bit RISC\-V Linux, we need a toolchain that targets`riscv32\-unknown\-linux\-gnu`\. This will give us a GCC compiler that can compile C code for our target platform as well as glibc as the C standard library implementation to interface with the Kernel\. ``` .$ ./bootstrapINFO :: *** Generating package version descriptions...INFO :: *** Done!$ makegmake[1]: Entering directory 'crosstool-ng'...GEN ct-ng$ ./ct-ng riscv32-unknown-elf...Now configured for "riscv32-unknown-elf"$ ./ct-ng menuconfig ``` This will open a visual configuration menu where we can change various options for the toolchain\. The crosstool\-ng configuration menuIn here, the main things that needs to be changed here are: - Enable`Target options \-\> Use the MMU` - Set`Operating System \-\> Target OS`to`Linux` - Set`C library \-\> C library`to`glibc` - Enable`C compiler \-\> C\+\+` Save the configuration and then run`\./ct\-ng build`to start the compilation process\. This can very well take up to an hour or more depending on your machine, on mine it took about 11 minutes\. After it’s done, you should have a fully working`riscv32\-unknown\-linux\-gnu`toolchain installed in`~/x\-tools/riscv32\-unknown\-linux\-gnu`that you can use to compile the Linux Kernel and any other software for your target platform\. ## [Compiling the Linux Kernel](https://werwolv.net/posts/linux_bringup/#compiling-the-linux-kernel) Now that we have a toolchain, we can move on to compiling the Linux Kernel\. After cloning the[torvalds/linux](https://github.com/torvalds/linux)repository, the first thing we need to do is to create a configuration for our platform\. Since our platform is quite minimal and doesn’t have any special hardware, we can just use the default RISC\-V configuration as a base and then modify it to fit our needs\. To do that, we can use`ARCH=riscv make menuconfig`to once again open a visual configuration menu\. The main thing we need to change here is to get the Kernel to be built for 32 bit RISC\-V instead of 64 bit\. For that the following options need to be changed: - `Platform type \-\> Allow configurations that result in non\-portable kernels`option needs to be enabled- Now the`Base ISA`option below can be set to`RV32I` - All the`extension support`options need to be disabled as they’re not implemented in the emulator\. - `Boot options \-\> UEFI runtime support`needs to be disabled- This is necessary only so the`Emit compressed instructions when building Linux`option can be disabled as our emulator doesn’t support the compressed instruction set\. - Under`Target options`, the`Generate code for the specific ABI`option needs to be set to`ilp32`and`Architecture Level`to`rv32ima`to prevent the compiler from using compressed instructions when building glibc and our init program later on\. - Under`Kernel Features`, I had to enable the`SBI v0\.1 support`for our Timer and`Build a relocatable kernel`since that allows us to load the Kernel at any address in memory which is quite nice for our use case\. - Finally we can disable`Virtualization`,`Enable loadable module support`,`Networking support`and`Cryptographic API`since we don’t need any of that and it just bloats the Kernel and increases boot time\. - If you like, many of the Device Drivers can also be disabled since we don’t have any of that hardware\. I left everything standard there though since it’s quite a few options to go through Everything else can be left as it is\. After saving the configuration, we can compile the Kernel using the following command: ``` $ make -j \ ARCH=riscv \ CROSS_COMPILE=/path/to/riscv32-unknown-linux-gnu- ``` If the make process asks questions if you want to enable or disable any other options, you can simply answer`n`to all of them\. After the compilation is done, you should have a`arch/riscv/boot/Image`file that contains the compiled Kernel\. This is the final file that can be executed\. ## [Compiling a`init`program](https://werwolv.net/posts/linux_bringup/#compiling-a-init-program) Using the same compiler we can also compile a simple`init`program that will be executed by the Kernel as soon as it has booted and reached userspace\. In a real world embedded system, this would generally be something likebusybox's`init`binary but it can really be any ELF executable\. I simply wrote a small C program that prints some text to the console\. ``` #include <stdlib.h>#include <stdint.h>#include <stdio.h>#include <fcntl.h>#include <unistd.h>int main() { // Open /dev/kmsg to write logs directly to the Kernel Log int kmsg = open("/dev/kmsg", O_WRONLY); if (kmsg == -1) { return EXIT_FAILURE; } // Print some message dprintf(kmsg, "Hello World from \033[32mLinux Userspace\033[0m!\n"); // Loop forever to prevent the init process from exiting while (true) { sleep(1); } // Close the file handle close(kmsg); return EXIT_SUCCESS;} ``` To make creating the initramfs a bit easier, we can compile the`init`program as a statically linked binary\. This way we don’t have to worry about any shared library dependencies and can just put the binary itself into the initramfs without needing to include any additional files\. ``` $ riscv32-unknown-linux-gnu-gcc -static init.c -o init ``` ## [Creating an initramfs](https://werwolv.net/posts/linux_bringup/#creating-an-initramfs) An initramfs or Initial RAM File System is usually just a`cpio`archive that contains all files and directories that should be available to the system after it boots\. It is mounted by the Kernel to`/`and usually contains things like`/bin`,`/dev`and`/proc`but it can really contain anything you want\. You can create a simple initramfs yourself by just creating a directory with the structure you want and then using the`find`and`cpio`commands to create the archive\. However, building a reproducible initramfs this way can be difficult due to permissions, timestamps and other metadata that`cpio`includes in the archive\. Instead, we can use a tool that was built automatically when we compiled the Kernel:`gen\_init\_cpio`\. This tool takes in a textual description of the initramfs structure and content and then generates a`cpio`archive from it\. The description file looks something like this: ``` .# /bindir /bin 755 0 0file /bin/init init 755 0 0# /devdir /dev 755 0 0nod /dev/kmsg 644 0 0 c 1 11nod /dev/initrd 644 0 0 b 1 250nod /dev/console 600 0 0 c 5 1nod /dev/null 666 0 0 c 1 3nod /dev/ram 644 0 0 b 1 0nod /dev/root 644 0 0 b 4 0nod /dev/ttyS0 660 0 0 c 4 64# /procdir /proc 755 0 0 ``` This file can be then finally compiled into a`cpio`archive using the`gen\_init\_cpio`tool like this: ``` $ gen_init_cpio initramfs.txt -i initramfs.cpio ``` ## [Writing a Device Tree](https://werwolv.net/posts/linux_bringup/#writing-a-device-tree) A device tree is a data structure that describes the hardware of the system to the Linux Kernel\. It is used by the Kernel to know which drivers to load and how to configure them\. Let’s start with the bare minimum and build up from there\. ``` /dts-v1/;/ { #address-cells = <0x01>; #size-cells = <0x01>; model = "WerWolv's Emulator"; cpus { #address-cells = <0x01>; #size-cells = <0x00>; timebase-frequency = <65000000>; cpu@0 { device_type = "cpu"; compatible = "riscv"; reg = <0x00>; riscv,isa = "rv32ima"; mmu-type = "riscv,sv32"; interrupt_controller: interrupt-controller { #interrupt-cells = <0x01>; #address-cells = <0x00>; compatible = "riscv,cpu-intc"; interrupt-controller; }; }; }; memory@0 { device_type = "memory"; reg = <0x00 (512 * 1024 * 1024)>; };} ``` This configuration is enough to describe the CPU and its timer to the Kernel\. However, what’s still missing is all the peripherals\. These don’t go under the`cpus`node but instead under the`soc`node ``` / { // ... soc@F0000000 { #address-cells = <0x01>; #size-cells = <0x01>; compatible = "simple-bus"; ranges = <0x00 0xF0000000 0x10000000>; serial@4000000 { compatible = "ns8250", "ns16550"; reg = <0x4000000 0x100000>; interrupt-parent = <&interrupt_controller>; interrupts = <0x01>; no-loopback-test; clock-frequency = <5000000>; }; };}; ``` The final thing that’s needed is configuration for the Kernel itself: ``` / { // ... aliases { serial0 = "/soc@F0000000/serial@4000000"; }; chosen { bootargs = "earlycon console=/dev/ttyS0 rdinit=/bin/init root=/dev/initrd"; stdout-path = "serial0"; linux,initrd-start = <0x1F700000>; linux,initrd-end = <0x1FEFFFFF>; };}; ``` The important bit here are the values in the`chosen`node\. - The`bootargs`property defines thekernel command line\. Here we tell the Kernel that we want to use the`earlycon`feature to get log output as quickly as possible, it will do that by sending all logs to the`serial0`device which is the alias we defined for our UART\. We also tell the Kernel to execute our`/bin/init`program from the initramfs as the init process as soon as it has booted and reached userspace\. - `linux,initrd\-start`and`linux,initrd\-end`define the location of the initramfs in memory\. That’s where we copied the initramfs we built previously to in the emulator\. Now that we have the full Device Tree ready, we can compile it to a Device Tree Blob \(`\.dtb`file\) using the`dtc`compiler\. This is the final file that can be loaded by the Kernel\. ``` $ dtc -I dts -O dtb -o riscv-emulator.dtb riscv-emulator.dts ``` ## [Putting it all together](https://werwolv.net/posts/linux_bringup/#putting-it-all-together) The 3 files we built now can be bundled into the emulator binary and then we can finally run the emulator and see Linux booting\! ``` constexpr std::uint8_t LinuxKernel[] = { #embed "Image"};constexpr std::uint8_t DeviceTreeBlob[] = { #embed "device_tree.dtb"};constexpr std::uint8_t InitRamFs[] = { #embed "initramfs.cpio"}; ``` ``` $ ./command[ 0.000000] Booting Linux on hartid 0[ 0.000000] Linux version 7.0.0-00166-g0f0013213293 (werwolv@fedora) ...[ 0.000000] Machine model: WerWolv's Emulator...[ 54.748992] Run /bin/init as init process[ 54.814251] Hello World from Linux Userspace! ``` ## [Running it in the Browser](https://werwolv.net/posts/linux_bringup/#running-it-in-the-browser) Now that we have a self\-contained executable that we can run to get the Linux Kernel to boot, there’s nothing stopping us from also getting it to run in a Browser\. For that, we can cross\-compile the binary to WebAssembly and call into it from JavaScript\. #### Performance It might take up to a minute or so for the Kernel to reach userspace and execute the`init`program since this is not exactly a greatly optimized emulator in a greatly optimized environment Click on the`Startup Machine`button below to boot up the emulator\. ## [Wrapping up](https://werwolv.net/posts/linux_bringup/#wrapping-up) We now have a working Linux system running on our fantasy machine\! While a few things here might feel a bit contrived, the general process is exactly the same for a real piece of hardware\. Though instead of just choosing where things is mapped into memory, you’d instead consult the datasheet of your SoC\. Also things like the Kernel or the initramfs of course can’t just exist in the memory magically but need to be put there instead by a Bootloader\. On embedded systems this is usually something like[u\-boot](https://github.com/u-boot/u-boot)or a proprietary thing provided by the chip manufacturer\. There, you might need to further bundle up the Kernel, Device Tree and initramfs in some custom file format that the Bootloader can understand then\. Check out the[Android Bootloader Manual](https://source.android.com/docs/core/architecture/bootloader)if you’d like to read about a real\-world example of this\. As always, the code that was used to get all of this running can be found on my GitHub Page: ---