Epoll vs. Io_uring in Linux

# epoll vs io_uring in Linux Source: [https://sibexi.co/posts/epoll-vs-io_uring/](https://sibexi.co/posts/epoll-vs-io_uring/) First, I want to tell you how exactly I got to this point and why I started researching different options for handling asynchronous I/O on Linux… Last year, my students and I built a reverse proxy server called TinyGate\. It was super simple, worker\-based, and it basically worked well\. Of course, I didn’t expect it to be very fast, but it was an educational project, and since we’d made a real, kind of production\-ready tool, I was really proud of it\. But my students weren’t as happy as I was \- they wanted to build something genuinely useful, and they were really disappointed that our “product” had strong architectural limits and couldn’t outperform titans like nginx and haproxy\. So they literally forced me to research together how those tools work under the hood and how to handle asynchronous I/O to cut down on the heavy overhead… Long story short, we made a second version of TinyGate, based on epoll\. It still lost to nginx/haproxy in benchmarks, but it had a dramatic performance boost compared to the first version\. But epoll isn’t perfect either \(as I’ll explain below\), and we eventually switched to io\_uring, which led to a full rewrite of our project from scratch, again… So it’s a really interesting topic, and today I’ll share an overview of the two queueing systems Linux gives you for asynchronous I/O\. ## [epoll heritage](https://sibexi.co/posts/epoll-vs-io_uring/#epoll-heritage) When I just started developing for Linux, epoll was a new feature, and basically it had no alternatives\. Everyone used it to manage asynchronous execution \- there was no other choice\. The problem is, epoll relies heavily on syscalls: it tells you when I/O is possible, but you still have to call read\(\)/write\(\) yourself afterward \- that’s two syscalls per I/O event, on top of the one\-time epoll\_ctl registration\. Each of these syscalls causes a context switch between user and kernel mode, which creates HUGE overhead once you’re handling a lot of connections\. But we have a solution\! About 17 years after epoll landed in the Linux kernel \(2002\), io\_uring appeared \(2019\)\! Instead of telling you when I/O is possible, it tells you when I/O is done \- no polling loop, and far less associated syscalls\. The kernel consumes submissions from memory shared between your app and the kernel, and posts completions back into that same shared memory \- both live in ring buffers, hence the name\. The catch: by default you still have to call`io\_uring\_enter\(\)`to tell the kernel “go check the submission queue” \- but one call can submit a whole batch of operations and reap a whole batch of completions, instead of one syscall pair per operation like with epoll \+ read\. If you want close to zero syscalls during steady state, there’s`IORING\_SETUP\_SQPOLL`, which spins up a dedicated kernel thread that polls the submission queue for you \- at the cost of that thread burning CPU \(more on this below\)\. ## [A little comparison](https://sibexi.co/posts/epoll-vs-io_uring/#a-little-comparison) Basic architecture: as I said before, epoll notifies you when I/O is possible, io\_uring notifies you when I/O is done\. Where epoll makes every I/O operation cross the kernel boundary, io\_uring lets you pay a small “setup fee” once \(creating the ring\) plus a per\-batch fee \(the`io\_uring\_enter\(\)`call\) instead of a fee per operation\. So instead of a syscall pair per I/O, you get a syscall per batch of I/Os \- or, with SQPOLL, close to none at all\. As you can see, with a ton of I/O happening, this saves a lot of syscalls\. On relatively new systems where io\_uring is supported \(kernel v5\.1\+, released in 2019\), there’s often not much reason to reach for epoll\. The shift from a readiness model to a completion model is a huge architectural change \- it moves a big part of the work out of your application and into the kernel\. ## [Let’s code\!](https://sibexi.co/posts/epoll-vs-io_uring/#let-s-code) Of course, I won’t leave you without some code showing how both systems work\. We’ll use C\. \(The io\_uring example uses liburing, the userspace helper library \- install it via`liburing\-dev`/`liburing\-devel`, or drop down to the raw`io\_uring\_setup`/`io\_uring\_enter`syscalls if you want zero dependencies\.\) ### [epoll](https://sibexi.co/posts/epoll-vs-io_uring/#epoll) Let’s make a simple example of how epoll works\. We’ll create the instance, register a file descriptor \(stdin, in our case\), and process the incoming event\. As you can see, this example uses three syscalls in total:`epoll\_ctl`\(a one\-time registration\), then`epoll\_wait`and`read`for the event \- so two syscalls per actual I/O event, like I mentioned above\. The code itself is pretty easy to follow\. ### [io\_uring](https://sibexi.co/posts/epoll-vs-io_uring/#io-uring) Now let’s do the same thing with io\_uring instead of epoll\. What can we see here? - Similar instance creation step\. - No epoll\_ctl registration step needed\. - No readiness check needed before submission\. - No separate read\(\) call at completion\. Yeah, io\_uring takes way fewer resources for this \- though, as noted above, there’s still one`io\_uring\_enter\(\)`call hiding inside`io\_uring\_submit\(\)`and`io\_uring\_wait\_cqe\(\)`unless you’re running with SQPOLL\. When you test these examples, keep in mind that for the sake of simplicity, some important parts are missing\. For example, it will block forever if`stdin`never produces any data, and the io\_uring example skips checking for a`NULL`sqe \(which`io\_uring\_get\_sqe\(\)`can return if the submission queue is full\)\. ## [Something additional about io\_uring](https://sibexi.co/posts/epoll-vs-io_uring/#something-additional-about-io-uring) - **Zero\-copy\.**For real zero\-copy I/O, register your buffers ahead of time with`io\_uring\_register\_buffers\(\)`\- this avoids the kernel re\-mapping memory on every single operation\. For network sends specifically, look at`IORING\_OP\_SEND\_ZC`\(kernel 6\.0\+ needed\), which skips copying the buffer into the kernel entirely\. - **SQPOLL uses CPU\.**Even when your queue is empty,`IORING\_SETUP\_SQPOLL`keeps a kernel thread spinning and polling, which burns CPU\. There’s an idle timeout \(`sq\_thread\_idle`\) after which it backs off to sleeping, but it’s not free\. - **Asynchronous error handling\.**Errors come back \(and must be handled\) asynchronously, as part of the`cqe`’s`res`field \- not as a direct return value like a normal synchronous syscall\. ## [Summary](https://sibexi.co/posts/epoll-vs-io_uring/#summary) io\_uring is the new standard for async I/O in the modern Linux world, and honestly, I don’t see much reason to still reach for epoll on a system that has it\. For a from\-scratch project on a modern Linux server, like our TinyGate rewrite, io\_uring is absolutely the way to go\. I’m a die\-hard supporter of dropping support for old systems as soon as it’s reasonable \- if you’re still running a kernel released more than 7 years ago, in my opinion, that’s not a great idea…