The epoll uaf https://t.co/grDutada6V

The epoll uaf

Source: https://guysrd.github.io/epoll-uaf A couple of weeks ago Nicholas Carlini burned an epoll uaf race infs/eventpoll\.c.Commit 07712db80857changed akfree\(\)tokfree\_rcu\(\). The commit message says: “eventpoll: defer struct eventpoll free to RCU grace period.”

That one call fixed a uaf that had been reachable from any unprivileged process for a few years on any Linux / Android running a 6.6 and above kernel with the affected optimization. This post is the story about the bug itself, what it gives you and my (failed) attepmts at exploiting this on a real modern device. I spent a bit on a Pixel 10 working on this bug and in the process learned more about CFS vruntime tricks, SLUB internals, and the ARM64 memory model than I probably needed to.

epoll in 2 seconds

If you’ve run a Linux server you’ve used epoll indirectly. It’s the kernel’s scalable I/O notification mechanism the thing that lets nginx watch tens of thousands of sockets without blocking a thread per connection. Three syscalls:epoll\_create\(\)makes an instance,epoll\_ctl\(\)adds or removes watched file descriptors,epoll\_wait\(\)blocks until something happens.

Linux manages everything as file, so epoll fd is itself a file descriptor. You can add an epoll to another epoll. This creates a directed graph of instances watching instances, and the kernel has validation code insideepoll\_ctl\(ADD\)that walks this graph to check for cycles and depth violations, that validation code is where the bug lives.

epoll has a history of cveshistory ofCVEshowever, their exploitation is not documented and is very scarce.

Structures

struct eventpoll: one perepoll\_create\(\). Has the wait queue, the RB tree of items being watched, andrefsat offset 176: an hlist head that links everyepitempointing back at this instance from somewhere else. It’s the incoming-edges list in the graph.

struct epitem: one per (epoll instance, watched fd) pair. Hasepi\-\>ep, a pointer to its owningeventpoll. If the watched fd is itself an epoll, this epitem is also linked intothatepoll’srefshlist viafllink.

The graph walker iteratesep\-\>refs, followsepi\-\>epfor each entry to reach a parenteventpoll, and recurses. Thatepi\-\>epdereference is the UAF.

The 2023 Optimization

Before March 2023, everyepoll\_ctl\(ADD\)with a nested target acquired a global mutex calledepmutex. Under HTTP benchmarks, 58% of CPU time was lost to contention on it.

A patch replacedepmutexwith a per-instancerefcount\_t, added adyingflag tostruct epitem, and narrowed the remaining lock to only be held during actual graph walks. Throughput went up 60%.

The race happens in the graph walkersep\_get\_upwards\_depth\_procandreverse\_path\_check\_proc. Both functions iterateep\-\>refsunderrcu\_read\_lock\(\)while other threads tear down the structures they’re pointing at. The oldepmutexhad been incidentally serializing this, but the new optimization was too open and nobody noticed the walkers race. The reason is they don’t touch any of the data the mutex was nominally protecting, they were only reading data.

The Bug

static int ep_loop_check(struct eventpoll *ep, struct eventpoll *to)
{
	int depth, upwards_depth;

	inserting_into = ep;
	/*
	 * Check how deep down we can get from @to, and whether it is possible
	 * to loop up to @ep.
	 */
	depth = ep_loop_check_proc(to, 0);
	if (depth > EP_MAX_NESTS)
		return -1;
	/* Check how far up we can go from @ep. */
	rcu_read_lock();
	upwards_depth = ep_get_upwards_depth_proc(ep, 0);
	rcu_read_unlock();

	return (depth+1+upwards_depth > EP_MAX_NESTS) ? -1 : 0;
}

..
snip
..

static int ep_get_upwards_depth_proc(struct eventpoll *ep, int depth)
{
    int result = 0;
    struct epitem *epi;

    if (ep->gen == loop_check_gen)
        return ep->loop_check_depth;

    hlist_for_each_entry_rcu(epi, &ep->refs, fllink)
        result = max(result, ep_get_upwards_depth_proc(epi->ep, depth + 1) + 1);
    ep->gen = loop_check_gen;
    ep->loop_check_depth = result;
    return result;
}

ep\_get\_upwards\_depth\_procruns underrcu\_read\_lock\(\). Eachepitemis safe when unlinked, it’s freed viacall\_rcu\(\), so RCU keeps it alive through the read-side critical section. There’s even a comment in the source that acknowledges the RCU reader:

/* The rcu read side, reverse_path_check_proc(), does not make
 * use of the rbn field.
 */
call_rcu(&epi->rcu, epi_rcu_free);

That comment is correct about theepitem. It says nothing about whatepi\-\>eppoints to.

Now look at the teardown path:

static void ep_free(struct eventpoll *ep)
{
    mutex_destroy(&ep->mtx);
    free_uid(ep->user);
    wakeup_source_unregister(ep->ws);
    kfree(ep);
}

kfree\(\). Immediate. No RCU grace period.

The walker loadsepi\-\>epa pointer read, then dereferences the target but thateventpollmay have already been freed and reused by a completely differentkmalloc\-256allocation.

Triggering it

I initially tried two threads on different CPUs, one walking the graph one closing an epoll fd, it didn’t work. The window between loadingepifrom the hlist and followingepi\-\>epis a handful of ARM64 instructions. What does work is same-CPU preemption. The Frankel device I was testing on runsCONFIG\_PREEMPT=yandCONFIG\_PREEMPT\_RCU=y, which meansrcu\_read\_lock\(\)just bumps a per-task counter it doesn’t disable preemption. A timer tick during the walk can yield the CPU to the closer thread even though the walker is mid-RCU.

Just to give you a sense on numbers (CONFIG\_HZ=250, tick every 4 ms):

• 4,096 parents: walk takes ~400 us. Rarely overlaps a tick.
• 8,000 parents: ~2 ms. Overlaps reliably. About 4% hit rate per attempt.

If the closer thread busy waits for the trigger signal, the scheduler treats it the same priority as the walker and never switches, but if you add the closerusleep\(1000\)in a loop while waiting. Sleeping threads get scheduling priority when they wake and the scheduler preempts the walker immediately.

The Pixel’s default governor throttles to 729 MHz at idle, at that frequency the traversal timing shifts enough that the race stops firing entirely :’)

What Gets Written

struct eventpoll {
    struct mutex               mtx;                  /*     0    48 */
    wait_queue_head_t          wq;                   /*    48    24 */
    wait_queue_head_t          poll_wait;            /*    72    24 */
    struct list_head           rdllist;              /*    96    16 */
    rwlock_t                   lock;                 /*   112     8 */
    struct rb_root_cached      rbr;                  /*   120    16 */
    struct epitem *            ovflist;              /*   136     8 */
    struct wakeup_source *     ws;                   /*   144     8 */
    struct user_struct *       user;                 /*   152     8 */
    struct file *              file;                 /*   160     8 */
    u64                        gen;                  /*   168     8 */ /* read, then WRITE loop_check_gen */
    struct hlist_head          refs;                 /*   176     8 */ /* READ as hlist pointer           */
    u8                         loop_check_depth;     /*   184     1 */ /* WRITE 0 or a kernel pointer     */
    refcount_t                 refcount;             /*   188     4 */
    unsigned int               napi_id;             /*   192     4 */

    /* size: 200, cachelines: 4, members: 15 */
};

struct eventpolllives inkmalloc\-256(order-1 slabs, 32 objects per slab,cpu\_partial=52on this device).init\_on\_free=1is set by default on Frankel devices and Android adds custom padding at the end of each object therefore the structure is different from mainline linux a bit.

Since the traversal ofrefs\.firstis at offset 176, this is our target offset, which is critical as my main attempt to exploit this as a one shot w/o any infoleaks:

If it’szero(theinit\_on\_freecase), the hlist looks empty. The walker skips the loop, writesloop\_check\_genat 168 and a zero byte at 184, returns. Silent corruption of 9 bytes in whatever object gets reused.

If it’snonzero, the walker follows it as a pointer to anepitem, computescontainer\_of\(\), dereferencesepi\-\>ep, and recurses into wherever that points. This is an arbitrary write primitive.

If you can grab the object where you control offset 176, you steer the recursion. Each level writesloop\_check\_gen(a global u64 counter that increments perepoll\_ctl\(ADD\)) and a zero byte at fixed offsets from the pointer target. That’s a constrained write primitive. What you do with it from there depends on whatkmalloc\-256object you use for reclaim, and how creative you’re feeling.

Note: There are other paths I did not include in this blog. One of them leads tomutex\_unlockthat if you are careful and brave enough to walk into. They require tremendous memory pressure and some of them might be fruitful. Trivia: We also control andgenandloop\_check\_depthwhich allows to zero out (or write somewhat deterministically yet very slowly) a controlled value to the freed chunk.

Can You Cross Cache This?

I wanted to exploit this vuln as one shot primitive and wanted to do this using PTE corruption, my attempts failed, but this was my strategy. If I were to infoleak, I’d use a different primitive and then solve everything pretty easily withrefs\.firstas a pointer. Note: This part is technical. If you are not familiar with PCPs, Page Table Entries or SLUB / Buddy internals, I encourage you to read about them before you try reading this part.

The freed objects goes intokmalloc\-256and uses order-1 slabs. ARM64 PTE pages are order-0 (4 KB). These sit on different PCP freelists. The order-1 page freed from the slab cache won’t satisfy an order-0 PTE request unless PCP overflows and buddy splits it. Arranging that overflow during the narrow race window turned out to be non-trivial. It was possible to perform the split w/o invoking the race, however, integrating both pieces together was never a succeess.

These pieces work separately. Shaping 244 out of 250 slab pages go to buddy with 16 children forking and faulting 8 GB each, all available UNMOVABLE order-1 gets split for PTE allocations. The slab2buddy transition works, the buddy2PTE transition works, the problem is combining them with the race. The walker finishes in about 2 ms. The full cross cache pipeline, SLUB discard, PCP drain, buddy insertion, PTE allocation with\_\_GFP\_ZEROtakes on the order of 100 ms. The gen write needs to land on a physical page that hasalreadycompleted the transition from slab to PTE, and those timelines don’t overlap. I couldn’t find a way to stretch the walk long enough without resorting toSCHED\_FIFOor similar privileged tricks, which defeats the purpose.

Same-cache reclaim ignores this entirely. SLUB’s per-CPU freelist is LIFO: last freed, first allocated. An immediatekmalloc\(256\)on the same CPU gets you the exact slot. The hard part is finding akmalloc\-256object with a useful layout at offsets 168 and 176, I did not invest too much time into this.

The Fix

Commit 07712db80857:

static void ep_free(struct eventpoll *ep)
 {
     mutex_destroy(&ep->mtx);
     free_uid(ep->user);
     wakeup_source_unregister(ep->ws);
-    kfree(ep);
+    kfree_rcu(ep, rcu);
 }

The fix adds astruct rcu\_headtoeventpoll.kfree\_rcu\(\)defers the free until the RCU grace period ends. Since the walker holdsrcu\_read\_lock\(\), the grace period can’t complete until it’s done.

Closing Thoughts

What stays with me about this bug isn’t the race condition or the allocator internals. It’s how much work it takes to understand which code paths in epoll are protected by what. Wait queue locks serialize callbacks file refcounts gateep\_free.\_\_fputsequences cleanup.call\_rcudefersepitemfrees. Each mechanism covers something. You have to hold all of them in your head at once before you can point atepi\-\>epand be sure that nothing is keeping the target alive. I spent several days just on that part.

I encourage anyone to try to exploit this on a modern Android system, it sounds fun and I’d be interested to see how u managed to get a stable arb read and write primitives based on this bug.