Is There I/O After Death? What Happens to Io_uring When a Process Dies

# Is There I/O After Death? What Happens to io_uring When a Process Dies Source: [https://blog.ydb.tech/is-there-i-o-after-death-what-happens-to-io-uring-when-a-process-dies-92c65354873f?gi=0d832297d707&postPublishedType=repub](https://blog.ydb.tech/is-there-i-o-after-death-what-happens-to-io-uring-when-a-process-dies-92c65354873f?gi=0d832297d707&postPublishedType=repub) [](https://medium.com/@eivanov89?source=post_page---byline--92c65354873f-----------------------------------------) Press enter or click to view image in full size Most people \(myself included\) would presume that I/O dies with the process\. Kill it, reap it with`waitpid\(\)`, and surely the old process can no longer touch your storage\. That intuition holds for Linux native AIO, at least on the kernel I tested\. But with ordinary io\_uring, it is wrong: a process can be dead and reaped while writes it submitted earlier are still alive inside the kernel — and can reach the device afterwards\. A very subtle difference\. This matters if a fail\-fast application treats`waitpid\(\)`as an I/O barrier\. A replacement process may start recovery, inspect storage, or begin writing while requests from the previous incarnation are still on their way\. ## Need a barrier? Take a lock There is a surprisingly simple way to turn this behavior into a barrier\. Let the writer open the device and take an exclusive lock: ``` int fd = open(path, O_RDWR | O_DIRECT);flock(fd, LOCK_EX); ``` After killing and reaping the writer, the replacement process must acquire the same lock: ``` int fd = open(path, O_RDWR | O_DIRECT);for (;;) { if (flock(fd, LOCK_EX | LOCK_NB) == 0) { break; } if (errno != EWOULDBLOCK && errno != EAGAIN) { perror("flock"); abort(); } usleep(1000); // retry in 1 ms} ``` The separate`open\(\)`matters\.`flock\(\)`locks are associated with an open file description\. A file descriptor inherited across`fork\(\)`or created with`dup\(\)`refers to the same open file description, so it would not contend with the writer's lock in the way we need here\. Outstanding io\_uring requests retain references to the writer’s struct file\. The lock belongs to the open file description and is removed only when its final reference is released\. So if the old requests are still alive, the new process cannot acquire the lock yet\. In fact, this is exactly how we[open](https://github.com/ydb-platform/ydb/blob/94a017b78fe62317b7d6d28471db7e296fcc8959/ydb/library/pdisk_io/aio_linux.cpp#L297-L324)devices in YDB, regardless of whether we use native AIO or io\_uring\. My hunch is that native AIO may have had the same issue in the past, and the lock protocol simply survived the transition\. **TL;DR** On Linux 6\.6\.79, I observed: Press enter or click to view image in full size For ordinary io\_uring, I could observe storage changing after the writer had already been killed and successfully reaped\. For SQPOLL and Linux native AIO, the same delayed writes instead made process teardown take longer:`waitpid\(\)`did not return until they had drained\. These are measured results backed by the Linux source, not a portable guarantee for every kernel, filesystem, device, or API called “AIO”\. ## Catching I/O from the grave I wrote a small[test](https://github.com/eivanov89/toys/tree/master/cpp/io_uring_from_grave)\. The child continuously keeps roughly 32K direct 4 KiB writes in flight\. It cycles over 1,024 known positions, storing a writer ID and generation number in every block\. The parent then: 1. sends`SIGKILL`; 2. waits for the child using`waitpid\(\)`; 3. reads all 1,024 positions; 4. reads them again; 5. compares every 4 KiB block\. If a block changes between the two snapshots, the supposedly dead writer has modified storage after`waitpid\(\)`returned\. ## First attempt: nothing happens On an idle NVMe device: ``` SIGKILL at +1000.061 mswaitpid took 7.514 msfirst snapshot: 1024/1024 blocks from writersecond snapshot: 1024/1024 blocks from writerall 1024 positions unchanged ``` At first glance, this looks reassuring\. It isn’t\. An idle NVMe is simply too fast\. Outstanding writes usually finish before, or while, the parent performs its first snapshot\. The interesting race window is there, but hard to see\. So I made it larger\. ## Slow the writes down Linux has a convenient device\-mapper target called`dm\-delay`\. It can delay writes by a fixed amount while reads remain fast\. I configured a 3\-second write delay\. The child writes through the delayed mapping, while the parent verifies the underlying partition directly\. ## GetEvgenii Ivanov’s stories in your inbox Join Medium for free to get updates from this writer\. Remember me for faster sign in Then: ``` SIGKILL at +1000.064 mswaitpid took 2.951 msfirst snapshot: 0/1024 blocks from current writersecond snapshot: 1024/1024 blocks from current writer1024/1024 positions changedI/O from the grave dettected ``` The process was gone\.`waitpid\(\)`had returned in about 3 milliseconds\. And almost three seconds later, its writes arrived\. Hooray, we caught I/O from the grave\. That is the entire bug in one experiment\.`waitpid\(\)`told us that the task was dead\. It did not tell us that its I/O was dead\. ## It also happens on a real NVMe queue Of course,`dm\-delay`is artificial\. So I tried to reproduce the same thing on the native NVMe path\. To make the queue deep without introducing another writer, I ran a read\-only`fio`workload: ``` fio --name=grave_readload \ --filename=/dev/nvme2n1p2 \ --readonly \ --rw=randread \ --bs=4096 \ --direct=1 \ --ioengine=io_uring \ --iodepth=2048 \ --numjobs=32 ``` Then I killed the writer as before\. One run produced: ``` SIGKILL at +1000.059 mswaitpid took 8.028 msfirst snapshot took 252.735 mssecond snapshot took 177.544 ms627/1024 positions changedI/O from the grave dettected ``` Both snapshots contained blocks from the same writer ID\. But generation counters advanced at 627 positions between the two reads\. No delayed mapper\. No second writer\. The old process was already reaped, and its writes were still completing\. ## What exactly survives? The lock experiment gives another way to observe this\. Before writing, the child opens the device separately and takes`LOCK\_EX`\. After killing and reaping it, the parent immediately tries: ``` flock(fd, LOCK_EX | LOCK_NB); ``` With ordinary io\_uring and the same 3\-second delayed writes: ``` +1000.086 ms: parent sent SIGKILL+1002.916 ms: waitpid returned+1002.922 ms: LOCK_EX | LOCK_NB -> EWOULDBLOCK+3189.371 ms: LOCK_EX | LOCK_NB -> acquired ``` The child cannot execute userspace after`SIGKILL`, so there is no explicit`flock\(LOCK\_UN\)`happening later\. Instead, outstanding io\_uring requests retain the child's`struct file`\. Linux removes locks from`\_\_fput\(\)`via`locks\_remove\_file\(\)`when the final reference disappears\. So the failed lock attempt tells us something stronger than “a write happened later”: after`waitpid\(\)`returned, kernel objects associated with the dead writer were still retaining its open file description\. Once the lock became available, verification saw no further changes\. That is why the same mechanism can be used as a barrier\. ## SQPOLL behaves differently Then I repeated the experiment with`IORING\_SETUP\_SQPOLL`\. Again, writes were delayed by 3 seconds\. This time: ``` SIGKILL at +1000.065 mswaitpid took 2149.286 msfirst snapshot: unchangedsecond snapshot: unchanged ``` The delay did not disappear\. It moved into teardown\. Instead of returning after a few milliseconds,`waitpid\(\)`spent roughly another two seconds waiting\. The lock agreed: ``` waitpid returnedLOCK_EX | LOCK_NB -> acquired on first attempt ``` I observed the same behavior under native NVMe queue pressure\. So on this kernel, SQPOLL teardown acted as a barrier for these writes\. ## And so does Linux native AIO I also implemented the writer using Linux native AIO: ``` io_setup()io_submit()io_getevents() ``` This is the kernel AIO API, not POSIX`aio\_write\(\)`\. With the same 3\-second`dm\-delay`setup: ``` SIGKILL at +1000.064 mswaitpid took 2246.371 msfirst snapshot: unchangedsecond snapshot: unchanged ``` Again, the remaining delay was paid inside process teardown\. And again, the parent’s first lock attempt after`waitpid\(\)`succeeded immediately\. Under heavy read pressure,`waitpid\(\)`stretched even further \- to several seconds in some runs — but I did not observe native\-AIO writes completing afterwards\. ## Why? I’m not a kernel developer, so what follows is source\-code archaeology rather than kernel expertise\. I also used AI heavily for this part and only did a quick manual review of the relevant kernel code\. The kernel teardown paths are different\. For ordinary io\_uring,`do\_exit\(\)`[calls](https://github.com/torvalds/linux/blob/v6.6/kernel/exit.c#L829)`io\_uring\_files\_cancel\(\)`, which in turn calls`\_\_io\_uring\_cancel\(false\)`\. The interesting bit is`false`: compare that with the`cancel\_all=true`path below\.`io\_uring\_cancel\_generic\(\)`treats these cases differently when deciding which in\-flight requests must be waited for\. This matches the experiment: with ordinary io\_uring, the raw\-block writes can survive far enough into teardown to retain their file references and complete after the process becomes reapable\. SQPOLL is different\. Its kernel submission thread exits through`io\_uring\_cancel\_generic\(true, sqd\)`\. Here`cancel\_all`is`true`, and the cancellation path keeps waiting for the relevant in\-flight requests before the SQPOLL thread finishes\. In my experiment, that wait happened before`waitpid\(\)`returned\. Linux native AIO takes yet another path\. When the last user of an address space goes away,`\_\_mmput\(\)`[calls](https://github.com/torvalds/linux/blob/v6.6/kernel/fork.c#L1346)`exit\_aio\(\)`[before](https://github.com/torvalds/linux/blob/v6.6/kernel/fork.c#L1346)`exit\_mmap\(\)`\. And`exit\_aio\(\)`[explicitly waits until all I/O for its AIO contexts is done](https://github.com/torvalds/linux/blob/v6.6/fs/aio.c#L921-L924)\. The kernel comment is refreshingly unambiguous: the function is called when the last user of the`mm`goes away, and outstanding I/O is waited for before teardown completes\. This also explains an important caveat:`SIGKILL`does not magically run an application's AIO destructor\. The barrier comes from kernel teardown of the final`mm`reference\. If another task still shares that`mm`, for example through`CLONE\_VM`, reaping one task does not necessarily trigger`exit\_aio\(\)`\. ## Conclusion The lesson I learned from this is simple: process lifetime and I/O lifetime are not the same thing\. With ordinary io\_uring,`waitpid\(\)`can tell you that the task is gone while its writes are still very much alive\. I focused on direct block\-device I/O here\. The broader lifetime lesson applies beyond block devices, but I would not assume the exact same teardown behavior for other kinds of I/O\. Filesystems add page cache and writeback semantics; networking has its own socket and packet lifetimes\. If`waitpid\(\)`needs to be a barrier in your recovery protocol, verify the particular I/O path you rely on — or introduce an explicit barrier yourself\.