Native inotify in FreeBSD

# Native inotify in FreeBSD - Klara Systems Source: [https://klarasystems.com/articles/native-inotify-in-freebsd/](https://klarasystems.com/articles/native-inotify-in-freebsd/) ## A Simple File Monitoring Problem Some time ago a customer presented us with a seemingly simple problem\. They had a workflow where files were uploaded to a server withscp, and they wanted to run an action on the server after the upload had finished\. Their initial solution was to use theinotifywaitutility to watch the upload directory and wait for aCLOSEnotification, upon which they would run their action\. \`inotifywait\` can monitor a file hierarchy for accesses or modifications, such as file creation, writes, deletes, and so on\. This solution worked perhaps 99% of the time, but once in a whileinotifywaitwould fail to emit theCLOSEnotification after an upload\. The root cause of the missing notifications had to do with howinotifywaitis implemented under the hood\. It used a FreeBSD\-specific library, \`libinotify\.so\`, which implements Linux's[inotify](https://www.man7.org/linux/man-pages/man7/inotify.7.html)interface in userspace using the[\`EVFILT\_VNODE\` kqueue filter](https://man.freebsd.org/cgi/man.cgi?kqueue)\. Linux's inotify and FreeBSD'sEVFILT\_VNODEare both system call interfaces which let an application passively monitor a file or directory for changes\. The interfaces themselves are quite different from each other; below is a sample function which monitors all files in a given directory for file close events and prints events when they occur, effectively the same functionality that our customer wants\. First, the inotify\-based version: ``` voidwatchdir_inotify(const char *dirname){ struct inotify_event *ev; size_t evsz; int ifd, wd; ifd = inotify_init(); if (ifd < 0) err(1, "inotify_init"); wd = inotify_add_watch(ifd,dirname, IN_CLOSE); if (wd < 0) err(1, "inotify_add_watch(%s)", dirname); evsz = sizeof(*ev) + NAME_MAX + 1; ev = malloc(evsz); if (ev == NULL) err(1, "malloc"); /* Main loop. */ while (true) { ssize_t n; n = read(ifd, ev, evsz); if (n < 0) err(1, "read"); if (ev->len > 0) printf("CLOSE: %s\n", ev->name); }} ``` This is straightforward: we create an inotify descriptor, ask it to watch the specified directory forIN\_CLOSEevents, and then loop, reading event descriptions from the descriptor\. This can easily be integrated into a larger event loop by usingkevent\(\)orepoll\(\)to wait for data to arrive on the inotify descriptor\. Here is a version which useskqueue\(\)andEVFILT\_VNODEto try and implement the same functionality: ``` struct entry { LIST_ENTRY(entry) link; char name[NAME_MAX + 1]; int fd; bool seen;};LIST_HEAD(entry_list, entry);static voidscan(int kq, DIR *d, struct entry_list *entries){ struct dirent *dp; struct kevent kev; struct entry *e, *tmp; int error; LIST_FOREACH(e, entries, link) e->seen = false; rewinddir(d); while ((dp = readdir(d)) != NULL) { int flags; if (strcmp(dp->d_name, ".") == 0 || strcmp(dp->d_name, "..") == 0) continue; /* Search for an existing entry. */ LIST_FOREACH(e, entries, link) { if (strcmp(e->name, dp->d_name) == 0) { e->seen = true; break; } } if (e == NULL) { int fd; /* Not found, create a new entry. */ fd = openat(dirfd(d), dp->d_name, O_RDONLY); if (fd < 0) { if (errno == EOPNOTSUPP) continue; err(1, "openat(%s)", dp->d_name); } e = malloc(sizeof(*e)); if (e == NULL) err(1, "malloc"); (void)strlcpy(e->name, dp->d_name, sizeof(e->name)); e->fd = fd; e->seen = true; LIST_INSERT_HEAD(entries, e, link); flags = EV_ADD | EV_CLEAR; } else { flags = EV_CLEAR; } /* Watch the file for close events. */ EV_SET(&kev, e->fd, EVFILT_VNODE, flags, NOTE_CLOSE | NOTE_CLOSE_WRITE | NOTE_RENAME | NOTE_DELETE, 0, e); error = kevent(kq, &kev, 1, NULL, 0, NULL); if (error < 0) err(1, "kevent"); } /* Remove entries that were not seen. */ LIST_FOREACH_SAFE(e, entries, link, tmp) { if (e->seen) continue; LIST_REMOVE(e, link); close(e->fd); free(e); } /* Add the directory itself to the queue. */ EV_SET(&kev, dirfd(d), EVFILT_VNODE, EV_ADD | EV_CLEAR, NOTE_WRITE, 0, 0); error = kevent(kq, &kev, 1, NULL, 0, NULL); if (error < 0) err(1, "kevent");}voidwatchdir_kqueue(const char *dirname){ struct kevent kev; struct entry_list entries; DIR *d; int error, kq; LIST_INIT(&entries); kq = kqueue(); if (kq < 0) err(1, "kqueue"); d = opendir(dirname); if (d == NULL) err(1, "opendir(%s)", dirname); /* Monitor all files already in the directory. */ scan(kq, d, &entries); /* Main loop. */ while (true) { error = kevent(kq, NULL, 0, &kev, 1, NULL); if (error < 0) err(1, "kevent"); if (kev.udata == NULL) { /* Directory itself changed, rescan. */ scan(kq, d, &entries); } else { struct entry *e; e = (struct entry *)kev.udata; if ((kev.fflags & (NOTE_RENAME | NOTE_DELETE)) != 0) { /* File was renamed, scan. */ scan(kq, d, &entries); } if ((kev.fflags & (NOTE_CLOSE | NOTE_CLOSE_WRITE)) != 0) printf("CLOSE: %s\n", e->name); } }} ``` We can immediately see that this is way more complex\.EVFILT\_VNODEoperates on file descriptors and can only watch the file referenced by the file descriptor\. This means that to watch a whole directory, we have to: 1. traverse the directory and open each file, allocating a tracking structure for it and registering it with our kqueue; 2. monitor the directory itself for changes, so we can rescan when files are added or removed; 3. keep track of each file's name in the directory, including monitoring each file for renames\. All of this is quite a lot of work\. Moreover, for large directories the performance is poor due to the need to scan the directory whenever entries are added or removed\. Furthermore, since each file in the directory needs to be opened, the application may need to use many file descriptors, increasing kernel memory usage and potentially running up against descriptor limits\. Finally, there is at least one race condition in the kqueue version: ifscp\(or any other utility\) is adding a new file to the watched directory, the underlying operations will look something like this: 1. create the file and open it for writing; 2. write the data to the file using the file descriptor obtained in step 1; 3. close the file descriptor\. Do you see the problem? Step 1 will cause thewatchdir\_kqueue\(\)function to rescan the directory, open the new file, and register it with the kqueue\. However, at the same time, the upload service is executing steps 2 and 3\. If it finishes step 3 before the new file is registered,watchdir\_kqueue\(\)will not report theCLOSEevent, as it happened before the file was registered\. This is exactly the problem our customer was running into\. This can be seen in practice by running the following commands to create a new file in a watched directory: ``` $ echo a > foo ``` versus ``` $ dd if=/dev/random of=foo bs=1M count=128 ``` The former will run very quickly and has a good chance of triggering the race, while the latter should be more reliable at generating \`CLOSE\` notifications from the sample program\. In any case, of the two solutions, the one using inotify is clearly preferable: it's simpler and more efficient, and at least on Linux, there is no such race condition\. There's only one catch: until recently FreeBSD had no native inotify implementation\. The \`libinotify\.so\` mentioned above suffers from the same race condition by design, so while we can implement the inotify interface in software that use it, these bugs are unavoidable\. ## Towards A Solution Confronted with these limitations ofEVFILT\_VNODE, it seemed clear that FreeBSD's file monitoring interfaces were inadequate, but the path forward was not immediately obvious\. We looked at other operating systems to see how they handle file monitoring\. Other BSDs \(OpenBSD, NetBSD and DragonflyBSD\) also implementEVFILT\_VNODE, but they also necessarily suffer from the same problems sinceEVFILT\_VNODE,\` requires each monitored file to be opened\. NetBSD implements the inotify interface in the kernel as part of a Linux system call compatibility layer, but there it is also implemented on top ofEVFILT\_VNODE, rather than being directly integrated into the VFS, and thus inherits much of the same limitations and complexity\. Linux on the other hand has the inotify interface\. inotify introduces a new type of file descriptor and lets one use it to monitor files and directories for filesystem events\. Crucially, monitoring a directory will automatically cause all files in the directory to be monitored, as well as any any files added to the directory after monitoring starts\. In contrast withEVFILT\_VNODE,, there is no need for the application to open each file or keep track of file names\. Aside from inotify, Linux also has the fanotify interface, which provides similar but even richer functionality, such as the ability to intercept file accesses\. In practice, inotify seems to be more commonly used in OSS, perhaps because it is older and simpler to use\. On MacOS, file monitoring can be done using the[\`FSEvents\`](https://developer.apple.com/documentation/coreservices/file_system_events)API\. This consists of a[kernel component](https://github.com/apple-oss-distributions/xnu/blob/main/bsd/vfs/vfs_fsevents.c)and a system daemon which runs in userspace and uses/dev/fseventsto obtain filesystem access event info, which is published to consumers\. In particular, consumers do not obtain the event stream directly from the kernel\. This design allows the system to implement event stream policies \(e\.g\., how to handle dropped events\) and state management in userspace, providing flexibility while keeping the kernel side relatively simple, but it is also rather complex and general for our modest use\-case\. Inspired byFSEvents, we spent some time implementing a new kqueue filter,EVFILT\_FSWATCH, which could provide some of the same functionality without a userspace daemon\. The idea was to provide a way to monitor all events on all files in a mount point, since that is fairly easy to implement in the kernel\. When registering the filter, the application also provides a pointer to an array of structures, each of which describes one or more events on a particular path\. The kernel knows the layout of these structures and can fill them out when the application polls for new events by callingkevent\(\)\. It avoids the disadvantages ofEVFILT\_VNODE, but we only went as far as implementing a proof\-of\-concept and had not addressed various practical problems, such as handling dropped events, efficient path filtering, or access restrictions\. ## Why inotify? Ultimately, we decided to implement a native inotify interface in FreeBSD\. While inotify has shortcomings of its own\-\-the inability to monitor file hierarchies recursively, poor control over back\-pressure and event drops, among others\-\-it is widely used in software run on FreeBSD \(hence the need for alibinotify\.socompatibility shim\), and it is relatively straightforward to implement\. Furthermore, a native, compatible implementation lets us use inotify in the Linuxulator "for free," which helps quite a few Linux binaries run directly on FreeBSD\. In fact, it turned out that the lack of a native inotify implementation was a sore point for quite a few FreeBSD users\. There is a great deal of OSS which uses it, including foundational libraries such as GLib and Qt \(which both provide higher\-level interfaces on top of it\); some software supportsEVFILT\_VNODEas an alternative selected at compile\-time, but quite a few applications do not, and indeed, until recently, over 100 ports in the FreeBSD ports tree referenced the userspace libinotify directly\. Whilelibinotify\.sois very useful and is a key part of enabling standard desktop software on FreeBSD, the above\-mentioned issues withEVFILT\_VNODEcaused a lot of frustration\. Aside from the race conditions which motivated this investigation in the first place, the resource overhead ofEVFILT\_VNODEwas prohibitive in some cases\. For instance, syncthing, a popular file synchronization service, will log a warning when using kqueue to monitor a large directory due to frequent reports of high CPU and memory usage in that scenario\. With our native inotify implementation, these problems will go away, though further work is needed to make FreeBSD's inotify symbols available in thegolang\.org/x/sys/unixpackage and then to make use of them in syncthing's inotify\-based file event notification package\. Rather than having two competing interfaces in FreeBSD, it would have been nice to somehow extendEVFILT\_VNODEto support the functionality we needed, but to do that we need a way forkevent\(\)to export file names out of the kernel\-\-removing the need for userspace to scan directories and keep track of file names\-\-and that means that it is simply not a good fit for our problem\. So we set ourselves the goal of implementing inotify natively, and in particular, staying source\-compatible with Linux to ensure that existing code just works on FreeBSD\. ## The inotify Interface The earlier example ofwatchdir\_inotify\(\)already shows much of the interface, which consists of three system calls:inotify\_init\(\),inotify\_add\_watch\(\)andinotify\_rm\_watch\(\)\. They respectively create an inotify descriptor, start monitoring a file or directory, and stop monitoring a file or directory\. Callingclose\(2\)on an inotify descriptor implicitly stops all monitoring associated with it\. Theinotify\_add\_watch\(\)system call takes an inotify descriptor, a path and an event mask, and returns awatch descriptor, which is effectively a pseudo\-file descriptor that can be passed toinotify\_rm\_watch\(\)\. In particular, each monitored object gets a unique watch descriptor associated with the inotify descriptor\. These watch descriptors are not subject to file descriptor resource limits, which allows many directories to be monitored simultaneously\. However, they are subject to a global limit defined by thevfs\.inotify\.max\_user\_watchessysctl\. In FreeBSD we provide an extra system call,inotify\_add\_watch\_at\(2\`, akin toopenat\(2\), which takes a directory descriptor \(or theAT\_FDCWDconstant\) and a relative path\. This is to enable the use of inotify in[Capsicum sandboxes](https://man.freebsd.org/cgi/man.cgi?capsicum), as sandboxed programs do not permit filesystem accesses without a directory descriptor\. The example program in[\`/usr/share/examples/inotify/inotify\.c\`](https://cgit.freebsd.org/src/tree/share/examples/inotify/inotify.c)offers a more full\-fledged example which demonstrates the different events that inotify can report\. All it does is add a watch for each path listed on the command line and report all events that it receives\. Theinotify\-toolspackage provides similar utilities; in FreeBSD 15\.0 these and other third\-party packages will make use of our native inotify implementation out of the box\. After several months of development, testing and code review, our inotify implementation for FreeBSD has[landed](https://cgit.freebsd.org/src/commit/?id=f1f230439fa48581f40a57f095627f667a9713c3)in the development branch and will be available in FreeBSD 15\.0 and 14\.4\. It aims to be completely compatible with Linux, but due to design differences between the two kernels, there are some small differences in behaviour\. We expect that the vast majority of applications will not notice them\. The main difference is in the way that files with multiple hard links are handled\. Recall that a given file can have multiple names since files themselves do not contain a name, but rather are referred to by directory entries, which consist of a name and an inode number\. The inode number uniquely identifies the file within its filesystem\. Thus, different entries in a directory might refer to the same underlying file, and a file can have names in multiple directories\. The following examples illustrate this:  Here we have a file with two hard links namedlink1andlink2out of an abundance of creativity\. Suppose an application uses inotify to monitordir1for events\. What happens if a process opens the file vialink2? On Linux, the documented behavior is that inotify will not report the open since only watches for accesses via the watched directory\. On FreeBSD however, an access via either hard link will generate an inotify event, and the name recorded in the event will be that of the link in the watched directory\. Using the[example program](https://cgit.freebsd.org/src/tree/share/examples/inotify/inotify.c)we can see this in action\. ``` $ mkdir /tmp/dir1 /tmp/dir2 $ echo a > /tmp/dir1/link1 $ ln /tmp/dir1/link1 /tmp/dir2/link2 $ inotify --libxo json,pretty /tmp/dir1 ``` Now, suppose we access \`/tmp/dir2/link2\` from a different shell: ``` $ cat /tmp/dir2/link2 a The inotify program will report these accesses from `cat(1)` even though they did not come via `/tmp/dir1`: {   "__version": "1",    "events": [     {       "wd": 1,       "event": "IN_OPEN",       "name": "link1"     },     {       "wd": 1,       "event": "IN_ACCESS",       "name": "link1"     },     {       "wd": 1,       "event": "IN_CLOSE_NOWRITE",       "name": "link1"     } ]} ``` In the second example, we have only a single directory: Here, if the directory \`dir\` is watched, any operation on the file via either hard link will generate two events, one for each link: ``` $ mkdir /tmp/dir $ touch /tmp/dir/link1 $ ln /tmp/dir/link1 /tmp/dir/link2 $ inotify --libxo json,pretty /tmp/dir ``` Then, accessing one link with \`cat\(1\)\` again yields the following output: ``` {   "__version": "1",    "events": [     {       "wd": 1,       "event": "IN_OPEN",       "name": "link2"     },     {       "wd": 1,       "event": "IN_OPEN",       "name": "link1"     },     {       "wd": 1,       "event": "IN_ACCESS",       "name": "link2"     },     {       "wd": 1,       "event": "IN_ACCESS",       "name": "link1"     },     {       "wd": 1,       "event": "IN_CLOSE_NOWRITE",       "name": "link2"     },     {       "wd": 1,       "event": "IN_CLOSE_NOWRITE",       "name": "link1"     }   ] } ``` This behaviour is due to the low\-level details of how the FreeBSD kernel keeps track of file names, which are quite different from Linux\. Specifically, in Linux, inotify events are tied to a particular open file handle, which contains the name of the link used to access the underlying inode\. Thus, when a file descriptor is used to access a file somehow, e\.g\., callingread\(2\), Linux can generate an event using the name associated with the descriptor\. FreeBSD, by contrast, does not store file names with file descriptions, so our inotify implementation needs to fall back to searching the name cache to find names for a given inode\. This architectural difference is the cause of the divergence described above\. In practice, hard links are not very commonly used, and we expect that users will not notice this difference in behaviour\. During development, we ran a number of third\-party test suites against the our implementation, and none of them reported any problems due to this difference\. The FreeBSD inotify implementation also neatly handles file accesses via the NFS server\. That is, if a watched directory is exported via NFS and accessed by a client, inotify events will be generated by the server, as can be seen with DTrace: ``` # dtrace -n 'vfs::vop_inotify:entry {stack();}' dtrace: description 'vfs::vop_inotify:entry ' matched 1 probe CPU     ID                    FUNCTION:NAME   0  79459                vop_inotify:entry               kernel`VOP_INOTIFY_APV+0x129               kernel`vop_read_post+0x1b4               kernel`VOP_READ_APV+0xcd               kernel`nfsvno_read+0x313               kernel`nfsrvd_read+0xf04               kernel`nfsrvd_dorpc+0x549               kernel`nfssvc_program+0x1054               kernel`svc_run_internal+0x144f               kernel`svc_thread_start+0xb               kernel`fork_exit+0xa3               kernel`0xffffffff81b176ee ``` Similarly, inotify is interoperable with FreeBSD'snullfsfilesystem, which allows arbitrary directories to be mounted elsewhere in the file hierarchy\. That is, one can monitor a directory in anullfsmountpoint and receive events for accesses via the "lower" layer\. Conversely, monitoring a directory in the lower layer will also report accesses via thenullfsmount\. ## Wrapping Up Finally, our inotify implementation aims to be compatible with Linux as much as possible, so it was trivial to add[Linuxulator support](https://cgit.freebsd.org/src/commit/?id=3965de642c29d831649c8307203303de560d721a)\. That is, Linux binaries which use inotify will now work on FreeBSD \(assuming of course there are no other missing features\)\. Quite a few applications make use of it in some way, so this should be a significant step forward for the FreeBSD emulation layer\. Klara helps organizations deploy, optimize, and support FreeBSD infrastructure at every scale, from storage platforms and virtualization environments to kernel development and performance engineering\. Organizations looking to modernize and maintain FreeBSD infrastructure can rely on[Klara Inc\.](https://klarasystems.com/?utm_source=chatgpt.com)for FreeBSD development, support, and focused solutions backed by deep kernel and systems expertise\.