Linux 7.2 Improves Anonymous/Unnamed Pipe Performance For Shell Pipelines & More

Source: https://www.phoronix.com/news/Linux-72-Faster-Anon-Pipe-Write

Yet another performance optimization merged for the in-developmentLinux 7.2kernel is improving the speed of anon_pipe_write, the kernel function used for writing data into anonymous/unnamed pipes such as when using shell pipelines or standard streams from applications.

Breno Leitao of Meta was profiling some of their caching code and found pipe to mutex contention in a hot path, which is now resolved by the newly-merged code to pre-allocate outside the lock for avoiding contention.

In theVFS misc pull requestthat situation is described as:

“anon_pipe_write() called alloc_page() once per page while holding pipe->mutex. The allocation can sleep doing direct reclaim and runs memcg charging, which extends the critical section and stalls any concurrent reader on the same mutex. Now up to 8 pages are pre-allocated before the mutex is taken, leftovers are recycled into the per-pipe tmp_page[] cache before unlock, and any remainder is released after unlock, keeping the allocator out of the critical section on both sides. On a writers x readers sweep with 64KB writes against a 1 MB pipe throughput improves 6-28% and average write latency drops 5-22%; under memory pressure - when the cost of holding the mutex across reclaim is highest - throughput improves 21-48% and latency drops 17-33%. The microbenchmark is added to selftests.”

Very nice gains. More of the numbers in detail can be found viathis patch cover letterby Breno Leitao.

That work is now merged for Linux 7.2.