Deploying RDMA for Large-Scale Inference and Training Workloads

# Deploying RDMA for Large-Scale Inference and Training Workloads
Source: [https://arxiv.org/html/2610.12815](https://arxiv.org/html/2610.12815)
Yi CuiAffiliation:Fudan University ByteDance Inc\.Yukai HuangAffiliation:Fudan University ByteDance Inc\.Huaping ZhouAffiliation:Fudan University ByteDance Inc\.Jiancong TongAffiliation:Fudan University ByteDance Inc\.Zhaoyu WuAffiliation:Fudan University ByteDance Inc\.Gen MiaoAffiliation:Fudan University ByteDance Inc\.Haoran WeiAffiliation:Fudan University ByteDance Inc\.Boyan DingAffiliation:Fudan University ByteDance Inc\.Zhuo JiangAffiliation:Fudan University ByteDance Inc\.Jianxi YeAffiliation:Fudan University ByteDance Inc\.Yibo ZhuAffiliation:Fudan University ByteDance Inc\.Chuanxiong GuoAffiliation:Fudan University ByteDance Inc\.Yang XuJie Wu

###### Abstract

RDMA can reduce communication overhead in distributed machine learning systems, but realizing these benefits in production requires more than replacing TCP\. Variable\-size messages, many concurrent connections, bursty incasts, and service failures place demands on software that hardware offload alone does not address\. This paper reports our experience deployingByteExpress, a software communication layer over hardware RDMA, in ByteDance’s parameter\-server\-based inference and training systems\. We describe three lessons from the deployment: efficient RDMA requires buffer management that spans the application and transport; scalable communication requires coordinated management of connections, receive capacity, and outstanding data; and service availability requires recovery above the transport layer\. Our design combines an RDMA\-aware allocator, zero\-copy message transfers, QP multiplexing, receive\-credit flow control, a software send window, and RPC\-assisted recovery\. In a production inference workload, the system reduces mean and 99th\-percentile model parameter fetch latency by 35\.7% and 43\.7%, respectively, and total CPU usage by 10\.69% relative to the widely deployed TCP stack\. In the training workload,ByteExpressreduces P99 model parameter push latency by up to 74\.0% relative to TCP, with no PFC pauses observed during the reported interval\.

## 1Introduction

Distributed inference and training underpin recommendation, search, and advertising services at ByteDance, including Toutiao and TikTok\. In the parameter\-server architecture used by these services, workers fetch model parameters from multiple servers; trainers also push updates back to those servers\. These exchanges place communication on the critical path of both user\-facing inference requests and iterative training\. By January 2022, ByteDance’s inference and training systems spanned hundreds of clusters and tens of thousands of computing nodes\.

Faster computation increases the pressure on this communication path\. GPU\-based workers can generate substantial network demand\[[4](https://arxiv.org/html/2610.12815#bib.bib29),[38](https://arxiv.org/html/2610.12815#bib.bib30),[21](https://arxiv.org/html/2610.12815#bib.bib22)\], while processing data through the kernel network stack consumes CPU cycles and introduces copying and scheduling overhead\. RDMA offers hardware transport offload and direct access to registered memory\. ByteDance had already deployed RDMA over Converged Ethernet \(RoCE\) in its data centers\[[21](https://arxiv.org/html/2610.12815#bib.bib22),[43](https://arxiv.org/html/2610.12815#bib.bib23)\]\. The remaining challenge was to integrate that infrastructure into inference and training services without sacrificing scalability or availability\.

This paper reports that integration experience\. We presentByteExpress\(BE\), a software communication layer built on hardware RDMA NICs \(RNICs\)\. Applications continue to use Archon, our existing RPC framework\. Archon passes messages toBEthrough a socket\-like interface, andBEissues verbs to the RNIC using Reliable Connection \(RC\) transport\. The RNIC provides transport offload;BEmanages buffers, schedules transfers, and bounds outstanding work; Archon handles request\-level recovery\. This division of responsibility is central to the design\.

The workload makes the integration challenging in three ways\. First, message sizes and buffer lifetimes vary with the model and request, making fixed receive buffers awkward to expose directly to applications\. Second, fan\-out requests and fan\-in responses create many logical connections and bursts of concurrent transfers\. Third, the completion of an inference request or training iteration can depend on its slowest sub\-request\. Hardware transport reliability alone cannot recover from every failed node, stalled connection, or unavailable service replica\.

Our experience yields three main lessons\.

- •Zero\-copy communication is an interface and memory\-management problem\.We buildbemallocon the existingrpmallocallocator to make dynamically allocated application buffers RDMA\-capable\. The resulting ownership model lets the communication layer hand received buffers to application threads without copying\. Batched work requests and completion\-based receive notification then reduce engine overhead \(§\\lx@sectionsign[5](https://arxiv.org/html/2610.12815#S5)\)\.
- •Scaling requires controlling distinct resources at distinct layers\.QP multiplexing reduces connection\-state pressure, and thread domains avoid contention in shared driver structures\. Receive credits protect posted receive capacity, whereas a separate per\-engine send window bounds outstanding bytes during incasts\. These mechanisms complement RNIC congestion control and fabric flow control; they address different bottlenecks \(§\\lx@sectionsign[6](https://arxiv.org/html/2610.12815#S6)\)\.
- •Reliable transport is only one part of service availability\.We retain RPC\-level timeouts, retries, replica selection, and TCP fallback, and combine them with connection monitoring and replicated deployment\. This allows recovery decisions to use service context that is unavailable to the RNIC \(§\\lx@sectionsign[7](https://arxiv.org/html/2610.12815#S7)\)\.

The contribution is the cross\-layer co\-design across these mechanisms and the operational lessons from deploying them\. At the time of this study, the RDMA\-based ML systems had supported production recommendation services for over two years\. For a production recommendation inference workload, RDMA reduced average PS fetch latency by 35\.7%, P99 latency by 43\.7%, and total CPU usage by 10\.69% relative to the widely deployed TCP\. In the training workload, RDMA with the software send window reduced P99 push latency by 74\.0% relative to TCP and eliminated observed PFC pauses during the reported measurement interval \(§\\lx@sectionsign[8](https://arxiv.org/html/2610.12815#S8)\)\.

## 2Background

### 2\.1Inference and Training Systems

Figure 1:Parameter\-server architecture: rankers fetch sharded model parameters for inference; trainers also push parameter updates\.At ByteDance, parameter\-server\-based systems\[[29](https://arxiv.org/html/2610.12815#bib.bib31)\]support model training and personalized ranking for recommendation, search, and advertising\. Figure[1](https://arxiv.org/html/2610.12815#S2.F1)shows the two main roles: workers and parameter servers \(PSs\)\. A worker is a*ranker*during inference and a*trainer*during training\. Model parameters are partitioned intonnshards, for example by hashing, and stored acrossnnparameter servers\.

For online inference, an application forwards a user request to a ranker\. A request may require evaluatingkkmodels\. The ranker fetches the required parameters, evaluates the models, and returns the ranking results\. Because the parameters can span multiple shards, one user request becomes multiple concurrent sub\-requests to parameter servers\. Each ranker also keeps multiple user requests outstanding through asynchronous network I/O\. Request and response sizes vary with user attributes and model structure, ranging from kilobytes to megabytes\. Prior work reports similarly variable message sizes\[[50](https://arxiv.org/html/2610.12815#bib.bib14),[5](https://arxiv.org/html/2610.12815#bib.bib20),[39](https://arxiv.org/html/2610.12815#bib.bib24)\]\.

For training, each trainer works on a model, loads training data, fetches parameters, performs forward and backward propagation, and pushes updates to parameter servers\. Repeated parameter exchanges generate incasts at both workers and servers\. Throughout the paper,Fetchdenotes parameter retrieval; the synthetic experiment labels the corresponding operationPull\.

The measurements in this paper cover different workloads\. In the training workload discussed in§\\lx@sectionsign[6\.3](https://arxiv.org/html/2610.12815#S6.SS3), mean and P99 message sizes are 20 KB and 50 KB, respectively\. The allocator benchmark instead uses synthetic large allocations of 64 KB–1 MB and small allocations of 16 B–16 KB \(§\\lx@sectionsign[5\.2](https://arxiv.org/html/2610.12815#S5.SS2)\)\. The production inference evaluation uses 300 parameter servers and 124 rankers; the training evaluation uses 124 parameter servers and 346 trainers \(§\\lx@sectionsign[8](https://arxiv.org/html/2610.12815#S8)\)\. These examples characterize the evaluated cases, not a single distribution shared by all services\.

### 2\.2RDMA: A Primer

Remote Direct Memory Access \(RDMA\) provides low\-latency, high\-throughput communication using specialized NICs\. RNICs offload transport processing and expose operations to user\-space software, avoiding kernel\-network\-stack overhead on the data path\. Software still allocates buffers, posts operations, and processes completions; RDMA does not eliminate application\-side CPU work\.

A queue pair \(QP\) is a communication endpoint with a send queue \(SQ\) and a receive queue \(RQ\)\. Applications access RNIC functionality through the*verbs*interface, which supports resource creation, work\-request submission, and completion polling\. Completed operations are reported through a completion queue \(CQ\)\.

Two\-sided messaging verbs,SENDandRECV, require participation from both endpoints: the receiver posts a receive buffer before the incoming send can be completed\. One\-sided memory verbs includeWRITE,READ, and atomic operations; RDX proposes extending RDMA beyond memory access to remote code execution\[[17](https://arxiv.org/html/2610.12815#bib.bib11)\]\. They access registered remote memory without requiring the remote CPU to execute a matching operation for each transfer\. The remote application must nevertheless arrange memory registration, access permissions, and buffer lifetimes\.

RCUCUDSEND/RECV✓✓✓WRITE✓✓✗READ/ATOMIC✓✗✗Connection\-oriented✓✓✗Reliable✓✗✗Maximum message size2GB2GBMTUTable 1:Comparison of RDMA transports\.Table[1](https://arxiv.org/html/2610.12815#S2.T1)summarizes three RDMA transport modes: Reliable Connection \(RC\), Unreliable Connection \(UC\), and Unreliable Datagram \(UD\)\. RC and UC are connection\-oriented and support large messages; their supported memory operations differ\. UD supports connectionless communication to multiple destinations, but each message is limited to the maximum transmission unit \(MTU\) and reliability must be supplied above the transport\.BEuses RC to retain hardware reliability and large\-message support while managing connection count in software\.

In our deployment, RC runs on RoCE\-capable hardware\. DCQCN\[[51](https://arxiv.org/html/2610.12815#bib.bib26)\]controls RNIC sending rates in response to congestion feedback, while Priority Flow Control \(PFC\) pauses traffic classes at the link layer to limit buffer overflow\. RC reliability, congestion control, and PFC serve distinct purposes\. Sections[5\.4](https://arxiv.org/html/2610.12815#S5.SS4)and[6\.3](https://arxiv.org/html/2610.12815#S6.SS3)explain the additional software controls used byBE\.

## 3Deployment Requirements

Our goal is to improve communication performance while retaining the scalability and availability expected of production inference and training services\. The following workload properties determine our design choices\.

Variable messages and buffer lifetimes\.Inference and training exchange messages whose sizes are not always known before a request is processed\. Application threads may also retain or transfer ownership of a received buffer while computing a result\. A fixed receive\-buffer layout can complicate zero\-copy integration in this setting\. Registering a new buffer for every message avoids that layout constraint but adds registration overhead and increases the number of memory regions\. We therefore need dynamic allocation from reusable, registered memory \(§\\lx@sectionsign[5\.2](https://arxiv.org/html/2610.12815#S5.SS2)\)\.

Large transfers favor RC, but RC connections must be managed\.UD\-based designs such as FaSST and eRPC\[[24](https://arxiv.org/html/2610.12815#bib.bib33),[25](https://arxiv.org/html/2610.12815#bib.bib37)\]demonstrate the benefits of connectionless communication, particularly for small messages\. Our workload also includes messages much larger than the MTU\. Sending them over UD requires software fragmentation, reassembly, and reliability\. Section[5\.1](https://arxiv.org/html/2610.12815#S5.SS1)compares RC and UD throughput under several link\-rate and MTU configurations\. We choose RC for its hardware support for reliable, large\-message transfers, then multiplex logical connections to limit QP\-state pressure \(§\\lx@sectionsign[6\.1](https://arxiv.org/html/2610.12815#S6.SS1)\)\. This is a choice for our workload and deployment, not a general argument against UD\.

Receive capacity and fabric congestion are separate constraints\.A sender can exhaust posted receive requests even when the network is uncongested\. Conversely, a receiver with available buffers can still be reached through a congested network\. We address the first problem with credits for operations that consume receive requests \(§\\lx@sectionsign[5\.4](https://arxiv.org/html/2610.12815#S5.SS4)\)\. We address the second by limiting outstanding bytes per engine, in addition to the existing DCQCN and PFC mechanisms \(§\\lx@sectionsign[6\.3](https://arxiv.org/html/2610.12815#S6.SS3)\)\. Keeping these controls separate makes their roles explicit: receive credits prevent receiver\-not\-ready \(RNR\) events; the send window limits the amount of data injected during concurrent transfers\.

Incast makes tail behavior a scaling concern\.Many workers can contact the same parameter server, and responses from many servers can converge on a worker\. Although a single flow’s bandwidth\-delay product may be small relative to switch buffering, the aggregate traffic from many concurrent flows can exceed that capacity\. In our training deployment, hardware congestion control alone did not prevent sustained PFC pauses and poor request latency\. This observation motivates a software bound on outstanding data, rather than assuming that the fabric can absorb every burst\.

Service recovery requires application context\.An RNIC can provide reliable delivery on a functioning connection, but it cannot select another service replica or decide how to retry a failed RPC\. Fan\-out operations are particularly sensitive to slow or failed sub\-requests\. We therefore retain Archon’s RPC abstraction and extend its existing availability mechanisms to cover RDMA connection failures and fallback to TCP \(§\\lx@sectionsign[7](https://arxiv.org/html/2610.12815#S7)\)\.

## 4System Overview

Figure 2:Communication stack: Archon provides RPCs;ByteExpressmanages buffers and message transfers over hardware RDMA\.Figure[2](https://arxiv.org/html/2610.12815#S4.F2)shows the communication path between a ranker or trainer and a parameter server\. Applications issue RPCs through Archon, our existing RPC framework\. Archon usesByteExpressthrough socket\-like endpoints, an abstraction also used by rack\-scale PCIe interconnects\[[16](https://arxiv.org/html/2610.12815#bib.bib7)\]\.BEtranslates message transfers into verbs operations on hardware RNICs\. Applications therefore retain an RPC interface rather than directly managing verbs, QPs, or remote addresses\.

BEseparates logical connections from physical transport resources\. A*socket*represents a logical connection, such as an RPC channel\. An*engine*owns one RC QP and its associated transport state\. Sockets with the same remote destination can share an engine, and a scheduler coordinates their transfers\. This multiplexing reduces the number of QPs without requiring applications to share RPC channels explicitly\.

Thebemallocallocator supplies registered buffers to both the application and communication layers\. On receipt, buffer ownership can pass to an application thread without a copy\. For large outgoing messages,BEexchanges destination\-buffer metadata and then transfers the payload using RDMA writes\. It divides these transfers into fragments so that the scheduler can interleave sockets and enforce a per\-engine send window\.

The design uses three distinct forms of transport control\. Receive credits track posted receive capacity forSENDandWRITE\_WITH\_IMM; a software send window limits outstanding bytes; and RNIC DCQCN controls the hardware sending rate\. PFC remains enabled in the fabric as a link\-level protection mechanism\. Archon provides recovery above these layers through timeouts, retries, replica selection, and TCP fallback\. The following sections describe how these components address performance, scalability, and availability\.

## 5Building a High\-Performance Communication Engine

This section describes the data path, from transport selection and buffer allocation to message transfer and work\-request processing\. The key requirement is to preserve the benefits of hardware RDMA while supporting variable\-size messages and application\-controlled buffer lifetimes\.

### 5\.1Choosing RC for Large\-Message IO

BEuses RC to retain hardware support for reliable, large\-message transfers\. UD avoids maintaining a dedicated transport connection to each peer, but its MTU\-sized messages require additional software processing for larger payloads\. The measurements in Table[2](https://arxiv.org/html/2610.12815#S5.T2)illustrate how this tradeoff changes with link rate and MTU\.

Link rateMTURC \(Gbps\)UD \(Gbps\)25G1024B23\.1522\.97100G1024B92\.2648\.29100G4096B97\.0395\.74200G4096B191\.36114\.70Table 2:RC and UD throughput measured withib\_send\_bwat different link rates and MTUs\.At 25 Gbps with a 1,024\-byte MTU, RC and UD both approach the link rate\. At 100 Gbps with the same MTU, RC reaches 92\.26 Gbps while UD reaches 48\.29 Gbps; increasing the MTU to 4,096 bytes brings both close to line rate\. At 200 Gbps with a 4,096\-byte MTU, RC reaches 191\.36 Gbps and UD reaches 114\.70 Gbps\. Thus, packet\-processing rate can constrain throughput as link bandwidth grows, even though UD performs well in other configurations\. These measurements support our choice of RC for the evaluated workload, rather than a general claim that UD is always bandwidth\-limited\. Section[6\.1](https://arxiv.org/html/2610.12815#S6.SS1)explains how we manage the connection\-state cost of that choice\.

### 5\.2RDMA Memory Management

To support zero\-copy transfers of large, variable\-size messages, we need dynamic allocation of registered memory\. Ideally, allocating and freeing RDMA buffers should be as straightforward as using a general\-purpose allocator such asjemalloc\[[8](https://arxiv.org/html/2610.12815#bib.bib41)\]ortcmalloc\[[28](https://arxiv.org/html/2610.12815#bib.bib40)\]\. We implement this interface by extending an existing allocator with memory registration and memory\-region tracking\.

We buildbemalloc\(ByteExpress malloc\) onrpmalloc\[[20](https://arxiv.org/html/2610.12815#bib.bib42)\], chosen for its compact implementation \(approximately 3,000 lines of C\) and allocation performance\. The underlying allocator obtains large blocks of sizeBB, for example 4 MB, from the operating system and subdivides them to serve allocation requests\. It can return a block when no allocations remain in use\. We use its block\-allocation and block\-release callbacks to invokeibv\_reg\_mr\(\)andibv\_dereg\_mr\(\), respectively, and store memory\-region information in the block metadata\. The same approach can be applied to allocators that expose comparable callbacks\.

Allocation interface\.bemallochides registration details behind a standalone interface while retaining the performance of the underlying allocator\. Figure[3](https://arxiv.org/html/2610.12815#S5.F3)shows a wrapper that exposes a global RDMA memory pool through static functions\. After initialization with an RDMA device context,Allocate\(\)returns registered memory,GetMR\(\)retrieves its memory\-region metadata inO⁡\(1\)O\(1\)time, andDeallocate\(\)returns the allocation to the pool\. The interface adds RDMA\-capable memory management rather than changingrpmalloc’s allocation algorithm\.

RdmaMemoryPool::Init\(ibv\_ctx\);

void\*block=RdmaMemoryPool::Allocate\(size\);

conststructibv\_mr\*mr=RdmaMemoryPool::GetMR\(block\);

RdmaMemoryPool::Deallocate\(block\);

Figure 3:bemallocinterface: allocate registered memory, look up memory\-region metadata, and release the allocation\.Registering large blocks also limits memory\-region proliferation\. Registering each small allocation separately can increase pressure on RNIC caches that hold memory\-region metadata\[[42](https://arxiv.org/html/2610.12815#bib.bib43),[25](https://arxiv.org/html/2610.12815#bib.bib37),[2](https://arxiv.org/html/2610.12815#bib.bib39),[47](https://arxiv.org/html/2610.12815#bib.bib15)\]\.bemallocinstead amortizes each registration across many allocations\. It also inheritsrpmalloc’s thread\-local and global caches, including support for freeing memory from a thread other than the allocating thread\. This supports the buffer\-ownership transfers used by our inference and training systems\.

\(a\)Large\-block operation rate\.\(b\)Large\-block peak memory\.\(c\)Small\-block operation rate\.\(d\)Small\-block peak memory\.
Figure 4:Allocator operation rate and peak memory versus thread count\. Large allocations span 64 KB–1 MB; small allocations span 16 B–16 KB\.Registration cost and pool reuse\.Registration remains necessary when the allocator acquires a new block, and deregistration is required when it releases one\. Their cost is amortized across allocations from that block\. Once the registered pool reaches a stable size, subsequent allocation and deallocation operations can be served from cached blocks without further registration\. Thus, registration is largely removed from the steady\-state allocation path; it is not free during pool growth or release\.

Figure[5](https://arxiv.org/html/2610.12815#S5.F5)illustrates this behavior on one online inference node\. Sampled every 30 seconds from startup, its registered\-memory footprint reaches approximately 2 GB in the first interval, then rises slowly to approximately 2\.2 GB and stabilizes\. Allocation and deallocation continue after this point, but cached registered blocks serve those operations\.

Figure 5:Registered\-memory footprint on one inference node, sampled every 30 s from startup and stabilizing near 2\.2 GB\.Allocator evaluation\.Figure[4](https://arxiv.org/html/2610.12815#S5.F4)comparesbemallocwith general\-purpose allocators\. The large\-allocation benchmark uses exponentially distributed sizes in the range 64 KB–1 MB; the small\-allocation benchmark uses sizes in the range 16 B–16 KB with a linear size distribution\. Both vary the thread count from one to eight and include approximately 10% cross\-thread operations\.bemalloctracksrpmallocclosely in operations per second and peak memory usage, indicating that the added RDMA bookkeeping has little overhead in these tests\. They provide the highest operation rates among the evaluated allocators, at the cost of peak memory usage up to approximately 50% higher than the alternatives\.

### 5\.3Message I/O

#### 5\.3\.1Zero\-Copy Buffer Ownership

bemalloclets the application and communication layers share the same buffers\. On receipt, the RNIC places message data directly into registered buffers allocated on demand\. The communication layer then passes those buffers to the application, which can retain them, hand them to another thread, and free them after processing\. This avoids coupling application buffer lifetimes to a fixed receive ring\. Designs based on fixed receive buffers\[[6](https://arxiv.org/html/2610.12815#bib.bib32),[22](https://arxiv.org/html/2610.12815#bib.bib17),[33](https://arxiv.org/html/2610.12815#bib.bib34),[48](https://arxiv.org/html/2610.12815#bib.bib35)\]require a different ownership discipline to reuse those buffers safely\.

On the send path, applications allocate network\-bound payloads throughbemalloc, making them directly accessible to the RNIC\. Other application allocations need not change\. Here,*zero\-copy*means avoiding additional payload copies between application buffers and communication buffers; it does not imply that all model processing or serialization is copy\-free\.

#### 5\.3\.2Transferring Large Messages

Figure 6:Large\-message transfer: exchange destination\-buffer metadata, write payload fragments, and notify the receiver with a finalWRITE\_WITH\_IMM\.BEuses one\-sidedWRITEoperations for large payloads\. Figure[6](https://arxiv.org/html/2610.12815#S5.F6)shows the transfer sequence\. The sender first obtains the destination address and remote memory key through a request/response exchange usingSEND/RECV\. The receiver allocates the destination buffer throughbemalloc\. This exchange adds one round\-trip time \(RTT\) before the payload transfer\. If an application can supply valid remote\-buffer metadata in advance, as with some statically allocated tensor buffers,BEcan omit that exchange\.

After exchanging metadata,BEwrites the payload in fragments, with scheduling opportunities between fragments\. For a destination addressppand fragment sizeSS, successive fragments are written topp,p\+Sp\+S, and so on, producing a contiguous message at the receiver\. Fragmentation lets the scheduler interleave sockets and enforce the send window described in§\\lx@sectionsign[6\.3](https://arxiv.org/html/2610.12815#S6.SS3); the receiver does not need to copy fragments into a separate reassembly buffer\.

BEsends the final fragment withWRITE\_WITH\_IMM, which carries 32 bits of immediate data and generates a receiver\-side completion\. The receiver uses this completion to detect that the message has arrived\. By contrast, polling a flag embedded in a receive buffer\[[6](https://arxiv.org/html/2610.12815#bib.bib32),[23](https://arxiv.org/html/2610.12815#bib.bib2),[22](https://arxiv.org/html/2610.12815#bib.bib17),[31](https://arxiv.org/html/2610.12815#bib.bib18)\]requires a safe ordering relationship between the payload and the flag\.

This distinction matters when memory regions use relaxed ordering\. Relaxed ordering can permit incoming writes to reach memory out of order, so observing a flag alone is not sufficient evidence that the payload is visible\. Completion semantics still ensure data visibility, including prior transfers\[[45](https://arxiv.org/html/2610.12815#bib.bib48)\]\. Our notification scheme relies on those completion semantics for fragments issued on the same RC QP\. It also lets an I/O thread poll a shared CQ instead of inspecting a separate flag for every outstanding message\. The performance benefit of relaxed ordering is hardware\-dependent; we do not isolate it in the evaluation\.

### 5\.4RDMA Engine

Figure 7:RDMA engine: one QP, receive resources, and a work\-request chain partitioned into in\-flight, pending, and free entries\.EachBEengine owns one QP and its associated state, including a work\-request chain and receive buffers \(Figure[7](https://arxiv.org/html/2610.12815#S5.F7)\)\. It is the lowest software layer: it posts verbs operations and processes completions\. Above it, sockets represent logical connections and a scheduler coordinates their I/O\. We use*engine*when referring to this software state and*QP*when referring to the RDMA resource\. Section[6\.1](https://arxiv.org/html/2610.12815#S6.SS1)describes socket multiplexing and scheduling\.

#### 5\.4\.1Work Request I/O

The engine uses preallocated resources and batching to reduce data\-path overhead\. Its send\-side work requests form a circular chain whose length equals the QP send\-queue depth\. Tracking occupied entries avoids dynamic work\-request allocation and prevents the engine from posting more requests than the send queue can hold\. We use relatively shallow queues, for example a depth of 32, provided that they allow enough outstanding work to sustain the target throughput\.

Work\-request chain\.The chain has three contiguous segments:*in flight*,*pending*, and*free*\. In\-flight requests have been posted to the RNIC and await completion; processing their completions frees the corresponding slots\. Pending requests have been prepared but not posted\. The engine submits them together with oneibv\_post\_send\(\)call, using doorbell batching\[[23](https://arxiv.org/html/2610.12815#bib.bib2)\]\. Only the last request in a batch requests a completion notification, reducing completion\-processing overhead\. When free slots are available, the scheduler polls sockets for work; each selected socket prepares a request of at most one fragment\. The engine tracks the segment boundaries in the circular chain\.

#### 5\.4\.2Flow Control

BEusesSEND/RECVfor control messages and small payloads\. ASENDconsumes a posted receive request at the peer; if none is available, the peer reports receiver not ready \(RNR\), causing retries and potentially a transport error\.WRITE\_WITH\_IMMalso consumes a receive request\. We therefore track receive capacity explicitly\. This flow control protects endpoint receive queues; it is separate from the network congestion control in§\\lx@sectionsign[6\.3](https://arxiv.org/html/2610.12815#S6.SS3)\.

Receive\-credit flow control\.Inspired by receiver\-driven transport\[[37](https://arxiv.org/html/2610.12815#bib.bib47)\], each sender maintains credits for available receive requests at its peer\. The endpoints exchange the initial credit count during connection setup\. Posting aSENDorWRITE\_WITH\_IMMconsumes one credit; ordinaryWRITEoperations do not consume credits because they do not use receive requests\. The sender waits when it lacks the required credit\.

The receiver replenishes capacity in batches\. After consumingkkreceive requests, it postskkreplacements with oneibv\_post\_recv\(\)call and sends a credit update to the peer\. Batching reduces the frequency of posting calls and credit messages\. We usek=d/2k=d/2, whereddis the receive\-queue depth\. Choosingk<dk<dleaves capacity available while an update is in transit, although a sufficiently delayed update can still stall the sender\. This scheme targets RNR avoidance with dedicated receive queues\. The optional shared\-receive\-queue configuration trades this per\-peer accounting for shared buffer capacity \(§\\lx@sectionsign[6\.2\.2](https://arxiv.org/html/2610.12815#S6.SS2.SSS2)\)\.

Figure 8:ByteExpress and eRPC latency with one message outstanding\.Figure 9:Peak throughput by link rate/MTU—ByteExpress vs\. eRPC vs\. RC/UD perftest\.Figure 10:Throughput vs\. fragment size on 25 and 100 Gbps links\.Figure 11:512\-byte message rate vs\. QP count in eight IO threads w/ or w/o contention\.

### 5\.5Evaluation

We evaluateByteExpresswith synthetic workloads\. Figure[11](https://arxiv.org/html/2610.12815#S5.F11)compares mean and P99 message latency with eRPC\[[25](https://arxiv.org/html/2610.12815#bib.bib37)\]for message sizes from 32 B to 16 KB\. The latency experiment permits one message to remain outstanding at a time\.

Figure[11](https://arxiv.org/html/2610.12815#S5.F11)compares peak throughput across four link\-rate/MTU configurations: 25 Gbps/1 KB, 100 Gbps/1 KB, 100 Gbps/4 KB, and 200 Gbps/4 KB\. It includes eRPC and RC/UDperftestbaselines\. The throughput setup uses one QP to send 1 MB messages between two nodes, with a 32\-byte response per message and sufficient concurrency to saturate the path\.

ByteExpressachieves latency and throughput comparable to or better than the plotted alternatives in these configurations\. These results measure the complete communication path; they do not isolate the contribution of each optimization\. The latency and throughput plots use different message\-size settings and should be interpreted separately\.

## 6Scaling Connections & Concurrent PS Ops

Scaling the communication path requires managing both connection state and aggregate traffic\. This section describes QP multiplexing, thread\-aware resource management, and a software send window for incast workloads\.

### 6\.1QP Management

RNICs have limited cache space for QP context\. When the active working set exceeds that capacity, the NIC must fetch missing context from host memory through DMA, increasing overhead\.BEreduces this pressure by multiplexing logical connections onto QPs\.

Archon keeps multiple requests outstanding through RPC channels, each with its own socket\. With TCP, each socket has a separate transport connection\. WithBE, sockets directed to the same remote address can share one engine \(§\\lx@sectionsign[5\.4](https://arxiv.org/html/2610.12815#S5.SS4)\)\.

Multiplexing sockets onto QPs\.BEmaintains a connection table that maps remote addresses to engines\. When an application callsconnect\(\),BEattaches the socket to an existing engine for that address, if available\. Otherwise, it creates an engine and reports connection success asynchronously after setup completes\. This allows one QP to serve multiple logical connections to each remote address\.

Sharing an engine introduces scheduling head\-of\-line \(HoL\) blocking: a large message from one socket can delay smaller messages from another\.BEfragments messages so that the scheduler can interleave their work requests\. Fragment size trades scheduling granularity against per\-request overhead\. Figure[11](https://arxiv.org/html/2610.12815#S5.F11)shows that approximately 2 KB fragments can saturate the tested 25 Gbps link with one QP, whereas fragments larger than 8 KB are needed for near\-line\-rate throughput on the tested 100 Gbps link\.

Each socket queues outgoing messages\. When selected by the scheduler, it prepares a work request for at most one fragment by supplying that fragment’s address and length; no payload copy is required\. Under round\-robin scheduling withnnactive sockets and maximum fragment sizeSS, at most\(n−1\)​S\(n\-1\)Sbytes from other sockets are selected between a socket’s turns\. This is a bound on scheduler service between turns, not on end\-to\-end latency: already\-posted work, congestion, or loss recovery can still stall the shared QP\. We use round\-robin scheduling because sockets have comparable roles in the evaluated systems; other policies could prioritize latency\-sensitive traffic\.

Figure[14](https://arxiv.org/html/2610.12815#S6.F14)adds large flows to an engine that already carries a small flow\. With 4 KB fragments, the small flow’s latency still increases as contention grows, but remains substantially lower than without fragmentation\. Fragmentation therefore mitigates message\-scheduling HoL blocking; it does not eliminate blocking caused by transport ordering, packet loss, or congestion control\.

Figure 12:RDMA engine assignment—each engine \(and its QPs\) belongs to one I/O thread; one thread manages multiple engines\.Figure 13:Small\-flow latency under competing large flows, w/ or w/o 4KB frag\.Figure 14:Synthetic incast topology showing workers, parameter servers, bandwidth\-test flows, incast traffic, and PFC propagation\.

### 6\.2Avoiding Contention in Multithreaded I/O

#### 6\.2\.1Thread Ownership and Provider Resources

Figure[12](https://arxiv.org/html/2610.12815#S6.F12)shows the mapping of engines to I/O threads\. Each engine is bound to one thread, so operations on its data structures do not require locks between I/O threads\. A thread can manage many engines\. We assign engines in round\-robin order, which is sufficient for the evaluated workload\.

Contention inside the RDMA provider\.Single\-thread ownership of each engine did not eliminate all contention\. In an experiment with eight threads and a varying number of QPs, the 512\-byte message rate dropped sharply beyond 64 QPs \(Figure[11](https://arxiv.org/html/2610.12815#S5.F11)\)\. Profiling showed substantial time in spin locks insideibv\_post\_send\(\)\. QPs assigned to different application threads were sharing provider\-side structures: the provider did not automatically inherit the application’s ownership arrangement\.

Thread\-aware resource allocation\.We use a separate thread domain \(TD\) for each I/O thread to communicate the ownership arrangement to the verbs provider\. Thread domains permit provider optimizations for resources accessed by a single thread\[[44](https://arxiv.org/html/2610.12815#bib.bib49)\]\. This requires the application to respect that ownership discipline\. Separating threads into independent protection domains \(PDs\) would complicate cross\-thread use of registered memory; instead, our configuration retains a common protection domain while associating resources with their respective thread domains\.

#### 6\.2\.2Sharing Queues Within an I/O Thread

Single\-thread ownership also lets engines share selected resources without introducing locks between I/O threads\. Table[3](https://arxiv.org/html/2610.12815#S6.T3)summarizes the allocation: QPs remain associated with remote addresses, while CQs and TDs are allocated per thread and registered memory belongs to a common PD\. This reduces per\-QP overhead, although QP\-specific state still grows with connection count\.

ResourceAllocationQPOne per remote addressRQOne per QP, without SRQSRQOne per thread, when enabledCQOne per threadTDOne per threadMROne per registered blockPDOne globallyTable 3:Allocation of RDMA resources inBE\. Dedicated RQs and per\-thread SRQs are alternative configurations\.Completion queues\.Each I/O thread maintains one CQ for the send and receive queues of its assigned QPs\. It retrieves completion queue entries \(CQEs\) in batches withibv\_poll\_cq\(\)and uses each entry’s QP number to identify the corresponding engine\. This extends the engine’s batched completion processing \(§\\lx@sectionsign[5\.4](https://arxiv.org/html/2610.12815#S5.SS4)\) across connections and avoids polling a separate CQ for every QP\.

Shared receive queues\.Dedicated RQs reserve receive buffers for individual peers\. As an alternative,BEcan create one shared receive queue \(SRQ\) per I/O thread, allowing all engines assigned to that thread to draw from a common buffer pool instead of using dedicated RQs\. This improves buffer utilization but changes the flow\-control contract\.

The credit scheme in§\\lx@sectionsign[5\.4\.2](https://arxiv.org/html/2610.12815#S5.SS4.SSS2)tracks per\-peer receive capacity and does not directly apply to a shared pool\. Engines can be created and destroyed dynamically, so precise allocation of shared capacity requires additional coordination\. In SRQ mode, we use larger queues and configure RNR timeouts and retry counts to reduce the impact of receive\-buffer exhaustion\. This mitigates RNR events but does not enforce the per\-peer receive\-capacity bound of the dedicated\-queue credit scheme\. The choice is therefore between explicit per\-peer capacity accounting and more flexible sharing of receive buffers\.

### 6\.3Application\-Level Congestion Control

Incast places a different limit on scale: many individually efficient senders can collectively overwhelm a receiver’s network path\. We supplement hardware congestion control with a software bound on outstanding bytes\.

Iterative incasts in training\.Both inference and training exhibit fan\-in and fan\-out, but repeated parameter synchronization in training can sustain concurrent transfers to the same destinations\. In the training workload considered here, mean and P99 message sizes are 20 KB and 50 KB\. ConcurrentFetch/Pushrequests can produce substantial incast congestion and frequent received PFC pause frames\.

Our objective is to sustain concurrent transfers with low tail latency while reducing PFC activity\. Absence of PFC pauses does not imply absence of congestion; it indicates that the measured workload did not trigger that protection mechanism\. We use request P99 latency as a proxy for communication stalls, rather than as a direct measure of end\-to\-end training throughput\.

Prior work has also identified congestion challenges in RDMA networks\[[25](https://arxiv.org/html/2610.12815#bib.bib37),[51](https://arxiv.org/html/2610.12815#bib.bib26),[10](https://arxiv.org/html/2610.12815#bib.bib27)\]\. In our deployment, DCQCN alone did not react quickly enough to the observed incasts\. Its rate\-control dynamics motivate additional control over how much data software makes outstanding\. Software transports such as eRPC\[[25](https://arxiv.org/html/2610.12815#bib.bib37)\]offer a different design point; we retain RC offload and add a software admission bound above it\.

A per\-engine send window\.BEbounds the bytes in flight on each engine using a configurable send window, denotedCwndin the experiments\. Together with fragmentation, this limits the amount of data an engine can make outstanding\. DCQCN continues to regulate the RNIC’s sending rate, and PFC remains enabled in the fabric\. RC supplies transport reliability\. These roles should not be conflated: the software window is an outstanding\-data limit, DCQCN is a rate\-control mechanism, and PFC is link\-level backpressure\.

Fragment and window sizes are chosen according to message sizes and incast scale\. This configuration\-based approach suits our training workloads with relatively stable traffic patterns, but it is not an adaptive congestion\-control algorithm\. A per\-engine bound also does not impose a fixed bound on aggregate traffic as the number of active engines grows\. We have not established the stability or fairness of the combined controls across arbitrary workloads\.

Figure[14](https://arxiv.org/html/2610.12815#S6.F14)shows the synthetic worker/parameter\-server setup, and Figure[15](https://arxiv.org/html/2610.12815#S6.F15)reports bandwidth and request latency\. Table[4](https://arxiv.org/html/2610.12815#S6.T4)reports the server count before PFC is observed for each window setting\. Reducing the window increases the reported count from 63 without software control to 1,218 with a 4 KB window\. The experiment illustrates the effect of limiting outstanding data in this setup; it does not establish a universal PFC\-free scale limit\.

\(a\)Worker and victim bandwidth\.\(b\)P50 and P99 pull latency\.
Figure 15:PS performance in the synthetic inference workloads with no software window or 4, 8, and 16 KB windows\.Window sizeServers before PFC4 KB1,2188 KB56716 KB273No software control63Table 4:Reported server count before PFC is observed, by software window size\.

## 7Maintaining Service Availability

### 7\.1Transport Reliability & Service Recovery

Online inference must return results within service latency targets, while training must make progress despite failures\. Fan\-out makes both sensitive to slow sub\-requests: a single stalled destination can delay a larger operation\. RC handles transport\-level delivery on a functioning connection, but recovery from node failures, persistent path failures, and unavailable replicas requires decisions above the transport layer\.

Network monitoring can help locate faults\[[12](https://arxiv.org/html/2610.12815#bib.bib45),[1](https://arxiv.org/html/2610.12815#bib.bib46)\], and prior deployments combine network and application mechanisms for recovery\[[9](https://arxiv.org/html/2610.12815#bib.bib21)\]\. We follow the same general principle: retain service\-level recovery in Archon and add the RDMA\-specific monitoring and fallback mechanisms needed by our deployment\.

### 7\.2Deployment and Recovery Mechanisms

Archon supported production inference and training before the RDMA deployment, including RPC\-level availability policies and load balancing\. IntegratingBEpreserves those policies\. Before deployment, we test RDMA\-equipped machines with Collie\[[26](https://arxiv.org/html/2610.12815#bib.bib13)\]to identify performance anomalies\. We also use a watchdog\-based approach to contain PFC storms\[[11](https://arxiv.org/html/2610.12815#bib.bib1)\]\.

Replication across failure domains\.Rankers and parameter servers are deployed across multiple geographic data centers\. Within a data center, parameter\-server replicas are placed in different mini\-pods, each replica containing thennshards described in§\\lx@sectionsign[2\.1](https://arxiv.org/html/2610.12815#S2.SS1)\. The network between mini\-pods within a big\-pod supports the lossless RDMA configuration\. Replication provides alternative service destinations when a node or path fails; integrated network monitoring helps locate the underlying problem\.

Connection monitoring\.BEdetects QP errors through completion polling and verbs return values\. It also sends keepalive messages to detect failures such as a remote kernel panic or power loss\. Each engine periodically sends its peer a monotonically increasing sequence number\. If the peer’s sequence number does not advance before the keepalive timeout, the engine enters an error state, notifies the upper layer, and reclaims connection resources\. The send interval and failure\-detection timeout must accommodate normal scheduling delays\.

RPC\-assisted recovery\.Archon uses separate connection and request timeouts, chosen for the cluster topology and service requirements\. Example thresholds areTc=10T\_\{c\}=10ms for connections andTr=40T\_\{r\}=40ms for requests\. A timed\-out request may be retried probabilistically to tolerate transient failures\. These transport and request timeouts serve different purposes from the keepalive mechanism\.

Recovery proceeds through several options\. First, the system attempts to change the connection path using the RNIC’s port\-affinity support\[[41](https://arxiv.org/html/2610.12815#bib.bib44)\]; the reported deployment uses MLNX OFED 5\.0 or later with compatible firmware\. If reconnecting does not resolve the failure, Archon selects a nearby replica in another mini\-pod\. If RDMA communication remains unavailable, it falls back to TCP, following the general approach used in prior RDMA deployments\[[9](https://arxiv.org/html/2610.12815#bib.bib21)\]\. In the reported deployment, this fallback restores high throughput within one minute\. Section[9](https://arxiv.org/html/2610.12815#S9)discusses the need to characterize recovery more systematically\.

Archon also temporarily quarantines destinations with repeated failures\. For example, within a five\-minute window, more thanM=30M=30connection failures orN=50N=50request failures to a node can place it on an exclusion list\. Other replicas serve requests while engineers investigate the node offline\. The effectiveness of this policy depends on replica health and available capacity\.

These mechanisms describe the recovery architecture, not an end\-to\-end availability guarantee\. In particular, retry safety for state\-changing operations such asPushdepends on RPC and application semantics\. The measurements in this paper do not establish exactly\-once execution or quantify availability under all failure modes\.

![Refer to caption](https://arxiv.org/html/2610.12815v1/fig/real-world/ps-latency-toutiao-v2.jpg)Figure 16:PS performance in a production recommendation inference workload: mean and P99 for TCP and RDMA in microseconds, sampled at one\-minute intervals for the busiest rankers\.

## 8Production Deployment and Evaluation

At the time of this study, ByteDance had deployed the RDMA\-enabled inference and training systems in tens of data\-center clusters over a period of more than two years, supporting services including Toutiao, TikTok, and Xigua Video\. This section reports CPU utilization and request latency from two production workloads\.

Our baseline is the deployed Linux kernel TCP/IP stack\. It provides an operational reference for the migration, but the available results do not compare all possible TCP tuning choices or alternative user\-space transports\. The microbenchmarks in§\\lx@sectionsign[5\.5](https://arxiv.org/html/2610.12815#S5.SS5)provide separate comparisons with eRPC and RDMA test baselines\.

### 8\.1Online Inference

CPU metricRDMATCPReductionTotal CPU usage51\.31%57\.45%10\.69%Software interrupts0\.16%1\.70%90\.6%System4\.18%7\.08%41\.0%User47\.0%48\.7%3\.5%Table 5:CPU usage during the peak\-load Toutiao inference interval\. Reductions are relative to TCP\.Figure[16](https://arxiv.org/html/2610.12815#S7.F16)shows a production dashboard for a Toutiao recommendation service\. The cluster contains 300 parameter servers and 124 rankers, each equipped with a 25 Gbps Mellanox ConnectX\-5 NIC\. We measure mean and 99th\-percentile \(P99\) latency for parameterFetchrequests\. The RDMA and TCP measurements are reported at matched operation loads\.

LatencyRDMATCPReductionMean8\.1 ms12\.6 ms35\.7%P9935\.5 ms63\.1 ms43\.7%Table 6:PS Pull latency at inference peak\.The relative latency gap is largest during peak load\. Tables[5](https://arxiv.org/html/2610.12815#S8.T5)and[6](https://arxiv.org/html/2610.12815#S8.T6)summarize the 21:00–22:00 interval\. RDMA reduces total CPU usage from 57\.45% to 51\.31%, a decrease of 6\.14 percentage points, or 10\.69% relative to TCP\. The largest relative reductions are in software\-interrupt and system CPU time, consistent with reducing kernel networking overhead\. The results do not separately quantify how the saved CPU capacity is used by the application\.

Mean fetch latency decreases by 4\.5 ms \(35\.7%\), and P99 latency decreases by 27\.6 ms \(43\.7%\)\. These are parameter\-fetch measurements, rather than end\-to\-end recommendation latency\. They show that reducing communication overhead benefits the measured service, while leaving open how much of the gap a differently configured TCP baseline could recover\.

### 8\.2Offline Training

MetricTCPRDMAHW onlyRDMAHW \+ SWFetchP99650 ms1\.04 s338 msPushP991\.03 s1\.12 s268 msPFC durationN/A1 s per 30 s0Loss indicator1%–4%3K NAKs per 30 s0Table 7:Metrics for the busiest trainers\. HW: hardware congestion control; SW: software send window\. TCP loss rates and RDMA NAK counts are not directly comparable\.The training evaluation runs an internal production job on TCP and RDMA clusters with 124 parameter servers and 346 trainers, each equipped with a 25 Gbps Mellanox ConnectX\-5 NIC\. Trainers repeatedly issueFetchandPushrequests to retrieve parameters and submit updates\.

Table[7](https://arxiv.org/html/2610.12815#S8.T7)distinguishes RDMA with hardware congestion control alone from RDMA with the additionalBEsoftware window\. Both RDMA configurations retain DCQCN and the fabric’s PFC configuration\. Without the software window, the measured P99 fetch and push latencies exceed those of TCP, and the deployment observes PFC pauses\. With the window, P99 fetch latency falls from 1\.04 s to 338 ms\. P99 push latency is 268 ms, a 74\.0% reduction relative to TCP’s 1\.03 s\. No PFC pauses or negative acknowledgments are reported during the observation interval\. These measurements support bounding outstanding data for this workload; they do not show that the fabric is congestion\-free or that PFC can be disabled safely\.

## 9Deployment Lessons in Production

Our experience deploying RDMA in production ML inference and training systems yielded several practical lessons\.

Integrate RDMA buffer ownership across layers\.The practical value ofbemallocis the buffer\-ownership contract it enables across the application and transport\. Registered memory becomes useful to applications when allocation, serialization, transfer, and reclamation follow a consistent lifetime model\. Integrating this contract beneath Archon’s RPC interface allows applications to benefit from zero\-copy transfers while retaining their existing communication abstraction\.

Control receiver capacity and outstanding traffic separately\.Receive credits protect posted receive capacity, while the software send window bounds outstanding data above DCQCN and PFC\. In the production training workload, hardware congestion control alone left P99 latency above the deployed TCP baseline\. Adding the software window reduced P99 push latency to 268 ms, with no PFC pauses observed during the measurement interval \(Table[7](https://arxiv.org/html/2610.12815#S8.T7)\)\. The operational lesson is to assign each control a specific resource to protect\.

Align RDMA resource sharing with IO thread ownership\.Assigning each engine to one I/O thread removed contention in our own data structures, but provider\-side spin locks still limited message rate as QP counts grew\. Thread domains aligned provider resources with application ownership, while per\-thread queue sharing reduced resource overhead \(§\\lx@sectionsign[6\.2](https://arxiv.org/html/2610.12815#S6.SS2)\)\. Effective scaling therefore required inspecting contention across the application–provider boundary\.

Keep RDMA transport reliability close to service policy\.Archon retained responsibility for request timeouts, replica selection, and TCP fallback, whileBEsupplied connection\-error and keepalive signals \(§\\lx@sectionsign[7](https://arxiv.org/html/2610.12815#S7)\)\. This division lets recovery use service context, including replica placement across failure domains and the exclusion of repeatedly failing destinations\. Preserving these policies was a central part of integrating RDMA into an existing production service\.

## 10Related Work

RDMA systems and machine learning\.RDMA has been used in key\-value stores\[[22](https://arxiv.org/html/2610.12815#bib.bib17),[31](https://arxiv.org/html/2610.12815#bib.bib18)\], distributed transactions\[[49](https://arxiv.org/html/2610.12815#bib.bib19),[7](https://arxiv.org/html/2610.12815#bib.bib36)\], distributed storage systems\[[14](https://arxiv.org/html/2610.12815#bib.bib3),[13](https://arxiv.org/html/2610.12815#bib.bib4)\], cloud security system\[[32](https://arxiv.org/html/2610.12815#bib.bib5)\], disaggregation\[[46](https://arxiv.org/html/2610.12815#bib.bib16),[19](https://arxiv.org/html/2610.12815#bib.bib12),[34](https://arxiv.org/html/2610.12815#bib.bib6)\], and RPC\[[25](https://arxiv.org/html/2610.12815#bib.bib37),[47](https://arxiv.org/html/2610.12815#bib.bib15)\]\. EFLOPS\[[5](https://arxiv.org/html/2610.12815#bib.bib20)\]studies RDMA\-enabled distributed training through system and algorithm co\-design, while Xueet al\.\[[50](https://arxiv.org/html/2610.12815#bib.bib14)\]integrate RDMA into distributed machine learning\. Rather, our focus is the operational RDMA co\-design across registered\-memory allocation, message transport, resource sharing, and RPC\-level recovery for PS\-based inference and training\.

Connection scaling\.QP context and resource sharing are established RDMA design concerns\[[23](https://arxiv.org/html/2610.12815#bib.bib2),[36](https://arxiv.org/html/2610.12815#bib.bib38)\]\. SkyRDMA\[[15](https://arxiv.org/html/2610.12815#bib.bib10)\]explores offloaded RDMA connection sharing policies at the hardware level in AI clouds\. Prior work also explores software\-defined scheduling of RNIC resources\[[18](https://arxiv.org/html/2610.12815#bib.bib9)\]and GPU fabrics\[[3](https://arxiv.org/html/2610.12815#bib.bib8)\]\. Dynamically Connected transport provides another approach: an initiator can communicate with multiple targets without a dedicated RC connection to each\[[40](https://arxiv.org/html/2610.12815#bib.bib50)\]\.BEinstead multiplexes logical sockets over RC QPs and uses thread domains to align provider resources with I/O\-thread ownership\.

Congestion control\.DCQCN\[[51](https://arxiv.org/html/2610.12815#bib.bib26)\], TIMELY\[[35](https://arxiv.org/html/2610.12815#bib.bib51)\], Swift\[[27](https://arxiv.org/html/2610.12815#bib.bib28)\], and HPCC\[[30](https://arxiv.org/html/2610.12815#bib.bib25)\]explore different congestion signals and transport\-control designs\. Instead, our software window provides application\-level control over outstanding RC traffic, layered above the existing DCQCN configuration and complementary to prior studies\.

## 11Conclusion

Deploying RDMA in production inference and training systems requires coordinating application interfaces, memory ownership, transport resources, and service recovery\.ByteExpresscombines an RDMA\-aware allocator and zero\-copy transfers with QP multiplexing, thread\-aware resource management, receive credits, and a software send window\. Archon retains the RPC\-level context needed for retries, replica selection, and TCP fallback\. The reported deployments demonstrate lower parameter\-fetch latency and CPU usage relative to the deployed TCP baseline, while exposing limits that hardware offload alone does not resolve\. The broader lesson is to assign each layer a clear responsibility and evaluate the resulting communication path under the concurrency and failure patterns of the target workload\.

## References

- \[1\]B\. Arzani, S\. Ciraci, L\. Chamon, Y\. Zhu, H\. H\. Liu, J\. Padhye, B\. T\. Loo, and G\. Outhred\(2018\)007: democratically finding the cause of packet drops\.In15th USENIX Symposium on Networked Systems Design and Implementation \(NSDI 18\),pp\. 419–435\.Cited by:[§7\.1](https://arxiv.org/html/2610.12815#S7.SS1.p2.1)\.
- \[2\]M\. Burke, S\. Dharanipragada, S\. Joyner, A\. Szekeres, J\. Nelson, I\. Zhang, and D\. R\. Ports\(2021\)PRISM: rethinking the rdma interface for distributed systems\.InProceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles,pp\. 228–242\.Cited by:[§5\.2](https://arxiv.org/html/2610.12815#S5.SS2.p4.1)\.
- \[3\]D\. Chen, Y\. Gu, Y\. Huang, C\. Pei, P\. Hua, Y\. Zhou, and Y\. Xiao\(2026\)Unlocking software\-defined gpu fabric scheduling in the llm era\.InProceedings of the 17th ACM SIGOPS Asia\-Pacific Workshop on Systems,pp\. 118–125\.Cited by:[§10](https://arxiv.org/html/2610.12815#S10.p2.1)\.
- \[4\]J\. Choquette, W\. Gandhi, O\. Giroux, N\. Stam, and R\. Krashinsky\(2021\)Nvidia a100 tensor core gpu: performance and innovation\.IEEE Micro41\(2\),pp\. 29–35\.Cited by:[§1](https://arxiv.org/html/2610.12815#S1.p2.1)\.
- \[5\]J\. Dong, Z\. Cao, T\. Zhang, J\. Ye, S\. Wang, F\. Feng, L\. Zhao, X\. Liu, L\. Song, L\. Peng,et al\.\(2020\)Eflops: algorithm and system co\-design for a high performance distributed training platform\.In2020 IEEE International Symposium on High Performance Computer Architecture \(HPCA\),pp\. 610–622\.Cited by:[§10](https://arxiv.org/html/2610.12815#S10.p1.1),[§2\.1](https://arxiv.org/html/2610.12815#S2.SS1.p2.1)\.
- \[6\]A\. Dragojević, D\. Narayanan, M\. Castro, and O\. Hodson\(2014\)FaRM: fast remote memory\.In11th USENIX Symposium on Networked Systems Design and Implementation \(NSDI 14\),pp\. 401–414\.Cited by:[§5\.3\.1](https://arxiv.org/html/2610.12815#S5.SS3.SSS1.p1.1),[§5\.3\.2](https://arxiv.org/html/2610.12815#S5.SS3.SSS2.p3.1)\.
- \[7\]A\. Dragojević, D\. Narayanan, E\. B\. Nightingale, M\. Renzelmann, A\. Shamis, A\. Badam, and M\. Castro\(2015\)No compromises: distributed transactions with consistency, availability, and performance\.InProceedings of the 25th symposium on operating systems principles,pp\. 54–70\.Cited by:[§10](https://arxiv.org/html/2610.12815#S10.p1.1)\.
- \[8\]Facebook\(2010\)Jemalloc memory allocator\.Note:[http://jemalloc\.net/](http://jemalloc.net/)Cited by:[§5\.2](https://arxiv.org/html/2610.12815#S5.SS2.p1.1)\.
- \[9\]Y\. Gao, Q\. Li, L\. Tang, Y\. Xi, P\. Zhang, W\. Peng, B\. Li, Y\. Wu, S\. Liu, L\. Yan,et al\.\(2021\)When cloud storage meets rdma\.In18th USENIX Symposium on Networked Systems Design and Implementation \(NSDI 21\),pp\. 519–533\.Cited by:[§7\.1](https://arxiv.org/html/2610.12815#S7.SS1.p2.1),[§7\.2](https://arxiv.org/html/2610.12815#S7.SS2.p5.1)\.
- \[10\]Y\. Gao, Y\. Yang, T\. Chen, J\. Zheng, B\. Mao, and G\. Chen\(2018\)Dcqcn\+: taming large\-scale incast congestion in rdma over ethernet networks\.In2018 IEEE 26th International Conference on Network Protocols \(ICNP\),pp\. 110–120\.Cited by:[§6\.3](https://arxiv.org/html/2610.12815#S6.SS3.p4.1)\.
- \[11\]C\. Guo, H\. Wu, Z\. Deng, G\. Soni, J\. Ye, J\. Padhye, and M\. Lipshteyn\(2016\)RDMA over commodity Ethernet at scale\.InProceedings of the 2016 ACM SIGCOMM Conference,pp\. 202–215\.Cited by:[§7\.2](https://arxiv.org/html/2610.12815#S7.SS2.p1.1)\.
- \[12\]C\. Guo, L\. Yuan, D\. Xiang, Y\. Dang, R\. Huang, D\. Maltz, Z\. Liu, V\. Wang, B\. Pang, H\. Chen,et al\.\(2015\)Pingmesh: a large\-scale system for data center network latency measurement and analysis\.InProceedings of the 2015 ACM Conference on Special Interest Group on Data Communication,pp\. 139–152\.Cited by:[§7\.1](https://arxiv.org/html/2610.12815#S7.SS1.p2.1)\.
- \[13\]B\. Huang, L\. Jin, Z\. Lu, M\. Yan, J\. Wu, P\. C\. Hung, and Q\. Tang\(2019\)RDMA\-driven mongodb: an approach of rdma enhanced nosql paradigm for large\-scale data processing\.Information Sciences502,pp\. 376–393\.Cited by:[§10](https://arxiv.org/html/2610.12815#S10.p1.1)\.
- \[14\]B\. Huang, L\. Jin, Z\. Lu, X\. Zhou, J\. Wu, Q\. Tang, and P\. C\. Hung\(2019\)BoR: toward high\-performance permissioned blockchain in rdma\-enabled network\.IEEE Transactions on Services Computing13\(2\),pp\. 301–313\.Cited by:[§10](https://arxiv.org/html/2610.12815#S10.p1.1)\.
- \[15\]Y\. Huang, L\. Gao, L\. Wang, Z\. Liu, C\. Pei, Y\. Guo, H\. Ren, X\. Li, J\. Zhang, C\. Pei,et al\.SkyRDMA: fully offloaded cloud rdma virtualization\.Cited by:[§10](https://arxiv.org/html/2610.12815#S10.p2.1)\.
- \[16\]Y\. Huang, Y\. Huang, M\. Yan, J\. Hu, C\. Liang, Y\. Xu, W\. Zou, Y\. Zhang, R\. Zhang, C\. Huang,et al\.\(2022\)An ultra\-low latency and compatible pcie interconnect for rack\-scale communication\.InProceedings of the 18th International Conference on emerging Networking EXperiments and Technologies,pp\. 232–244\.Cited by:[§4](https://arxiv.org/html/2610.12815#S4.p1.1)\.
- \[17\]Y\. Huang, Y\. Qiu, D\. Ding, P\. T\. J\. Kon, Y\. Zhang, Y\. Mao, A\. Bhatnagar, M\. Chowdhury, S\. Devadas, J\. Xing,et al\.\(2025\)Remote direct code execution\.InProceedings of the 24th ACM Workshop on Hot Topics in Networks,pp\. 300–307\.Cited by:[§2\.2](https://arxiv.org/html/2610.12815#S2.SS2.p3.1)\.
- \[18\]Y\. Huang, Y\. Qiu, Y\. Xiao, A\. Bhatnagar, S\. Ratnasamy, and A\. Chen\(2025\)Exposing rdma nic resources for software\-defined scheduling\.InProceedings of the 9th Asia\-Pacific Workshop on Networking,pp\. 1–8\.Cited by:[§10](https://arxiv.org/html/2610.12815#S10.p2.1)\.
- \[19\]Y\. Huang, Y\. Qiu, Z\. Yang, Y\. Dai, D\. Wu, F\. Lai, J\. Xing, and A\. Chen\(2025\)Towards fully disaggregated recommendation model serving\.InProceedings of the 16th ACM SIGOPS Asia\-Pacific Workshop on Systems,pp\. 38–45\.Cited by:[§10](https://arxiv.org/html/2610.12815#S10.p1.1)\.
- \[20\]M\. Jansson\(2017\)Rpmalloc \- general purpose memory allocator\.Note:[https://github\.com/mjansson/rpmalloc](https://github.com/mjansson/rpmalloc)Cited by:[§5\.2](https://arxiv.org/html/2610.12815#S5.SS2.p2.1)\.
- \[21\]Y\. Jiang, Y\. Zhu, C\. Lan, B\. Yi, Y\. Cui, and C\. Guo\(2020\)A unified architecture for accelerating distributed dnn training in heterogeneous gpu/cpu clusters\.In14th USENIX Symposium on Operating Systems Design and Implementation \(OSDI 20\),pp\. 463–479\.Cited by:[§1](https://arxiv.org/html/2610.12815#S1.p2.1)\.
- \[22\]A\. Kalia, M\. Kaminsky, and D\. G\. Andersen\(2014\)Using RDMA efficiently for key\-value services\.InProceedings of the 2014 ACM Conference on SIGCOMM,pp\. 295–306\.Cited by:[§10](https://arxiv.org/html/2610.12815#S10.p1.1),[§5\.3\.1](https://arxiv.org/html/2610.12815#S5.SS3.SSS1.p1.1),[§5\.3\.2](https://arxiv.org/html/2610.12815#S5.SS3.SSS2.p3.1)\.
- \[23\]A\. Kalia, M\. Kaminsky, and D\. G\. Andersen\(2016\)Design guidelines for high performance RDMA systems\.In2016 USENIX Annual Technical Conference \(USENIX ATC 16\),pp\. 437–450\.Cited by:[§10](https://arxiv.org/html/2610.12815#S10.p2.1),[§5\.3\.2](https://arxiv.org/html/2610.12815#S5.SS3.SSS2.p3.1),[§5\.4\.1](https://arxiv.org/html/2610.12815#S5.SS4.SSS1.p2.1)\.
- \[24\]A\. Kalia, M\. Kaminsky, and D\. G\. Andersen\(2016\)FaSST: fast, scalable and simple distributed transactions with two\-sided rdma datagram rpcs\.In12th USENIX Symposium on Operating Systems Design and Implementation \(OSDI 16\),pp\. 185–201\.Cited by:[§3](https://arxiv.org/html/2610.12815#S3.p3.1)\.
- \[25\]A\. Kalia, M\. Kaminsky, and D\. Andersen\(2019\)Datacenter RPCs can be general and fast\.In16th USENIX Symposium on Networked Systems Design and Implementation \(NSDI 19\),pp\. 1–16\.Cited by:[§10](https://arxiv.org/html/2610.12815#S10.p1.1),[§3](https://arxiv.org/html/2610.12815#S3.p3.1),[§5\.2](https://arxiv.org/html/2610.12815#S5.SS2.p4.1),[§5\.5](https://arxiv.org/html/2610.12815#S5.SS5.p1.1),[§6\.3](https://arxiv.org/html/2610.12815#S6.SS3.p4.1)\.
- \[26\]X\. Kong, Y\. Zhu, H\. Zhou, Z\. Jiang, J\. Ye, C\. Guo, and D\. Zhuo\(2022\)Collie: finding performance anomalies in RDMA subsystems\.In19th USENIX Symposium on Networked Systems Design and Implementation \(NSDI 22\),Renton, WA,pp\. 287–305\.External Links:ISBN 978\-1\-939133\-27\-4,[Link](https://www.usenix.org/conference/nsdi22/presentation/kong)Cited by:[§7\.2](https://arxiv.org/html/2610.12815#S7.SS2.p1.1)\.
- \[27\]G\. Kumar, N\. Dukkipati, K\. Jang, H\. M\. Wassel, X\. Wu, B\. Montazeri, Y\. Wang, K\. Springborn, C\. Alfeld, M\. Ryan,et al\.\(2020\)Swift: delay is simple and effective for congestion control in the datacenter\.InProceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication,pp\. 514–528\.Cited by:[§10](https://arxiv.org/html/2610.12815#S10.p3.1)\.
- \[28\]S\. Lee, T\. Johnson, and E\. Raman\(2014\)Feedback directed optimization of tcmalloc\.InProceedings of the workshop on Memory Systems Performance and Correctness,pp\. 1–8\.Cited by:[§5\.2](https://arxiv.org/html/2610.12815#S5.SS2.p1.1)\.
- \[29\]M\. Li, D\. G\. Andersen, J\. W\. Park, A\. J\. Smola, A\. Ahmed, V\. Josifovski, J\. Long, E\. J\. Shekita, and B\. Su\(2014\)Scaling distributed machine learning with the parameter server\.In11th USENIX Symposium on Operating Systems Design and Implementation \(OSDI 14\),pp\. 583–598\.Cited by:[§2\.1](https://arxiv.org/html/2610.12815#S2.SS1.p1.1)\.
- \[30\]Y\. Li, R\. Miao, H\. H\. Liu, Y\. Zhuang, F\. Feng, L\. Tang, Z\. Cao, M\. Zhang, F\. Kelly, M\. Alizadeh,et al\.\(2019\)HPCC: high precision congestion control\.InProceedings of the ACM Special Interest Group on Data Communication,pp\. 44–58\.Cited by:[§10](https://arxiv.org/html/2610.12815#S10.p3.1)\.
- \[31\]H\. Lim, D\. Han, D\. G\. Andersen, and M\. Kaminsky\(2014\)MICA: a holistic approach to fast in\-memory key\-value storage\.In11th USENIX Symposium on Networked Systems Design and Implementation \(NSDI 14\),pp\. 429–444\.Cited by:[§10](https://arxiv.org/html/2610.12815#S10.p1.1),[§5\.3\.2](https://arxiv.org/html/2610.12815#S5.SS3.SSS2.p3.1)\.
- \[32\]H\. Liu, J\. Xing, Y\. Huang, D\. Zhuo, S\. Devadas, and A\. Chen\(2023\)Remote direct memory introspection\.In32nd USENIX Security Symposium \(USENIX Security 23\),pp\. 6043–6060\.Cited by:[§10](https://arxiv.org/html/2610.12815#S10.p1.1)\.
- \[33\]Y\. Lu, J\. Shu, Y\. Chen, and T\. Li\(2017\)Octopus: an rdma\-enabled distributed persistent memory file system\.In2017 USENIX Annual Technical Conference \(USENIX ATC 17\),pp\. 773–785\.Cited by:[§5\.3\.1](https://arxiv.org/html/2610.12815#S5.SS3.SSS1.p1.1)\.
- \[34\]H\. Mei, L\. Gao, Y\. Zhu, L\. Wang, P\. Yang, C\. Pei, C\. Chen, Z\. Li, J\. Zhao, D\. Gu,et al\.\(2026\)DistDPU: a disaggregated dpu architecture for high\-performance and cost\-efficient ai clouds\.InProceedings of the ACM SIGCOMM 2026 Conference,pp\. 519–534\.Cited by:[§10](https://arxiv.org/html/2610.12815#S10.p1.1)\.
- \[35\]R\. Mittal, T\. Lam, N\. Dukkipati, E\. Blem, H\. Wassel, M\. Ghobadi, A\. Vahdat, Y\. Wang, D\. Wetherall, and D\. Zats\(2015\)TIMELY: RTT\-based congestion control for the datacenter\.InProceedings of ACM SIGCOMM,pp\. 537–550\.Cited by:[§10](https://arxiv.org/html/2610.12815#S10.p3.1)\.
- \[36\]S\. K\. Monga, S\. Kashyap, and C\. Min\(2021\)Birds of a feather flock together: scaling rdma rpcs with flock\.InProceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles,pp\. 212–227\.Cited by:[§10](https://arxiv.org/html/2610.12815#S10.p2.1)\.
- \[37\]B\. Montazeri, Y\. Li, M\. Alizadeh, and J\. Ousterhout\(2018\)Homa: a receiver\-driven low\-latency transport protocol using network priorities\.InProceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication,pp\. 221–235\.Cited by:[§5\.4\.2](https://arxiv.org/html/2610.12815#S5.SS4.SSS2.p2.1)\.
- \[38\]D\. Narayanan, M\. Shoeybi, J\. Casper, P\. LeGresley, M\. Patwary, V\. Korthikanti, D\. Vainbrand, P\. Kashinkunti, J\. Bernauer, B\. Catanzaro,et al\.\(2021\)Efficient large\-scale language model training on gpu clusters using megatron\-lm\.InProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis,pp\. 1–15\.Cited by:[§1](https://arxiv.org/html/2610.12815#S1.p2.1)\.
- \[39\]M\. Naumov, J\. Kim, D\. Mudigere, S\. Sridharan, X\. Wang, W\. Zhao, S\. Yilmaz, C\. Kim, H\. Yuen, M\. Ozdal,et al\.\(2020\)Deep learning training in facebook data centers: design of scale\-up and scale\-out systems\.arXiv preprint arXiv:2003\.09518\.Cited by:[§2\.1](https://arxiv.org/html/2610.12815#S2.SS1.p2.1)\.
- \[40\]NVIDIARDMA Aware Networks Programming User Manual\.Note:[https://docs\.nvidia\.com/rdma\-aware\-networks\-programming\-user\-manual\-1\-7\.pdf](https://docs.nvidia.com/rdma-aware-networks-programming-user-manual-1-7.pdf)Section 8\.6\.1, Dynamically Connected Transport\. Accessed October 8, 2026Cited by:[§10](https://arxiv.org/html/2610.12815#S10.p2.1)\.
- \[41\]NVIDIA\(2021\)MLNX\_OFED new features\.Note:[https://docs\.nvidia\.com/networking/display/MLNXOFEDv471001/Changes\+and\+New\+Features](https://docs.nvidia.com/networking/display/MLNXOFEDv471001/Changes+and+New+Features)Cited by:[§7\.2](https://arxiv.org/html/2610.12815#S7.SS2.p5.1)\.
- \[42\]NVIDIA\(2021\)Optimized memory access\.Note:[https://docs\.nvidia\.com/networking/display/MLNXOFEDv531001/Optimized\+Memory\+Access](https://docs.nvidia.com/networking/display/MLNXOFEDv531001/Optimized+Memory+Access)Cited by:[§5\.2](https://arxiv.org/html/2610.12815#S5.SS2.p4.1)\.
- \[43\]Y\. Peng, Y\. Zhu, Y\. Chen, Y\. Bao, B\. Yi, C\. Lan, C\. Wu, and C\. Guo\(2019\)A generic communication scheduler for distributed dnn training acceleration\.InProceedings of the 27th ACM Symposium on Operating Systems Principles,pp\. 16–29\.Cited by:[§1](https://arxiv.org/html/2610.12815#S1.p2.1)\.
- \[44\]rdma\-core contributorsibv\_alloc\_td\(3\): Allocate a Thread Domain\.Note:[https://github\.com/linux\-rdma/rdma\-core/blob/master/libibverbs/man/ibv\_alloc\_td\.3](https://github.com/linux-rdma/rdma-core/blob/master/libibverbs/man/ibv_alloc_td.3)Accessed October 8, 2026Cited by:[§6\.2\.1](https://arxiv.org/html/2610.12815#S6.SS2.SSS1.p3.1)\.
- \[45\]rdma\-core contributorsibv\_reg\_mr\(3\): Register or Deregister a Memory Region\.Note:[https://github\.com/linux\-rdma/rdma\-core/blob/master/libibverbs/man/ibv\_reg\_mr\.3](https://github.com/linux-rdma/rdma-core/blob/master/libibverbs/man/ibv_reg_mr.3)Accessed October 8, 2026Cited by:[§5\.3\.2](https://arxiv.org/html/2610.12815#S5.SS3.SSS2.p4.1)\.
- \[46\]Y\. Shan, S\. Tsai, and Y\. Zhang\(2017\)Distributed shared persistent memory\.InProceedings of the 2017 Symposium on Cloud Computing,pp\. 323–337\.Cited by:[§10](https://arxiv.org/html/2610.12815#S10.p1.1)\.
- \[47\]S\. Tsai and Y\. Zhang\(2017\)Lite kernel RDMA support for datacenter applications\.InProceedings of the 26th Symposium on Operating Systems Principles,pp\. 306–324\.Cited by:[§10](https://arxiv.org/html/2610.12815#S10.p1.1),[§5\.2](https://arxiv.org/html/2610.12815#S5.SS2.p4.1)\.
- \[48\]X\. Wei, Z\. Dong, R\. Chen, and H\. Chen\(2018\)Deconstructing rdma\-enabled distributed transactions: hybrid is better\!\.In13th USENIX Symposium on Operating Systems Design and Implementation \(OSDI 18\),pp\. 233–251\.Cited by:[§5\.3\.1](https://arxiv.org/html/2610.12815#S5.SS3.SSS1.p1.1)\.
- \[49\]X\. Wei, J\. Shi, Y\. Chen, R\. Chen, and H\. Chen\(2015\)Fast in\-memory transaction processing using RDMA and HTM\.InProceedings of the 25th Symposium on Operating Systems Principles,pp\. 87–104\.Cited by:[§10](https://arxiv.org/html/2610.12815#S10.p1.1)\.
- \[50\]J\. Xue, Y\. Miao, C\. Chen, M\. Wu, L\. Zhang, and L\. Zhou\(2019\)Fast distributed deep learning over RDMA\.InProceedings of the Fourteenth EuroSys Conference 2019,pp\. 1–14\.Cited by:[§10](https://arxiv.org/html/2610.12815#S10.p1.1),[§2\.1](https://arxiv.org/html/2610.12815#S2.SS1.p2.1)\.
- \[51\]Y\. Zhu, H\. Eran, D\. Firestone, C\. Guo, M\. Lipshteyn, Y\. Liron, J\. Padhye, S\. Raindel, M\. H\. Yahia, and M\. Zhang\(2015\)Congestion control for large\-scale rdma deployments\.ACM SIGCOMM Computer Communication Review45\(4\),pp\. 523–536\.Cited by:[§10](https://arxiv.org/html/2610.12815#S10.p3.1),[§2\.2](https://arxiv.org/html/2610.12815#S2.SS2.p5.1),[§6\.3](https://arxiv.org/html/2610.12815#S6.SS3.p4.1)\.