Skip to main content

What is RDMA (Remote Direct Memory Access)

RDMA (Remote Direct Memory Access) is a high-performance networking technology that allows direct memory access between computers over a network without involving the CPU, cache, or operating system of either the sender or the receiver.

What RDMA Does

Traditionally, when data is transferred between systems:

    It’s copied from user space to kernel space.

    Passed through the network stack.

    Received in the kernel and copied again to user space.

With RDMA, data can be read/written directly from the memory of one machine to another, bypassing the kernel and reducing CPU usage and latency.

How RDMA Works

    No CPU interrupts on the receiving side.

    No context switches or system calls during data transfer.

    Uses zero-copy principles.

    Memory regions are pre-registered with the NIC.

    The NIC (RDMA-capable) directly reads/writes from/to memory.

RDMA Protocols

    InfiniBand

       High-performance computing (HPC) standard.

        Low-latency, high-bandwidth.


    RoCE (RDMA over Converged Ethernet)


        Runs RDMA on standard Ethernet.

       Needs lossless Ethernet (DCB - Data Center Bridging).

    iWARP

        RDMA over standard TCP/IP stack.

       More compatible but slightly higher latency.


RDMA Advantages

Benefit Description

Ultra-low latency No kernel involvement or context switches.

Zero copy No intermediate memory buffers or CPU copying.

High throughput NIC handles data transfer directly.

Low CPU utilization Frees CPU for application-level processing.

RDMA vs Traditional Networking

Feature Traditional Networking RDMA

CPU Involvement High Low

Memory Copy Multiple copies Zero copy

System Calls Yes No (once set up)

Latency Higher Ultra-low

Performance Good Excellent


Hardware Requirements

To use RDMA, you typically need:

   RDMA-capable NICs (e.g., Mellanox, Intel, Broadcom)

    Lossless network support (especially for RoCE)

RDMA drivers and libraries:rdma-core, libibverbs, libmlx5 For programming: Verbs API, RDMA CM, or higher-level frameworks (e.g., DAOS, NVMe-oF, Libfabric)

Common Use Cases

    High-speed datacenter communication

    Distributed databases (e.g., CockroachDB, Oracle RAC)

    HPC clusters

    Storage systems (e.g., NVMe over Fabrics)

    Machine learning training clusters

    Real-time data replication


RDMA in Code (Simplified Concept)

Here’s a pseudo-view of RDMA vs traditional send:

// Traditional

send(socket, buffer, length, 0);  // system call, data copied

// RDMA

rdma_post_write(qp, local_addr, remote_addr, length);  // no CPU copy


Brian Wilson (GT1) 7-7-25

Comments

Post a Comment

Popular posts from this blog

Maximize the performance of your NVIDIA GeForce RTX 5070 on Windows 11

Maximize the performance of your NVIDIA GeForce RTX 5070 on Windows 11, you can optimize both hardware and software settings. Below is a step-by-step guide focusing on driver updates, NVIDIA Control Panel settings, overclocking with MSI Afterburner, and Windows 11 optimizations. Each step includes relevant code or configuration details wrapped in an <xaiArtifact> tag where applicable. ***Note that some steps involve configuration rather than code, but I’ll provide scripts where possible to automate or illustrate the process. Step 1: Update NVIDIA Drivers to the Latest Version Keeping your GPU drivers up to date ensures optimal performance and compatibility with games and applications, especially for the RTX 5070, which requires the latest Game Ready Drivers for DLSS 4 and Multi Frame Generation support. Action: Download and install the latest NVIDIA Game Ready Driver from the NVIDIA website or use the NVIDIA App. Why: The latest drivers (e.g., 572.47 WHQL) include o...
Post a Comment
Read more

WINGET: The Pros and Cons of Using Windows Package Manager for Software Updates

 Need to update your programs?  WINGET: The Pros and Cons of Using Windows Package Manager for Software Updates Maintaining up-to-date software is a key component of ensuring system security, stability, and performance on any Windows machine. As part of its modernization efforts, Microsoft introduced WINGET, the Windows Package Manager, a command-line tool designed to simplify the process of installing, updating, and managing applications. WINGET is particularly useful for IT professionals, power users, and system administrators looking for a more efficient way to maintain software across single machines or entire fleets. This article explores the benefits and limitations of using WINGET for software updates, along with the basic command-line syntax required to use it effectively. What Is WINGET? WINGET is a command-line utility for Windows that interacts with an open-source repository of software packages. It enables users to quickly install, update, and uninstall supported a...
Post a Comment
Read more

A Passage for a Republic Unraveling

So, I have been reading much poetry of late, Whitman, Poe, Plath, Ginsberg, and their kindred spirits. At the same time, I have been witnessing the melodrama unfolding in our once-great country, and I felt compelled to write the following in the spirit of these poets. Our nation stands in peril, not from enemies without, but from those within. A Passage for a Republic Unraveling By Brian Wilson I sing not of triumphs, not of the sea-wind nor the plow, not of the railways stretching across the prairie, nor of the bridges binding one shore to another, but of a nation folding inward, a republic gnawing its own bones. O America, I address you you who once declared liberty on parchment, you who thundered with voices of orators, you who bent iron for freedom’s sake, who sent your sons into fields of battle with liberty as a banner. Now you stand trembling, your knees buckling beneath your own weight. The halls of Congress echo not with reason, but with the clamor of children fighting over sc...
Post a Comment
Read more