What is RDMA (Remote Direct Memory Access)

RDMA (Remote Direct Memory Access) is a high-performance networking technology that allows direct memory access between computers over a network without involving the CPU, cache, or operating system of either the sender or the receiver.

What RDMA Does

Traditionally, when data is transferred between systems:

It’s copied from user space to kernel space.

Passed through the network stack.

Received in the kernel and copied again to user space.

With RDMA, data can be read/written directly from the memory of one machine to another, bypassing the kernel and reducing CPU usage and latency.

How RDMA Works

No CPU interrupts on the receiving side.

No context switches or system calls during data transfer.

Uses zero-copy principles.

Memory regions are pre-registered with the NIC.

The NIC (RDMA-capable) directly reads/writes from/to memory.

RDMA Protocols

InfiniBand

High-performance computing (HPC) standard.

Low-latency, high-bandwidth.

RoCE (RDMA over Converged Ethernet)

Runs RDMA on standard Ethernet.

Needs lossless Ethernet (DCB - Data Center Bridging).

iWARP

RDMA over standard TCP/IP stack.

More compatible but slightly higher latency.

RDMA Advantages

Benefit Description

Ultra-low latency No kernel involvement or context switches.

Zero copy No intermediate memory buffers or CPU copying.

High throughput NIC handles data transfer directly.

Low CPU utilization Frees CPU for application-level processing.

RDMA vs Traditional Networking

Feature Traditional Networking RDMA

CPU Involvement High Low

Memory Copy Multiple copies Zero copy

System Calls Yes No (once set up)

Latency Higher Ultra-low

Performance Good Excellent

Hardware Requirements

To use RDMA, you typically need:

RDMA-capable NICs (e.g., Mellanox, Intel, Broadcom)

Lossless network support (especially for RoCE)

RDMA drivers and libraries:rdma-core, libibverbs, libmlx5 For programming: Verbs API, RDMA CM, or higher-level frameworks (e.g., DAOS, NVMe-oF, Libfabric)

Common Use Cases

High-speed datacenter communication

Distributed databases (e.g., CockroachDB, Oracle RAC)

HPC clusters

Storage systems (e.g., NVMe over Fabrics)

Machine learning training clusters

Real-time data replication

RDMA in Code (Simplified Concept)

Here’s a pseudo-view of RDMA vs traditional send:

// Traditional

send(socket, buffer, length, 0); // system call, data copied

// RDMA

rdma_post_write(qp, local_addr, remote_addr, length); // no CPU copy

Brian Wilson (GT1) 7-7-25

Comments

Cybersecurity for Small Businesses: What It Means and Why It Matters

Cybersecurity for Small Businesses: What It Means and Why It Matters In today’s digital landscape, cybersecurity is no longer just a concern for large corporations. Small businesses are increasingly becoming prime targets for cybercriminals, often due to their limited security measures and lack of awareness. Understanding cybersecurity and its implications is critical for protecting sensitive data, maintaining customer trust, and ensuring business continuity. What is Cybersecurity? Cybersecurity refers to the practices, technologies, and processes designed to protect digital systems, networks, and data from cyber threats such as hacking, malware, phishing, and data breaches. For a small business, this means safeguarding everything from customer records and financial data to employee information and proprietary business strategies. Why Should Small Businesses Care? Many small business owners assume that cybercriminals only target large enterprises. However, statistics sh...

“Calm Under Fire: The Secret Weapon for Customer Service Management”

“Calm Under Fire: The Secret Weapon for Customer Service Management” In today’s fast-paced, customer-driven world, businesses are constantly seeking exceptional leadership to manage their customer service departments. While resumes filled with corporate experience might catch a recruiter’s eye, one of the most overlooked goldmines of talent lies in a surprising place: the world of emergency communications. That’s right, former 911 dispatchers bring a powerhouse of skills perfectly aligned with the demands of customer service management. Here’s why hiring a former 911 dispatcher could be one of the smartest decisions your company makes. 1. Unmatched Composure Under Pressure 911 dispatchers thrive in high-stress environments. They handle life-or-death situations with a calm voice and a clear head, often juggling multiple crises at once. Transition that to a customer service setting, and you get a manager who won’t flinch when tensions rise, customers escalate, or systems go down....

WINGET: The Pros and Cons of Using Windows Package Manager for Software Updates

Need to update your programs? WINGET: The Pros and Cons of Using Windows Package Manager for Software Updates Maintaining up-to-date software is a key component of ensuring system security, stability, and performance on any Windows machine. As part of its modernization efforts, Microsoft introduced WINGET, the Windows Package Manager, a command-line tool designed to simplify the process of installing, updating, and managing applications. WINGET is particularly useful for IT professionals, power users, and system administrators looking for a more efficient way to maintain software across single machines or entire fleets. This article explores the benefits and limitations of using WINGET for software updates, along with the basic command-line syntax required to use it effectively. What Is WINGET? WINGET is a command-line utility for Windows that interacts with an open-source repository of software packages. It enables users to quickly install, update, and uninstall supported a...

GT1-Blog

Search This Blog