RDMA Connection

Question

How i can build my own RDMA Connection with 2 PCS

KFSys · Answer

Heya, Short answer: RDMA needs hardware that can do it, and the path you take depends on whether you want to learn the API or actually get the performance . Those are two very different setups, and the good news is you can start on the first one today with no special hardware at all. Here’s the honest map. First, the thing that trips everyone up RDMA bypasses the kernel network stack — the NIC writes straight into remote memory. That means a normal Ethernet card physically cannot do it. You need one of: InfiniBand — a separate fabric, needs IB HCAs and a subnet manager RoCE (RDMA over Converged Ethernet) — runs on Ethernet, needs a RoCE-capable NIC iWARP — RDMA over TCP, needs an iWARP NIC SoftRoCE / RXE — a software implementation, works on any NIC, no special hardware That last one is your starting point. Path 1: Start today with SoftRoCE (no hardware needed) SoftRoCE emulates RoCE in the kernel. You get zero of the performance benefit — it’s still going through the CPU and the normal stack — but the verbs API is byte-for-byte identical. Code you write here runs unchanged on real hardware later. This works on your two PCs, or on two Droplets, or two VMs on one box. On Ubuntu 22.04/24.04, on both machines: sudo apt update sudo apt install -y rdma-core ibverbs-utils perftest \ libibverbs-dev librdmacm-dev rdmacm-utils sudo modprobe rdma_rxe Bind an RXE device to your real interface (check yours with ip a — on a Droplet it’s usually eth0, on a PC often enp3s0): sudo rdma link add rxe0 type rxe netdev eth0 rdma link show You should see link rxe0/1 state ACTIVE physical_state LINK_UP netdev eth0. Confirm the verbs layer sees it: ibv_devinfo Now test it. On machine A (server): ib_send_bw -d rxe0 On machine B (client), pointing at A’s IP: ib_send_bw -d rxe0 And a connection-manager smoke test, which is closer to what your own code will do: # server rping -s -a -v # client rping -c -a -v Two gotchas that will cost you an afternoon if you miss them: RoCEv2 runs over UDP port 4791. Open it on both ends or nothing connects: sudo ufw allow 4791/udp On a Droplet, also add it to your Cloud Firewall if you use one. Locked memory limits. RDMA pins memory, and the default memlock limit will cause registration failures that look like cryptic ENOMEM errors. Set in /etc/security/limits.conf: * soft memlock unlimited * hard memlock unlimited Log out and back in, verify with ulimit -l. One caveat worth knowing: Red Hat has deprecated Soft-RoCE and dropped it in RHEL 10. It’s still in the mainline kernel and works fine on Ubuntu/Debian, but treat it as a development and learning tool, not something to build a product on. Path 2: Real RDMA between your two PCs This is very achievable on a hobbyist budget, and the trick is you don’t need a switch — connect the two cards back-to-back with a single cable. Hardware: 2× Mellanox ConnectX-3 VPI cards — used on eBay for roughly $20–40 each. These do both InfiniBand and 40GbE/RoCE, which makes them great for learning. 1× QSFP+ direct-attach copper (DAC) cable , 1–3 m, around $15–25. Both machines need a free PCIe x8 slot. If you want better longevity, ConnectX-4 Lx or ConnectX-5 cost more but are driven by the modern mlx5 driver. ConnectX-3 uses the older mlx4 path — still supported by the inbox rdma-core on Ubuntu, but NVIDIA has dropped it from current MLNX_OFED releases. For learning, ConnectX-3 is fine and cheap. Setup — the inbox drivers in Ubuntu are usually all you need, no MLNX_OFED required: sudo apt install -y rdma-core ibverbs-utils perftest infiniband-diags opensm lspci | grep -i mellanox # confirm the card is seen ibv_devinfo # confirm verbs sees it If you run the cards in InfiniBand mode , an IB fabric needs a subnet manager. With no switch there’s nothing to run it, so run it on one of the two PCs: sudo systemctl enable --now opensm Then ibstat should move the port from Initializing to Active . That state transition is the single most common place people get stuck — Initializing almost always means no subnet manager is running. If you run them in Ethernet/RoCE mode , skip opensm entirely, just assign IPs to the two interfaces on the same subnet and go. Back-to-back means you don’t need to worry about PFC/DCB lossless configuration on a switch, which is the other classic RoCE headache. Benchmark it: # server ib_send_bw -d mlx4_0 -a # client ib_send_bw -d mlx4_0 -a # latency ib_send_lat -d mlx4_0 On ConnectX-3 at 40 Gb you should see single-digit microsecond latency and ~35+ Gb/s. Compare that to your SoftRoCE numbers and the reason RDMA exists becomes obvious immediately. Path 3: What this looks like on DigitalOcean Being straight with you, since this is the DO community and it matters for planning: Regular Droplets and GPU Droplets do not expose RDMA hardware. They’re virtualized with standard virtio networking — there’s no InfiniBand HCA or SR-IOV RDMA function passed into the guest. You can absolutely run SoftRoCE on two Droplets (Path 1 works there perfectly, and it’s a genuinely nice way to develop without owning hardware), but you won’t get kernel-bypass performance. Real RDMA on DO lives in the Bare Metal GPU and multi-node GPU cluster products: Bare Metal GPUs (H100, H200, AMD MI300X) ship with 8× NVIDIA ConnectX-7 cards at 400 Gbps (InfiniBand/Ethernet VPI, NDR OSFP), plus ConnectX-6 Dx at 100 Gbps. RDMA gets its own auto-assigned IPv6 on the ConnectX-7 network, with up to 400 Gbps East/West VPC bandwidth and 3.2 Tbps NVLink between GPUs in the box. Multi-node GPU clusters on DOKS expose the fabric as NICs named fabric0…fabric7 (or fabric0…fabric15 on B300), alongside the usual eth0/eth1. The Mellanox k8s-rdma-shared-dev-plugin is installed automatically when you add a node pool with a fabric-connected slug, surfacing RDMA as Kubernetes resources: resources: requests: nvidia.com/gpu: 8 rdma/fabric0: 1 rdma/fabric1: 1 # ... through rdma/fabric7 limits: nvidia.com/gpu: 8 rdma/fabric0: 1 # ... matching You also install Multus CNI and create a NetworkAttachmentDefinition per fabric NIC using the host-device plugin, then reference them in pod annotations. Both Bare Metal GPUs and multi-node clusters are contract-only (talk to sales, not on-demand), currently in NYC and Amsterdam. So it’s the right answer for production AI/HPC workloads, and the wrong answer for “I want to learn RDMA this weekend.” A practical hybrid: develop and debug your verbs code on two cheap Droplets with SoftRoCE, then move the identical binary onto real hardware when you need the performance. The API doesn’t change. Writing the actual code Two layers, and pick based on what you’re doing: librdmacm (rdma_cm.h) — connection management with a sockets-like flow: rdma_create_id, rdma_resolve_addr, rdma_connect, rdma_accept. Start here. It handles address resolution and the QP handshake for you. libibverbs (verbs.h) — the raw layer: allocate a PD, register memory regions with ibv_reg_mr, create a CQ and QP, transition the QP through INIT → RTR → RTS, then post work requests with ibv_post_send / ibv_post_recv and reap completions with ibv_poll_cq. The mental model shift from sockets: you register memory once, then post work requests that complete asynchronously. There is no read()/write(). For IBV_WR_RDMA_WRITE and IBV_WR_RDMA_READ, the remote side isn’t involved at all at the CPU level — you need its remote address and rkey, which you exchange during connection setup. The best reference implementations are in the rdma-core repo itself: git clone https://github.com/linux-rdma/rdma-core # look at librdmacm/examples/ — rping.c and rdma_client.c/rdma_server.c rping.c is genuinely the best starting point — it’s a complete, working, readable client/server using the CM. Suggested order SoftRoCE on your two PCs (or two Droplets) — an hour, no money Get rping working between them, then build your own client/server from rdma_client.c Buy two ConnectX-3s and a DAC cable, run the same code, benchmark the difference Only then think about Bare Metal / DOKS if you’re heading toward production If you tell us what you’re actually building on top of this — storage, a message passing layer, GPU-to-GPU transfer — the advice gets a lot more specific, since the right verbs and QP type differ quite a bit between those. Hope that helps!

alexdo · Answer

Hi there, The walkthrough above is thorough enough to be a mini RDMA course on its own. One thing worth flagging, since the whole answer assumes Linux: if your two PCs happen to be running Windows rather than Ubuntu, Path 1 doesn’t have a real equivalent. Windows RDMA support goes through NVIDIA/Mellanox’s WinOF-2 driver package, and it needs actual RDMA-capable hardware to work at all. There’s no built-in software emulation shipped by Microsoft the way rdma_rxe gives you on Linux for free, so on Windows you’d effectively start straight at Path 2, buy the ConnectX-3 cards and a DAC cable, install WinOF-2, and connect the two machines back to back the same way. Once it’s working, the easiest real-world thing to test against on Windows is SMB Direct, copy a large file between the two machines over the RDMA link and watch CPU usage stay flat instead of spiking the way a normal network copy would. If both machines are already Linux, ignore all of this and everything above just works as written. Regards

Bobby · Answer

One thing not covered yet: if you just want to see RDMA-style zero-copy transfer without touching verbs code at all, nvme-of (NVMe over Fabrics) using RDMA transport is worth a look if storage is your end goal rather than raw networking. It rides on the same RXE/RoCE setup KFSys described but gives you a working, mountable block device between the two machines instead of writing your own client/server, useful if you want to validate the hardware/software stack works before writing any verbs code yourself.

Become a contributor for community

DigitalOcean Documentation

Resources for startups and AI-native businesses

The developer cloud

Start building today

Report this

RDMA Connection

Related questions

Which computer skill is most valuable for getting a job in 2026?

Write a program with three functions: upper , lower , and reverse .

ESP executable analysis

Report this

RDMA Connection

Related questions

Which computer skill is most valuable for getting a job in 2026?

Write a program with three functions: upper , lower , and reverse .

ESP executable analysis