RDMA via RoCE 101 for Storage Spaces Direct (S2D)

We’ve decided to run with RoCE (RDMA over Converged Ethernet) for our Storage Spaces Direct (S2D) proof of concept (PoC).

• (4) Intel Server Systems R2224WTTYS
• Dual Intel Xeon Processors, 256GB ECC, Dual Mellanox ConnectX-3, and x540-T2
• Storage is a mix of 10K SAS and Intel SATA SSDs to start
• (2) Mellanox MSX1012 56Gbps Switches
• (2) NETGEAR XS712T 10GbE Switches
• (2) Cisco SG500x-48 Gigabit Switches
• APC 1200mm 42U Enclosure
• APC 6KV 220v UPS with extended runtime batteries

The following is a list of resources we’ve gathered together as of this writing:

• RoCE How-To Video
• Hyper-V Amigos Showcast Episode 9: RDMA, RoCE, PFC and ETS (YouTube)
• Carsten  interviews Didier about his RoCE endeavours
• EXCELLENT and well worth the time (~45 minutes)
• Didier’s Working Hard in IT content
• RoCE Tagged: https://blog.workinghardinit.work/tag/roce/
• RDMA Tagged: https://blog.workinghardinit.work/tag/rmda/
• Data Centre Bridging Tagged: https://blog.workinghardinit.work/category/data-center-bridging/
• The two are sometimes exclusive of each other but still contain RoCE content
• SMB Direct Secret Decoder Ring Slides (PDF)
• Deck from the above video
• Lots of PoSh
• Mellanox has a fair amount of info on setting things up but is a bit convoluted
• HowTo Configure SMB Direct (RoCE) over PFC on Windows 2012 Server
• RDMA/RoCE Considerations for Windows 2016 on ConnectX-3 Adapters
• HowTo Configure RoCE v2 for ConnectX-3 Pro
• HowTo Enable PFC on Mellanox Switches
• HowTo Configure QoS on Mellanox Switches
• Mellanox Community: Tagged RoCE
• Microsoft
• Jose: Deploying Windows Server 2012 with SMB Direct (SMB over RDMA) and the Mellanox ConnectX-3 using 10GbE/40GbE RoCE
• How-To with good resources
• Ignite: Enabling Private Cloud Storage Using Servers with Local Disks
• S2D demo

This is, by far, not the most comprehensive of lists. The best place to start in our opinion is with Didier’s video and the PDF of the slides in that video. Then move on to the Mellanox resources.

We’ll update this blog post as we come across more materials and eventually get a process guide in place.

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service