Lately we've been seeing some discussion around the amount of time required to resync a S2D node's storage after it has come back from a reboot for whatever reason.
Unlike a RAID controller where we can tweak rebuild priorities, S2D does not offer the ability to do so.
It is with very much a good thing that the knobs and dials are not exposed for this process.
Why?
Because, there is a lot more going on under the hood than just the resync process.
While it does not happen as often anymore, there were times where someone would reach out about a performance problem after a disk had failed. After a quick look through the setup the Rebuild Priority setting turned out to be the culprit as someone had tweaked it from its usual 30% of cycles to 50% or 60% or even higher thinking that the rebuild should be the priority.
S2D Resync Bottlenecks
There are two key bottleneck areas in a S2D setup when it comes to resync performance:
- East-West Fabric
- 10GbE with or without RDMA?
- Anything faster than 10GbE?
- Storage Layout
- Those 7200 RPM capacity drives can only handle ~110MB/Second to ~120MB/Second sustained
The two are not the mutually exclusive culprit depending on the setup as they both can play together to limit performance.
The physical CPU setup may also come into play but that's for another blog post. ;)
S2D East-West Fabric to Node Count
Let's start with the fabric setup that the nodes use to communicate with each other and pass storage traffic along.
This is a rule of thumb that was originally born out of a conversation at a MVP Summit a number of years back with a Microsoft fellow that was in on the S2D project at the beginning. We were discussing our own
Proof-of-Concept that we had put together based on a Mellanox 10GbE and 40GbE RoCE (RDMA over Converged Ethernet) fabric. Essentially, at 4-nodes a 40GbE RDMA fabric was _
way_ too much bandwidth.
Here's the rule of thumb we use for our baseline East-West Fabric setups. Note that we always use dual-port NICs/HBAs.
- Kepler-47 2-Node
- Hybrid SSD+HDD Storage Layout with 2-Way Mirror
- 10GbE RDMA direct connect via Mellanox ConnectX-4 LX
- This leaves us the option to add one or two SX1012X Mellanox 10GbE switches when adding more Kepler-47 nodes
- 2-4 Node 2U 24 2.5" or 12/16 3.5" Drives with Intel Xeon Scalable Processors
- 2-Way Mirror: 2-Node Hybrid SSD+HDD Storage Layout
- 3-Way Mirror: 3-Node Hybrid SSD+HDD Storage Layout
- Mirror-Accelerated Parity (MAP): 4 Nodes Hybrid SSD+HDD Storage Layout
- 2x Mellanox SX1012X 10GbE Switches
- 10GbE RDMA direct connect via Mellanox ConnectX-4 LX
- 4-7 Node 2U 24 2.5" or 12/16 3.5" Drives with Intel Xeon Scalable
Processors
- 4-7 Nodes: 3-Way Mirror: 4+ Node Hybrid SSD+HDD Storage Layout
- 4+ Nodes: Mirror-Accelerated Parity (MAP): 4 Nodes Hybrid SSD+HDD Storage Layout
- 4+ Nodes: Mirror-Accelerated Parity (MAP): All-Flash NVMe cache + SSD
- 2x Mellanox Spectrum Switches with break-out cables
- 25GbE RDMA direct connect via Mellanox ConnectX-4/5
- 50GbE RDMA direct connect via Mellanox ConnectX-4/5
- 8+ Node 2U 24 2.5" or 12/16 3.5" Drives with Intel Xeon
Scalable Processors
- 4-7 Nodes: 3-Way Mirror: 4+ Node Hybrid SSD+HDD Storage Layout
- 4+ Nodes: Mirror-Accelerated Parity (MAP): 4 Nodes Hybrid SSD+HDD Storage
Layout
- 4+ Nodes: Mirror-Accelerated Parity (MAP): All-Flash NVMe cache + SSD
- 2x Mellanox Spectrum Switches with break-out cables
- 50GbE RDMA direct connect via Mellanox ConnectX-4/5
- 100GbE RDMA direct connect via Mellanox ConnectX-4/5
Other than the Kepler-47 setup we always have at least a pair of Mellanox ConnectX-4 NICs in each node for East-West traffic. It's our preference to separate out the storage traffic and the rest.
All-Flash Setups
There's a lot of talk in the industry about all-flash.
It's supposed to solve the biggest bottleneck of them all: Storage!
The catch is, bottlenecks are moving targets.
Drop in an all-flash array of some sort and all of a sudden the storage to compute fabric becomes the target. Then, it's the NICs/HBAs on the storage _
and_ compute nodes, and so-on.
If you've ever changed a single coolant hose in an older high miler car you'd see what I mean very quickly. ;)
IMNSHO, at this point in time, unless there is a very specific business case for all-flash and the fabric in place allows for all that bandwidth with virtually zero latency, all-flash is a waste of money.
One business case would be for a cloud services vendor that wants to provide a high IOPS and vCPU solution to their clients. So long as the fabric between storage and compute can fully utilize that storage and the market is there the revenues generated should more than make up for the huge costs involved.
Using all-flash as a solution to a poorly written application or set of applications is questionable at best. But, sometimes, it is necessary as the software vendor has no plans to re-work their applications to run more efficiently on existing platforms.
Caveat: The
current PCIe bus just can't handle it. Period.
A pair of 100Gb ports on one NIC/HBA can't be fully utilized due to the PCIe bus bandwidth limitation. Plus, we deploy with two NICs/HBAs for redundancy.
Even with the addition of more PCIe Gen 3 lanes in the new Intel Xeon Scalable Processor Family we are still quite limited in the amount of data that can be moved about on the bus.
S2D Thoughts and PoCs
The Storage Spaces Direct (S2D) hyper-converged or SOFS only solution set can be configured and tuned for a very specific set of client needs. That's one of its beauties.
Microsoft remains committed to S2D and its success. Microsoft Azure Stack is built on S2D so their commitment is pretty clear.
So is ours!
Proof-of-Concept (PoC) Lab
S2D 4-Node for Hyper-Converged and SOFS Only
Hyper-V 2-Node for Compute to S2D SOFS
This is the newest edition to our S2D product PoC family:
Kepler-47 S2D 2-Node Cluster
The Kepler-47 picture is our first one. It's based on
Dan Lovinger's concept we saw at Ignite Atlanta a few years ago. Components in this box were similar to Dan's setup.
Our second generation Kepler-47 is on the way to being built now.
Kepler-47 v2 PoC Ongoing Build & Testing
This new generation will have an Intel Server Board DBS1200SPLR with an E3-1270v6, 64GB ECC, Intel JBOD HBA I/O Module, TPM v2, and Intel RMM. OS would be installed on a 32GB Transcend 2242 SATA SSD. Connectivity between the nodes will be Mellanox ConnectX-4 LX running at 10GbE with RDMA enabled.
Storage in Kepler-47 v2 would be a combination of one Intel DC P4600 Series PCIe NVMe drive for cache, two Intel DC S4600 Series SATA SSDs for performance tier, and six HGST 6TB 7K6000 SAS or SATA HDDs for capacity. The PCIe NVMe drive will optional due it is cost.
We already have one or two client/customer destinations for this small cluster setup.
Conclusion
Storage Spaces Direct (S2D) rocks!
We've invested _
a lot_ of time and money in our Proof-of-Concepts (PoCs). We've done so because we believe the platform is the future for both on-premises and data centre based workloads.
Thanks for reading! :)
Philip Elder Microsoft High Availability MVPMPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Web SiteOur Cloud Service
