Here is a good start for a lab environment:
A set of DAS JBOD units can used with two small nodes and 2 SAS based HBAs per node to stand up a Storage Spaces cluster using the Server 2012 R2 bits. A couple of MicroServer Gen8 boxes would round out the Hyper-V side of it.
On this blog there are a lot of configurations discussed that utilize intelligent direct attached storage.
- Basic setup
- Intel Server System SR1695GPRX2AC pair with Promise VTrak E610sD or E310sD
- This category search on the blog has a number of really good posts including configuration examples based on the SR1695GPRX2AC (blog category link)
- This server unit has been our go-to for base configurations as it is an excellent and flexible platform
- Advanced setup
- Intel Server System R2208GZ4GC pair with the Promise VTrak E610sD or E310sD
- All-Out Setup
- All of the above plus LSI SAS 6160 Switch pair and Intel Modular Server with 3 nodes.
In the above setups the key is the intelligent storage providing mitigation services to the SAS HBAs and OS access to the central storage.
With the 2012 R2 bits we are going to put together a redundant JBOD setup for a Storage Spaces cluster. This is the next direction we are delving into as we can put together a small SS cluster for a very reasonable cost.
Today, we are working on the following (similar to David Ziembicki’s setup) setup for clustered Storage Spaces:
- Basic
- Intel Server System R1208JP4OC with pair of SAS HBAs (RS25GB008) (2 nodes)
- 32GB of ECC per node to start
- Intel Storage System
JBOD2224S2D2JBOD2224S2DP Intel JBOD units (2 units)- JBOD is dual expander and dual port backplane
- Seagate Savvio SAS drives are dual port
- 1m SAS Cables (4)
- Windows Server 2012 R2 beta – Storage Spaces Cluster Setup
- Intel Server System R2208GZ4GC pair for Hyper-V nodes (we have had these in our lab for a year or so now).
- 64GB to 128GB of ECC
- Windows Server 2012 R2 beta or RTM – Hyper-V Nodes
- Intel Server System R1208JP4OC with pair of SAS HBAs (RS25GB008) (2 nodes)
- Advanced
- Add a pair of 8-Port or 12-Port NETGEAR 10Gbit Ethernet switches
- Ports on each NIC would be split between switches for redundancy
- Add a pair of Dual-Port 10Gbit PCIe and/or I/O Module NICs to each node
- 10Gbit Ethernet would SMBv3 Storage Spaces located VHDX
- 10Gbit Ethernet would be for Live Migration Network
- LSI SAS Switches (we have a pair of these in our lab setting)
- Additional Intel JBOD units to test switches and scaling storage out
- Add a pair of 8-Port or 12-Port NETGEAR 10Gbit Ethernet switches
Using David Ziembicki’s setup though one would be able to start at the base level and put together a similar configuration on a budget.
An HP MicroServer Gen8 would make an excellent platform for testing as they are relatively inexpensive and have pretty close to the full Intel virtualization Acceleration feature set.
Note that the Sans Digital MS28X listed in his blog post splits drives 0-3 and 4-7 between the two available external SAS connections. That means that there is no ability to use this storage unit without an LSI SAS 6160 Switch pair (Sans Digital MS28X Quick Installation Guide PDF)!
However, the Sans Digital MS8X6 unit does support redundancy and therefore they could be used to test Storage Spaces clustering configurations (Sans Digital MS8X6 Quick Installation Guide PDF).
Of course, for the added functionality there will be an extra cost involved, however one could drop the LSI SAS Switch for a set of these units for about the cost of the original MS28X plus SAS Switch!
- Storage Spaces Cluster
- Storage Spaces Node
- Intel Xeon E3-1230
- Intel Server Board S1200BTLSH
- 16GB ECC
- Intel Integrated RAID RMS2AF040
- 120GB Intel 320 Series SSD (or small 10K SAS) RAID 1 pair for host OS
- Quad-Port Intel Gigabit Server NIC PCIe
- Intel certified chassis (whether Intel or other)
- Storage
- Sans Digital MS8X6
- 300GB 15K 3.5" SAS drives can be found for a good deal today
- Sans Digital MS8X6
- Hyper-V Node
- Intel Xeon E3-1230
- Intel Server Board S1200BTLSH
- 32GB ECC
- Intel Integrated RAID RMS2AF040
- 120GB Intel 320 Series SSD (or small 10K SAS) RAID 1 pair for host OS
- Quad-Port Intel Gigabit Server NIC PCIe
- Intel certified chassis (whether Intel or other)
- OPTIONS
- Add Intel RMM4 for full KVM over IP
- Add Dual-Port 10Gbit Intel Ethernet for SMBv3 and Live Migration Networks
- Add Intel Storage Systems JBOD2224S2DP at a later date for full SAS Dual Port Redundancy
- Storage Spaces Node
There are so many different ways to go about this.
The main thing is to start small and work one’s way up to a full scale server grade lab as the jobs come in! That’s how we built our own lab systems up and how we built up the knowledgebase and experience!
EDIT: Oops, Star Wars on the mind. Intel Storage Systems part number should be JBOD2224S2DP (I had JBOD2224S2D2 above!). :)
Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book
Chef de partie in the SMBKitchen
Find out more at
www.thirdtier.net/enterprise-solutions-for-small-business/
Hey Phil, I always find your postings really interesting.
ReplyDeleteQuick question: for a minimum "lab" setup to test basic storage spaces clustering, do I really need a JBOD/DAS enclosure and SAS switch?
Seems to me I should be able to get SS and clustering working with just direct connection between the SAS HBAs and one or two SAS drives. Why bother with SAS switches and JBODs for only one or two hard drives?
-- Ken
Ken,
ReplyDeleteYou do need a JBOD.
Minimum would be two servers with SAS HBAs and a JBOD that would support both connected to the same disk group via SAS (2 ports).
With that you could build an asymmetric cluster based on Storage Spaces. In this configuration we set up a small 1.5GB Space for the Quorum/Witness after standing up the cluster. Then we split the rest of the storage up 50/50 with each portion attached to one node.
Not perfect, as that means that only one node deals with I/O for its Space but it can be done quite inexpensively.
I had a quick question, i currently have 4 of the intel JBOD units in your guide, when i connect them the command get-storageenclosure only shows 1 enclosure in an unhealthy state. for the life of me i cant see why.
ReplyDeletethis a more detailed description of my issue
http://social.technet.microsoft.com/Forums/windowsserver/en-US/dd3ff115-e44c-4313-b8df-f9718f1f213d/getstorageenclosure-command-concerns?forum=winserverfiles
Anonymous,
ReplyDeleteI've posted my comments.
Ref:
ReplyDeleteHi this is the case number 8000979910 from Intel.
Hi Philip,
ReplyDeleteWe are attempting to implement an SOFS cluster with JBOD fail-over.
We have easily been able to implement the SS solution (including tiering and WBC) when "-IsEnclosureAware" is not part of the equation.
Unfortunately, we have not had good luck once we introduce the "-IsEnclosureAware" option into the vDISK creation scheme. From a creation stand-point, the command runs and we are able to create a volume. However, we have no idea how Windows is allocating the space (splitting the JBODs / drives). We are under the assumption that there is a specific way to distribute drives across the multiple JBODs to maximize space and resiliency when "-IsEnclosureAware" is being leveraged. Unfortunately, there is little to no documentation on this subject. For simplicity, we are testing 2-Way mirror with JBOD fail-over (3 total JBODs). However, once we have figured it out, our goal is to leverage four (4) JBODs and dual-parity (with enclosure awareness / JBOD fault-tolerance).
Thus far, we have attempted the following while testing:
Trial 1: Distribute Disks evenly amongst 3 JBODs (2 x SAS SSD, 8 x 10K SAS disk per JBOD unit)
Trial 2: Split disk evenly amongst 2 JBODs (4 x SAS SSD, 12 x 10K SAS disk per JBOD Unit). Add extra 10K SAS disk to third JBOD. Make sure it is included in the Storage pool
Both of the above configurations work with the '-IsEnclosureAware' option, but the second allows for more space when the mirror-stripe (RAID10) is configured. Originally, we assumed that the third JBOD is only necessary to serve as a quorum arbitrator, so-to-speak, and we read a blog that stated just to ensure there are always a greater number of "online" disks when a single JBOD fails in the pool than offline. Hence, make sure that no single JBOD take down an equal or greater number of disks to those still online.
Anyway, needless to say, regardless of the reasoning, both of the configurations above can withstand a JBOD failure. Regardless of which JBOD we disconnect / pull the SAS cables, the CSV goes offline and available.
We have searched countless hours on the Web, including all of the Microsoft Technet documentation, but it just seems there is nothing out there in terms of a "best practices" or "how to" when configuring Storage Spaces with JBOD Fault-Tolerance. The only info that is common seems to be information pertaining to the number of JBODs one must have installed for a specific vDISK type (e.g. 3 for 2-way mirror, 4 for dual-parity, etc). Funny enough, even the reasoning as to why there needs to be 3 JBODs for a 2-way mirror and 4 JBODS for a dual-parity volume differs. Microsoft doesn't even comment on it in their documentation, and MVPs that have written about it do not provide any example with the exception of the one gentlemen (I do not have the link currently in front of me) who mentioned that a greater number of disks need to be online in the pool than offline at any one time for Enclosure Awareness to work properly... ...hence the reason for three JBODs in a 2-way mirror config.
If you could shed any light, it would be greatly appreciated.
Paul,
ReplyDelete2-Way Mirror does indeed require 3 JBODs in order to keep things going. We evenly distribute the disks across the three.
3-Way Mirror resilience requires 5 JBODs to allow for 2 enclosure failures.
We are not looking to use parity for anything beyond static file storage so have not run any tests on that configuration.
That being said, yes, there is indeed a dearth of documentation. The SS FAQ and SS Performance documents are probably the only totally comprehensive documentation next to Jose Barreto's blog posts.
For us we have been building out our lab and Proof of Concept setups for a year or two for testing and now deployment scenarios.
Thanks for the comments.
Philip
Philip,
ReplyDeleteThanks for your response.
Just to be clear, you have been able to get Enclosure Fail-over working (e.g. unplugging any single enclosure and CSVs / cluster disks remain online)?
If so, please shed some light on the following (assuming two-node SOFS cluster connected to 3+ JBODs):
1. Do you allocate the quorum disk to a single enclosure, or let SS manage where it places it?
2. Do you use the "IsEnclosureAware $true" parameter for the quorum disk, assuming you do not allocate it to a specific (e.g. third) enclosure?
3. Do you create the quorum vDisk as a second or third volume as part of your primary storage pool, or do you create a separate storage pool specifically to host just the quorum vDisk?
Based on your response to my previous question, with our configuration, please tell me if my assumption for testing this properly is correct (assuming the goal is Enclosure Fail-over with a 2-Way Mirror configuration for the CSV(s)):
a) Distribute disks evenly amongst all three (3) JBODs connected: 3 x 200 GB SSDs per JBOD, 8 x 900 GB SAS 10K Disks per JBOD
b) Create one large Clustered Storage Pool with all disks from all JBODs: Hyper-V Pool
c) Create two (2) vDisks using the "-IsEnclosureAware $true" parameter, Quorum & VMs CSV, using the following settings:
Quorum vDisk: No tiering, no write-cache, just a 2-column, 2-way mirror
CSV vDisk: 4-column, 2-way mirror with 100 GB SSD write-cache, and tiering enabled.
Also, we will enough space is left in pool (SSD and 10K SAS) to sustain a single disk failure in any tier and still have capacity left for parallel rebuild of the failed disk.
Given the above, we should be able to unplug any one JBOD and still remain online and operational, albeit, in a degraded state? Correct? In other words, our CSV should remain up and all VMs should continue to hum along?
We will test this tomorrow and post the results here when done. Keeping fingers crossed.
Paul,
ReplyDelete1: We slice off a 1.5GB 3-Way Mirrored Space for disk witness/quorum for the SOFS cluster. This is done before anything else as far as storage.
2: All Spaces would use this setting including the disk witness/quorum.
3: One Pool several Spaces with the first being witness/quorum.
3a: Yes but disks should be in multiples of Mirror type times columns (SSD Tier should have a minimum of 4 disks to start in your 2-Way Mirror setup).
3b: Yes so long as we don't exceed SS maximums in a clustered setting.
3c: Not sure about this? We would use the same settings for all virtual disks as a rule.
And yes, with three enclosures and the IsEnclosureAware setting your virtual disks should stay online. _Should_ _Could_ _Would_ lots of variables at play here. :)
Hope it goes well!
Thanks,
Philip