drink the sweet feeling of the colour zero

Storage for the SMB

TAGS: None

As some of you following me on Twitter know, I am in the process of trying to spec out a storage and virtualisation upgrade for one of my clients.  The requirements are as follows:

12TB initial deployable storage, scalable to 50TB.
30K IOPS.
1TB of virtual host RAM across the network.
It can cost no more than $35K.

The client has several hosts already; 8x 64GB hosts are already in play given them an installed capacity of 512GB of virtual RAM.  The storage subsystem, however, is over 6 years old and 100% virtual-host local.  I am losing disks on this thing left right and center (300GB WD Velociraptors are crafted from the raw evil of the universe from whence politicians and lawyers spring) so the time to replacement on this is measured – at maximum – in the 3 month range.

This setup must be – quite literally – bulletproof.  I won’t be in the office 16 hours a day any more.  This has to be able to deal with hardware failure.

Call it paranoia if you must, but I don’t trust the standard SAN configurations.  Redundant PSUs, dual controllers, dual-port hard drives…but single motherboard, CPU and RAM.  The storage and power subsystems are redundant, but there are still single points of failure!  You’ll pardon me if I don’t place faith in platitudes that say “the statistics on motherboard/CPU/RAM failure are so low that the risk calculations make our product awesome!”

We are talking about a company betting their entire business on one device that has a single point of failure here.  There isn’t room for that company to ever be in the wrong spot on the statistical curve.

So my preliminary solution is block-level replication across multiple hosts.  In this case, a Supermicro server using an LSI controller with CacheCade running on Windows Server 2012.  Server 2012 gives me dedupe for file-level sharing, Distributed File System Replication to move file-level-shared items from primary to failover node as well as SMB 3.0 and MPIO support.  I plan to use the native Server 2012 target for block-level replication between the two hosts unless it proves unstable, at which point Starwind would do fine.

The LSI controller allows me to provide bulk storage using spinning rust, then accelerate that storage by simply plugging in SSDs and designating them as cache.  This makes the most frequently used blocks go faster, driving up IOPS while keeping the cost of overall storage down.

Preliminary calculations on the server hardware without disks have the following components for around $5000:

Chassis: Supermicro SC846BE16-R1K28B
Motherboard: Supermicro X9SRi-F
CPU: Intel Xeon E5-2620
RAM: Kingston 64GB (4x 16GB) DDR3-1600 ECC REG Kit
NIC: Intel X520-DA2 E10G42BTDA
RAID: LSI 9280-16i4e
RAID BBU: LSI LSI00264 MegaRAID LSIiBBU08 Battery Backup Unit
RAID Module: LSI LSI00292 MegaRAID CacheCade Pro 2.0
RAID Cables:   LSI Mini-SAS cable CBL-SFF8087SB-06M (x3)
OS: Windows Server 2012 Standard

Disks are another matter.  Fully laden we’re talking about $8800 worth of disks per server. 4x Crucial M500 960GB SATA 2.5″ 7mm SATA3 (CT960M500SSD1) for flash acceleration and 20 WD RE 3TB SATA3 (WD3000FYYZ).

That would make my storage nodes $13800 each or $27600 for the pair.  That could be a little higher if I need to use the Starwind iSCSI target.  Still, that leaves around $5000 to buy a pair of compute nodes with 512GB of RAM between them; entirely doable in today’s world.

More realistically, however, I would deploy these systems without being fully laden.  I would stick to 12 of the spinning rust hard drives and 2 of the SSDs.  That would put the disk load at $5000.  Which means $10K per node for ~30TB usable at ~30K IOPS with room to expand that by another 24TB of spinning rust and 2x SSDs.

So for ~20K$ I get ~30TB of storage at ~30K IOPS redundant enough that I can actually *shoot* one of the nodes and the system will keep working.   What’s more, when PernixData hits general availability (and assuming it is priced anywhere near reach for an SMB like this) then you can simply “make it go faster” by tossing some Intel 520 480GB SSDs into the compute nodes and enabling PernixData.  Instant upgrade!

This is a preliminary design, mind you.  There’s still a lot of work to be done – not the least of which is to see if off-the-shelf solutions can match or beat it – but it’s still really, really cool.

TAGS: None

Comments are closed.

© 2009 drink the sweet feeling of the colour zero. All Rights Reserved.

This blog is powered by the Wordpress platform and beach rentals.