Ideally, we'd all have all the top-shelf enterprise-class storage we need--not just for critical production apps but for development, testing, disk-to-disk backup and our disaster-recovery hot site. But real-world budgets mean we sometimes must make do with less powerful or second-hand storage.

If you have a spare server and an SCSI cabinet in your IT boneyard, you can turn them into an iSCSI disk array by adding an inexpensive iSCSI target program. Even if you have to buy the pieces, you can put together a 2-TB array for about $4,000 (see "We Built It, But Would We Buy It?"). Your new iSCSI array will be running on top of a Windows or Linux OS, so you can allocate space on your new server to file service as well as iSCSI logical drives.

You also can use this array as a stopgap. When a state agency was waiting for approval to purchase a new disk array, its Exchange servers began running short of space. The agency used a server, a SCSI cabinet and a downloaded copy of WinTarget to add storage to the servers overnight.

Maybe building an iSCSI disk array out of a server seems wacky. But many of the iSCSI disk arrays on the market, including those from well-known vendors like LeftHand Networks and MPC (formerly MicronPC), use standard Xeon motherboards. Naturally, a server-based array won't have redundant disk controllers, but the combination of a good RAID controller and iSCSI target software can produce an iSCSI array that performs as well as name-brand arrays.

Once you add iSCSI target software to a Windows or Linux server, you can create logical disks or LUNs (logical unit numbers). Most folks with enterprise arrays will create RAID sets and then slice them into LUNs to present to their servers, but having access to the underlying OS gives you more flexibility. Most iSCSI target software will let you create virtual LUNs as file objects in an existing file system on your server, or turn a volume, partition or other unformatted disk space into a LUN.

File LUNs can take advantage of features like compression and snapshots, and you can use third-party programs like Double-Take or WANsync for replication. As host server volumes fill up, you can expand a file-based LUN with just a few mouse clicks--expanding a partition or physical disk is a much bigger job. If you want to try a homemade iSCSI array in your development environment, file-based LUNs are a good fit. And deleting LUNs from test servers you're no longer using opens up lots of free space.

On the downside, the additional overhead of running your iSCSI I/O through the host OS's file system can make file-based LUNs slower than partition LUNs. This is especially true if you use your file-based LUN volume for other files or LUNs and then expand the LUN several times, which fragments the file. Thin provisioning lets you create 10 1-TB LUNs on a 5-TB array, because it doesn't use disk space until it's written. Although it might be tempting to make your home-brew array emulate a high-end SAN array's thin provisioning feature by using sparse files for LUNs, we'd advise against it--it would create a very fragmented LUN file and have a significant performance impact.

It's also a bad idea to use other file-system filters like antivirus software or CDP apps on file LUN data. Antivirus programs misinterpret LUN data as a virus, slowing your system to a snail's pace. If your server's G: volume is the LUN and someone saves a virus-infected file to that drive, with antivirus on the iSCSI server, the G: drive will be quarantined, not just the infected file. If you're using a single volume for file service and file-based LUNs, you're better off excluding the file LUNs from scanning and running the antivirus program only on the server mounting the LUN.

Dedicated arrays running proprietary software aren't susceptible to the usual viruses, worms or other malware and can go longer between firmware updates than a Windows server should go between patches. And since you'll have to shut down all the host servers using logical drives when you apply patches to your homemade array, you could have more downtime than with dedicated arrays.

Is It Safe?

Although the iSCSI spec defines how the protocol should be sent over encrypted links using IPsec, most manufacturers have not implemented that optional portion of the spec. If you build an iSCSI array using Windows Server or a Linux distribution that supports IPsec in the TCP/IP stack, the iSCSI target software will use that stack. IPsec can then encrypt your data in transit between your server and drive array or validate that only those servers that should be accessing LUNs on your array have access to it via certificates--more secure that the usual CHAP (Challenge-Handshake Authentication Protocol) authentication.

The disadvantage of IPsec: All that encryption and decryption eats CPU cycles like candy. The TOE and iSCSI HBAs we use to off load the iSCSI computational overhead don't support IPsec off-load. Although major network card vendors, including Intel and 3Com, make 10/100 security NICs that support IPsec, we could only find one Gigabit Ethernet IPsec card: Cavium Network's Nitrox XL. Remember that IPsec only encrypts data in transit.

So, what software do you look for in an iSCSI target? Opt for flexibility. You'll want to expand LUNs to use space as your servers need it, so go with an iSCSI target app that can span a LUN across several RAID sets or physical drives, with a tight set of file-based LUNS and a logical volume manager that can span volumes across multiple drives.

Also, look for snapshot capability. On-demand snapshots replicate virtual servers or development configurations, but really shine when they're run on a schedule and coordinate with your server applications through a snapshot provider like Windows 2003 VSS (Volume Shadowcopy Service). Your databases can flush their internal buffers when you take a snapshot, and hourly snapshots let you roll back to the last snapshot if they get corrupted.

Multipath I/O support allows multiple GigE connections between servers and the iSCSI array, and also lets these connections use separate Ethernet switches. This way, your servers won't lose the connection to their data if a cable is disconnected or a switch goes down.

Replication is an advanced feature that can make your local disk infrastructure more fault-tolerant or send your data off-site to another disk array for disaster recovery. An organization looking for the high availability of replication will be unlikely to use a home-brew iSCSI array.

Windows Target Options

When Microsoft purchased the rights to WinTarget from String Bean Software in March, it effectively removed this popular iSCSI target application from the market. Microsoft says it will integrate WinTarget's functionality--supporting both file- and volume-based LUNs, using VSS to provide snapshots--into a future version of Windows Storage Server it sells to Dell and Hewlett-Packard for network attached storage. That means this technology will no longer be available for the "do-it-yourself" crowd.

That leaves Rocket Division Software's StarWind as the basic Windows target. StarWind can create fixed and expandable file LUNs and it supports snapshots. In addition, you can create RAMdisk LUNs and share .ISO files, tape drives or any other SCSI device on your StarWind server. It doesn't support LUN masking by IP address or IQN, so you'll have to use CHAP authentication to control which host servers have access to specific LUNs. The enterprise version of StarWind is $995; a version that only supports one attached host is $395. MySAN, a free iSCSI target from Nimbus Data Systems, provides volume-based LUNs and only IQN masking, but at that price, we can't complain.

Organizations with more sophisticated needs should look at FalconStor Software's iSCSI Storage Server and DataCore Software's SANmelody. Both have been producing storage virtualization applications for years and they've used that technology to build full-featured iSCSI targets.

FalconStor's iSCSI Storage Server was the Editor's Choice in our iSCSI target roundup (ID# 1522f2). In addition to supporting file LUNs, it lets you combine any disks attached to your server into a storage pool and create and expand LUNs from the common storage in a pool. FalconStor has snapshot providers for 10 database and messaging apps as well as a VSS provider to work with iSCSI Storage Server's Snapshot service, supporting on-demand, scheduled and backup app-initiated snaps. It also supports local mirroring, snapshot-style remote replication and clustering of iSCSI storage servers--good for organizations that want to add iSCSI access to an FC array. It's priced from $2,500.

Like the FalconStor solution, SANmelody throws physical disks into a storage pool from which an administrator can build LUNs. SANmelody even supports thin provisioning, which lets you create 10 20-GB LUNS from a 150-GB disk pool. SANmelody's coolest feature is the migration facility, which lets you move a server's data from one set of disks to another while the server is running--all priced from $1,200, depending on options.

Linux Target Options

Hardcore open-source users can download iSCSI Enterprise Target Project from iscsitarget.sourcesforge.net and use it to export their Linux block devices. This basic driver supports server authentication via CHAP, but relies on your Linux logical volume manager for block device management and is managed by editing configuration files.

Pavitrasoft's Mayastor iSCSI Storage Server uses a Web interface to let an admin combine multiple disks into a volume group and then allocate logical drives from that pool. LUNs can be allocated by IP address and iSCSI IQN, as well as CHAP.

Several other vendors--including Chelsio Communications, Wasabi Systems and SBE--are marketing iSCSI target implementations for Linux to OEM customers, and they aren't actively looking for do-it-yourself customers.

Perhaps the most interesting Linux iSCSI implementation is from Germany's Open-E, which packages its solution in a flash module with an IDE interface. It needs to be the master on your first channel, but supports 3Ware RAID controllers for the data drives. Just plug the module into the primary IDE channel of your motherboard and you're off and running. Open-E has SOHO, SMB and enterprise editions selling for $299 to $750. The enterprise edition supports RAID-6, MPIO, IPsec, on-demand and scheduled snapshots, online volume and capacity expansion, and NIC bonding, all through a simple Web interface. Open-E enterprise iSCSI can even present a single server to the network as multiple iSCSI targets, which could come in handy in development or disaster-recovery environments which emulate a production SAN with multiple iSCSI arrays.

Howard Marks is founder and chief scientist at Networks Are Our Lives, a network design and consulting firm in Hoboken, N.J. Write to him at hmarks@naol.com.

Water Under the Bridge

Software iSCSI targets aren't the only way to attach your servers to existing storage. You could use an iSCSI-to-SCSI or -Fibre Channel bridge instead. In fact, Hewlett-Packard's EVA iSCSI array is actually its Fibre Channel (FC) array with an iSCSI to FC bridge bundled in.

If your SCSI arrays are underutilized because you can only attach one or two host server to them, or if you're retiring a FC disk array, adding a low cost bridge will let you share that resource across more servers for just a couple of hundred dollars per server.

For years, storage administrators have used FC-to-SCSI bridges to attach tape libraries to their SANs. Many tape library vendors have bundled bridges from vendors like Crossroads into their libraries so they can use the same SCSI tape drives across their product lines.

You can follow their lead and stick an iSCSI bridge in front of your SCSI library. This will allow you to share between multiple backup servers and enable faster LAN-free or server-free backups.

Bridges, also called storage routers, are available from several vendors including Emulex, Paralan, Bridgeworks, Atto Technology and, of course, Cisco Systems. Prices range from around $1,800 for a single-port Gigabit Ethernet-to-SCSI bridge like the ATTO 1550D, to $5,500 for a four-port GigE-to-dual-port-FC bridge like the ATTO 2700R/D or Cisco's SN5428, which includes an eight-port FC switch with a street price of about $8,500. ISCSI-to-FC bridging is also a feature of higher-end FC routers and switches, such as the Cisco MDS 9216i and McDATA 1620 and 2640.

SANRAD's V-Switch products go a couple of steps beyond simple bridging, adding features like LUN management, data mirroring and fault tolerance, along with support for FC, SCSI and iSCSI back-end disks. Where a simple bridge provides iSCSI access to LUNs already defined on the back-end storage, a V-Switch can slice a disk array LUN into several smaller logical drives for iSCSI-attached hosts. You could put a V-Switch in a development lab and assign it a large LUN from a FC array; from then on the developers could allocate space to their servers without needing access to, or training on, the disk array's management interface.

V-Switches can also concatenate, stripe and/or mirror logical drives, making big logical drives out of multiple smaller ones. In campus or MAN environments, where you have gigabit bandwidth between two datacenters, you can use the V-Switch's mirroring to send data to arrays in both datacenters. SANRAD makes several models of V-Switches, starting at about $20,000. This may seem a little pricey but, if you have a bunch of SCSI RAID, you'll get a lot of replication, mirroring and volume management out of the deal.

We Built It, But Would We Buy It?

We built an iSCSI disk array from scratch and ran it through the Storage Pipeline torture-test suite as though it were a vendor-supplied product.

The first step was rounding up the hardware. Those building their own iSCSI drives would probably be limited to a meager budget, so we used SATA drives and the Dell 1600SC server already in the lab. We added an Adaptec 2850SA and AMCC's 3Ware 9550SX-8LP, both eight-port SATA II PCI-X controllers with 128 MB of memory.

For storage, we used five Western Digital 10,000-RPM 150-GB Raptors. Designed to offer enterprise SCSI or Fibre Channel drive performance with a SATA interface, our five-drive RAID should be as fast as any home-brew iSCSI system.

For the iSCSI target, we selected FalconStor's iSCSI Storage Server, which had impressed us in earlier reviews. Trying to eke out the last bit of performance, we used an Alacritech TOE card for the iSCSI connections.

It took an hour to install software and create a couple of logical drives. Then we hooked up a server and ran IOMeter benchmarks from a single server. The results were comparable to those of our Editor's Choice Equallogic PS100E for sequential I/O and still acceptable for small random I/Os.

We were curious as to how much the hardware for a capacity-oriented iSCSI array might cost, so we checked Newegg.com and put together a 3.2-GHz Pentium 4 server with 2 TB of raw storage and the same 3Ware 9550 controller used in our performance experiment. You can build it yourself with Windows Server 2003 and basic iSCSI target software, such as StarWind Professional, for less than $4,200--or about $2 per GB.

Executive Summary

We'd all like to be able to install top-notch, enterprise-class storage when the need arises. But in the real world, most IT professionals can't afford, justify or twiddle their thumbs waiting for a new enterprise array when they run out of disk space. We suspect every IT administrator was an amateur robot builder, rocket launcher or inventor in childhood. Here's how to use those skills to turn a server into an iSCSI disk array.

If you recently replaced direct attached storage with a new array to handle your critical applications, you can use iSCSI to provide temporary storage for a development or test environment, as storage for a disaster-recovery hot site or just to provide a lower-cost storage tier. We show you how to wrangle LUNs (logical unit numbers) to your advantage and avoid performance pitfalls, and explain file-based versus partition LUNs. While you may not be one of the scientists giving Steve Austin his bionic eye, we can give you the knowledge to fashion a homemade iSCSI disk array that's better, stronger, faster--without spending $6 million.