The gear we’re testing today:
- Dell Equallogic PS6500 (7200rpm disks ~48 drives)
- EMC VNXe3200 (7200rpm disks ~24 drives)
- Dell/EMC SC4020 (ssd version ~12 drives)
- 45Drives S45 (7200rpm ~45 drives)
- 45Drives Q30 (ssd version ~12 drives)
- Supermicro 45 bay JBOD (7200rpm disks ~45 drives)
- My local machines ssd (just for fun ~1 drive)
- All backed by a 10gig fiber network
For this test we’ll be primarily using CrystalDiskMark, but later we do end up using some old school linux dd commands. All CrystalDiskMark readings were done over iSCSI connections to backend storage devices, and had defaults set to: 8gb test file, 32 queues, 8 threads, 5 passes
…One other important note, crystaldiskmark appears to have a bug when testing the 45drives Q30, it was flagging about 300mb/s read speeds even while pegging out a 10gig pipe. This leads me to believe there is an unknown bug in the software, and as such I’ve normalized that result in the bar graphs by just setting it to its theoretical max.
First up we have all networked devices (iSCSI 10gig link). Note that the SC4020, and both 45drive units capped out the max write speeds on the 10gig link, with only the 45drive S45 unit not maxing out the 10gig link on read speeds.
Next up we have the same chart, but it now includes my local disk ssd, as well as a direct attached (SAS) jbod device. Notice how the JBOD device wins across the board, only because of its lack of hitting the 10gb link cap speed.
An interesting note, most likely to the benefit of ZFS, is that random read and write performance across all devices. Here we can see a few things much more clearly, as well as something closer to real world use. Notice how the traditional spinning disk devices score very low, while ssds score fairly well. Another interesting thing is how well ZFS handled randoms across the spinning disk device, actually outperforming both purely SSD arrays, most likely because of the higher spindle count.
I also did some non networked raw benchmarking between the 45drives units using the linux dd command to test ssd vs spinning disk (12 vs 45 drives). Results showed a 17% performance increase when using ssd drives, at 3.1GB/s and 2.2GB/s respectively. These results are based on a single sequential write.
One final note on CPU usage and the 45Drive units. The 45Drive units nearly maxed out CPU usage on their respective units while using ZFS for raidz2 and compression lz4 only. The S45 unit has two processors at 8 cores each and would peak at around 70% cpu usage while the Q30 with only one processor would hit 80% usage. Usage spikes always matched random write speed tests. As with anything ZFS related, memory was in full use for ARC cache.
Raw result data:
Because sometimes you want to look at hard numbers right from the crystaldiskmark results..
Dell SC4020 all SSD array:
EMC VNXe3200 spinning disk array
Dell Equallogic PS6500
45Drives S45 spinning disk array
45Drives Q30 SSD array (note the bug in read speed results here, I ended up normalizing it at 1222mb/s for the final result graphs)
Supermicro JBOD with hardware raid and SAS
My local machines SSD drive
45Drive S45 raw write performance
45Drive Q30 raw write performance