Until now, data redundancy in a Ceph environment has been a very expensive line item for any data center due to the requirement of creating three copies of data. If a drive fails, there are still two additional copies of the data to recover from. The model of three times replication is expensive as you need to buy and maintain three times the useable storage you need today, in order to store the data with redundancy. And so we turn to erasure coding.
Erasure Coding (EC) works by processing data through an algorithm that breaks it up into chunks, writing a single copy with extra parity bits that can be used to rebuild the data in the event of lost media. Normally, EC is also expensive as it requires high-power CPUs, essentially shifting the cost of data parity to the CPUs of the storage appliance. This extra load slows storage writes and the only way around it is to use expensive, power-hungry CPUs.
SoftIron’s engineering team created a dual-purpose EC accelerator + 10GbE SFP network interface to solve this issue, and in so doing, have radically reduced the costs and complexity of data redundancy. It does so without the huge expense of triple replication and without the CPU penalty of traditional Erasure Coding.
Custom-built for HyperDrive
Offered as an I/O (NIC) option for HyperDrive® – our custom-built, dedicated Ceph appliance for software-defined storage – it works as an I/O module that computes Erasure Coding on the fly at line rate, removing the load from the CPU while also providing a 10GbE SFP interface.