Erasure Coding (EC), is a method of data storage which helps improve data redundancy and reliability. In an age where data integrity and reliability of storage structures is so important, techniques such as Erasure coding go a long way to building robustness into Storage Architectures. In Erasure coding, the data stored across the disk is fragmented into bits and each of these fragments is then encoded for a high level of protection and then stored in different disks for high data reliability.
If you are someone who is looking to learn what is Erasure coding, then this post will take you through it in great detail. We are going to talk about aspects such as the origins of erasure code, the objective of erasure coding, how erasure coding works and the benefits erasure coding brings to the table. Let’s get started and get Erasure coding explained.
Origins of Erasure Coding
Erasure coding has a rich 40+ years of heritage and was introduced in communications systems to make them a lot more fault tolerant. Since the beginning of time, there have been many storage types in a wide range of shapes and sizes. However, there was one thing that was common across all of these storage types and devices, which was that they all could fail. Failure can happen because of multiple reasons including physical failure, natural calamities, power outages, etc. This can lead to a lot of hassle and waste of time and energy. Given this, engineers started thinking of ways to make the storage devices and data within them redundant.
The very first version of erasure coding was introduced in a very popular storage system reliability technique called as RAID-1. What happens here is basically every Byte of data which is stored on a drive is replicated on to another drive. Because of this if the main storage drive fails, the data is safe on the secondary drive and can be plugged in to resume operations. But this comes with a huge drawback which is also fairly obvious. Replicating all this data twice takes up a lot of storage space and a lot of resources are just spent on backup, which is not very efficient.
Overtime erasure coding has become a lot more advanced but it is important to understand the origins to have a very clear understanding of it. Let us move on.
Objectives of Erasure Coding
The basic objective of erasure code is, of course, to increase the data redundancy and secure a failsafe in case your primary data drive or storage system fails. But it does not end here. Owing to the high amount of storage space being utilized just for backup, a higher level need of ensuring data redundancy with better resource utilization was need of the hour. So simply put, erasure coding strives to create a failsafe for the data by data replication across multiple drives without sacrificing on resource efficiency. It has been observed that erasure coding is a perfect fit for advanced storage architectures such as Ceph Storage Architecture which is used in our Cloud Hosting, making it a more reliable and scalable hosting option.
How does Erasure Coding work?
This part is a little tricky as the data replication needs to happen without occupying a lot of storage space. Let me try and explain this in the simplest way possible. What erasure coding does is that it generates a mathematical equation known as polynomial interpolation. This equation splits the data into a considerable number of chunks, then extra or redundant symbols are added to provide protection from failures and to help in accurate data retrieval. These chunks are then distributed across multiple disk drives. The ratio of data chunks to checksum chunks determines the fault tolerance the system can achieve.
For instance, let ‘k’ be the number of chunks and ‘m’ be the number of redundant symbols. Now, a higher number of ‘k’ than ‘m’ would result in higher speed and capacity of the storage system and on the other hand, the inverse would mean greater redundancy. Depending on the requirement, the ratio of ‘k’ and ‘m’ can be set up to find the balance between capacity and redundancy.
Let’s look at this mathematically. Consider we keep ‘k’ as 10 and ‘m’ as 6 i.e. we have 10 data chunks and 6 redundant symbols. These 16 chunks will be placed on 16 different drives/nodes. What this would mean is that from the 16 drives that we have, any 10 of them can be used to recover the original 10 chunks of data and this also means any 6 drives can fail, which is much more than traditional RAID where it can only handle a failure of 2-3 drives. Moreover, based on the above calculation, you only need to add an additional 60% of the capacity to attain this high level of fault tolerance. The result of this can be summarized in the benefits of erasure coding in the upcoming point.
Benefits of Erasure Coding
First of all, erasure gives you a lot more control over the system to maintain the levels of capacity vs. data redundancy. If the system needs to work on a higher capacity to better performance, that can be done without sacrificing on the data redundancy. And if having a stronger failsafe is more important, the system can be made a lot more fault tolerant. This can vary from time to time as per the need of the hour. This makes it perfect for applications such as Cloud Hosting where you need a perfect balance of scalability to reliability and the option to move in one direction whenever required.
Another advantage of erasure coding is that it reduces the costs and resources needed to have a very reliable and data redundant storage system. As seen from the above example, even if 6 out of 16 drives fail, you will still be able to restore all data and all this by only adding 60% more capacity. This definitely saves storage and resource costs.
We at CloudRocket have adopted this in some of our advanced hosting options so that you can enjoy the benefits of this cutting-edge technology. Erasure Coding is integrated into our Cloud Hosting services, integrated with Ceph storage architecture.
Hope this helps you understand what is Erasure coding. Moreover, get a better grip on how it works and what are its benefits. If you have any further questions, please feel free to talk about them in the comments section below. We would love to see all your thoughts and spur up a discussion to discover more!
