The File System Powering Nimble Flash Arrays
by Umesh Maheshwari – Co-Founder and CTO
Businesses rushing to add flash storage to their data centers are discovering that there are massive differences in the performance, capacity, reliability, and scalability of flash arrays from different vendors. What makes the difference? It’s the file system – the core software that manages data on storage media.
In 2010 Nimble launched its family of hybrid flash arrays containing both flash and disk drives. Underpinning these hybrid flash arrays is a file system we call CASL. Earlier this year, Nimble launched its all flash arrays. The file system underpinning these all flash arrays is an extension of CASL.
Which raises a good question: How is it that CASL can work so well for both hybrid flash arrays and all flash arrays?
Let’s start with the foundation established by the original version of CASL. Although many vendors offer hybrid flash arrays, CASL differentiates Nimble hybrid flash arrays in two fundamental aspects:
- Cache accelerated: Reads are cached in flash, which excels at fast reads. Writes are cached in NVRAM, which excels at fast writes and does not wear out like flash does.
- Sequential layout: For both flash and disk, CASL translates random writes in the logical address space to sequential writes on the storage medium. This helps sustain write throughput and supports compression.
The result is that Nimble hybrid flash arrays adapt much better to diverse workloads than other flash arrays; that’s why we call them adaptive hybrid arrays.
Given this foundation, there are three main reasons why CASL works as well for all flash arrays:
- It was designed for flash right from the beginning.
- Sequential layout is highly beneficial for both disk and flash.
- Nimble Engineering continues to evolve CASL to leverage advances in hardware.
Designed for Flash
Nimble Storage developed its first all flash system in 2008, long before flash arrays became popular – a 2U box with 24 flash drives. It would intercept NFS (Network File System) traffic between clients and servers, store recently accessed data, and accelerate future accesses. We called it the “accelerator”.
In designing the first version of our file system, we pored over the specs published by flash chip vendors, and noticed a big gap in the performance of random and sequential writes. So we opted for a write-optimizing sequential layout, an architectural choice that has worked out well.
In 2009, we decided to forgo the accelerator in pursuit of the whole storage system – a hybrid flash array. We already had our own file system for flash, and we considered using a pre-existing file system for disk. We soon realized that sequential layout is good for both flash and disk, and that even modern file systems such as ZFS do not implement sequential layout fully.
So we decided to use our own file system for both flash and disk, though we did have to extend it beyond the original accelerator. The fact is that we designed our file system for flash first, and then extended it for disk.
No wonder CASL works so well for our all flash arrays. Flash is where it was born.
Sequential Layout is Beneficial for Both Disk and Flash
Whenever a storage medium performs sequential writes faster than random writes, it is helpful for the file system to translate random writes to sequential writes. This is true for disk because random writes require mechanical movement of the head. It is also true for flash because, before a multi-kilobyte page of flash can be written, a multi-megabyte block containing the page must be erased.
A major problem with other file systems is that they try to optimize writes by merely adding a write cache. That trick works well for reads, but not for writes. A read cache greatly reduces the number of reads reaching the backend storage, but a write cache only buffers and defers the writes from reaching the backend storage. When the write cache gets full, it needs to be drained. In many systems, this causes the write throughput to fall off a cliff. Employing sequential layout on backend storage enables fast draining and therefore helps sustain fast writes.
There are two other benefits to using sequential layout:
- It accelerates parity-based RAID. Suppose each RAID stripe has 12 data and 3 parity chunks. Without sequential layout, every random write might result in updating 3 parity chunks, thus amplifying the number of writes by a factor of 4. With sequential layout, the update of parity chunks is amortized across the full stripe, thus reducing the write amplification factor to (12+3)/12 or 1.25.
- Sequential layout supports variable block sizes because blocks can be concatenated into stripes without regard to aligning block boundaries with sector boundaries on the storage medium. Variable block sizes, in turn, support fast compression, application-specific tuning, and the ability to append to each block a self descriptor including checksums.
Nimble Engineering Continues to Evolve CASL
In designing storage arrays, our goal is to excel across the many dimensions that storage users care about, while keeping the cost low. This goal is a moving target, because hardware keeps advancing.
Disk storage was invented in the 1950s. Flash was invented in the 1980s, but did not appear in enterprise data storage until late 2000s. Now, less than a decade after flash arrived, there is speculative talk of new forms of non-volatile memories.
The pursuit of excellence requires that we evolve our file system to leverage the advances in underlying technologies. So far, we have evolved the file system from supporting all flash accelerators through adaptive hybrid flash arrays to all flash arrays. The figure below shows a simplified timeline of CASL’s evolution.
We are fortunate that CASL has continued to serve as an excellent foundation, but we have also extended that foundation in significant ways. More than the fortunate choice of sequential layout, it is our will and ability to evolve that enables CASL to excel.
The evolution does not stop with all flash arrays. When the next generation of non-volatile memories is ready, there will be a new wave of hybrid arrays combining flash and such a memory that will make the all flash arrays of today look old. And the cycle will continue.
Our goal is to continue to evolve CASL to leverage advances in hardware technologies – not because we want to be the first to use them, but because we want to provide an excellent product to users at low cost.