What File System Should You Use for a NAS?

Sourced from real discussions on r/homelab, r/zfs and r/DataHoarder, cross-checked against live PricePerGig listings.

The ‘best’ file system for your data safety is ZFS, 100%, no questions asked, that IS the best file system for your data safety when put in any kind of redundancy pool. Btrfs should be just as good, but there is a fundamental design flaw that unless handled carefully, will leave you losing data the moment you try to rebuild the array. So if you want the simplest, most straightforward setup, ZFS it is, but ZFS requires you plan your setup, buy all the drives in one go (even though you’re not going to use that storage space for a few years, yes years, you’re supposed to be planning THAT far ahead with ZFS) and btrfs solves that problem, just add a disk whenever you want, ZFS is the way to go.

Because I’m a skinflint, I’m going to discuss the why’s and wherefores with ZFS and btrfs and then go how to overcome its main shortcoming, which, should leave you with the best setup ever. So let’s go.

The table above shows live prices for new CMR internal drives of 6TB and up on your marketplace, the drives that will sit under whichever file system you pick.

What are ZFS and btrfs for anyway?

You’ve probably heard of RAID, here is a quick rundown.

Hard drives fail, and when they do you lose all your data, that’s bad. Wouldn’t it be good if you could have a copy of all your data in two separate drives at all times? Yes that would solve the problem. Ok, I’m going to invent RAID 1.

The invention of RAID1

So I’m going to make a copy of all my files, and put them on another drive, and remember these are spinning drives we’re talking here. If we invent RAID1 correctly, we’ll also get a read speed boost, double the speed, the data exists on two drives now, so I can use the HDDs cleverly and when reading a large file, make each disk read different parts and join them together in just the right way so we get double the speed for reading… we did suffer slightly during write with slightly slower speeds, but that’s ok.

We just invented the ‘mirror’ and it’s great, but to store 2TB of data I need 4TB of disk space. That’s pretty bad efficiency.

The invention of RAID5

Fed of the inefficiencies of RAID1, I’m going to do more clever maths, and use 3 disk drives. Now, when I write a file, I’ll put it on one drive, and make sure a mathematical equation means that I’ll write a bit of data to drive 2 and 3 too, should any one drive fail, I can work out what the data was.

Fantastic, now I buy 3x2TB disks, and I can use 4TB, that’s 66% efficiency vs RAID 1’s 50%, not amazing, but pretty good.

I can also add 5 drives, 10 drives, however many I can connect, and just one is ‘wasted’ with redundancy, so with 5 disks, we’re up at 80% efficiency.

RAID5 causes a very stressful moment

So when one of my 3 disks fail, I will simply buy another drive, pop it in, and all the maths means that, all my files magically get copied to the new disk in such a way I have redundancy again and I can rest easy should a drive fail in the future. Fantastic.

Except.

What if, while we are ‘rebuilding the array’ aka resilvering; should a drive fail THEN, I mean, it’s a lot of maths, and takes about 24 hours of constant reading of my two aging drives, I will lose ALL my data if just one more drive fails, ALL of it, not just a bit, the whole array, even if I had 5 drives worth of data!

The invention of RAID6

I’m going to make RAID 5, but with TWO redundant drives, yes it’s more wasteful, but during a resilvering, I know I can still handle one more failure, now I can relax.

So now we’ve invented RAID5 and RAID6, I want it

Great, well along comes Sun Microsystems, they make CRAZY expensive Unix based system (I remember the revered computer rooms at UMIST University in Manchester UK - yes the same location that the world’s first computer program was loaded from memory no less, full of them). Nothing about Sun Microsystems is cheap, so do we think they are going to open source and give away their bomb proof implementation of ZFS?

Yes and No, Sun Microsystems released the code but you can’t use it commercially or bundle it with other licences such as GPL which is what Linux is distributed under, and to this day even the OpenZFS version is still under an incompatible licence.

So why doesn’t everyone and everything just use ZFS?

What all this licensing shenanigans means is that if you’re ‘giving away’ your product such as Linux, you have to jump through some hoops to also include ZFS, those hoops are what you jump through on your Linux distro to get it, if however you’re a commercial entity selling your product, it’s a no go.

Synology, UGREEN, TerraMaster and at the time Oracle system want ZFS but can’t due to licensing so…

Oracle set out to invent a ZFS equivalent and make it freely distributable, well, Chris Mason did, who was employed by Oracle at the time and then went on to work at Facebook via a few other places including Twitter.

The Invention of BTRFS, and its problem

Oracle sets out to make or remake a new RAID filesystem, and it turns out it’s harder than you think. A RAID file system has to cope with a lot of edge cases, including a power failure at any point in time, including when you’re half way through writing one file block, either in the first place or its parity information.

So imagine you’re the file system, you’re told to write down some information on one piece of paper (disk) and when you’re finished that or even at the same time write down some maths formula on another piece of paper so if you lose the first, you can recreate it. BUT, your pen runs out of ink just as you’re writing down the math formula. Basically everything is ok, I mean, the maths was only needed if you lose the first piece of paper and you’ve got it, so it’s fine… until you lose the first piece of paper.

If you lose a disk/the paper, and the maths formula didn’t finish, now you have a problem, you’ve lost that data forever.

That is exactly the antithesis of what RAID is supposed to do!

BTRFS can’t cope with a power failure at just the wrong time, so what can it do?

BTRFS has so many advantages over ZFS, not least one that would benefit every data hoarder, incremental addition to the data pool. The main problem with ZFS is you have to plan ahead and buy all your disks up front, and then use them. So buying a pool full of 28TB drives at huge cost, to fill them up over a number of years, at which point you would hope 28TB would be much, much cheaper, so you’re front loading the full cost.

With BTRFS, you can just add another drive when you’re ready, a little like unRAID.

One final warning about BTRFS

So how can I use BTRFS, I want incremental drive additions. The safest way is to have a UPS (a power system that gives your computer 5 mins or so to finish off before a controlled power down).

However, you’re still not protected from some random failure of a drive, which I’ve never known to happen, but apparently happens.

The other thing you can do is rebuild the parity every now and then, so that will help. But rebuilding is the one of the most stressful things you can do to your drives, so it’s counter productive to do it too often, you can do it just after an uncontrolled shutdown though - you could easily automate that with a file written on startup and deleted on controlled shutdown, a ‘dead man’s switch’ kind of thing.

Other incremental drive addition RAID systems

The one I’d recommend is unRAID, although that’s more of a whole operating system than just a drive format/filesystem. So you’re signing up to more than you think, it’s pretty capable though and I use it myself. But if you want to use BTRFS, get yourself a UPS for about £50 and you will be golden.

← Back to the NAS & Enterprise hub