I’d been using ZFS with Void linux on both my laptop and desktop for a couple of months. And ZFS is cool! But I’m thinking not great for my use case, especially for my laptop with it’s more constrained resources. Memory usage was a real problem, even after imposing low ARC limits. And the kernel module compile time was long enough to be a bit annoying, especially for a few kernels (I like to keep the last few around, to be safe) as it happens fairly often on a rolling release.
I switched the laptop to LUKS/btrfs a couple of days ago. And I’m thinking that was the correct choice for that. And now I’m considering doing the same for my desktop. As they seem comparable but btrfs is in-kernel and seemingly more system resource friendly. But before doing so I figured I’d ask the community about it. Maybe some important factors or features for either setup that I might not be considering.
Here’s the stuff I care about. All of which both offer, but I’m not an expert at either and I don’t know how equal they are.
- Disk encryption. For ZFS everything (except the EFI partition) is encrypted. I use ZFSBootMenu in this scenario. For the btrfs setup I have the kernel/initramfs on an ext2 partition. I do not store any decryption keys in the initramfs. I know grub can decrypt LUKS with limitations, but I prefer this setup. And it feels secure enough to me. Any pitfalls I’m missing?
- Pools/subvolumes
- Snapshots. ZFSBootmenu has an option to load a snapshot. For btrfs it looks like I’d need to create a subvolume from a snapshot, which in a recovery situation might mean doing this from recovery media. That’s ok, given this is an unlikely thing to encounter. But if anyone knows of an easier way, I’d love to hear it.
- CoW
- RAID 1
- Compression is nice, especially for the laptop
Edit: typo in title.
You must log in or register to comment.
(half replying to other comments as well as yours)
If you have a look at the btrfs mailing lists post that introduced RAID1c34, they were created because RAID56 were not considered viable or fixable. It’s in couched language but reasonably clear. I don’t think you’re thinking of using those (RAID56) but don’t.
Never had any btrfs problems that weren’t self generated or date from a really sticky period in btrfs’s history (years ago, 4.13 or maybe 3.13). I’ve used RAID56 until RAID1c34 became available and RAID10 where I could.
Haven’t tried LUKS - btrfs though, although effectively no worse than putting btrfs in a VM (which is fine if slow at the time), albeit a bit more computationally intensive.
IME, btrfs is easier to work with than ZFS. It has all of the features you asked for; its RAID ≠ 0/1/10 are buggy, but 0/1/10 are considered reliable. In the past year, I heard a rumor that they were going to announced RAID > 1 to be also stable, but that’s hearsay; I haven’t read anything authoritative on the subject - the Arch btrfs page and the btrfs man page both still say 5/6 are not reliable.
I’ve been using btrfs on a variety of computers and VMs, from tiny little ODROIDS, to laptops, to VPSes, to desktops for… over a decade? I’ve had much better reliability than ext4. I was attracted to the POLS of the commands, vs ZFS.
I don’t know how much my opinion weighs; I have a feeling a data center person would suggest ZFS as being more “enterprise”. I’ve been really happy with it. I’ve been watching bcachefs for the caching and target options - really neat features useful for home gamers - but otherwise I wouldn’t bother - btrfs has been solid and done everything I could want. It was a huge upgrade from mdadm and lvm in UX, and was only possible when disks got so cheap they outpaced my need for RAID5, and I could afford multiple backup drives that held years worth of nightly incremental backups.
Hearing roughly a decade of successful use, especially on systems with constrained resources, certainly makes me lean further towards btrfs.
its RAID ≠ 0/1/10 are buggy, but 0/1/10 are considered reliable.
btrfs has been solid and done everything I could want. It was a huge upgrade from mdadm and lvm
@[email protected] said that btrfs is poor at software RAID. I’ll do a little research in to how it fares for RAID 1 vs mdadm. I don’t see any reason I couldn’t do mdadm>luks>btrfs if that’s the better choice. But if btrfs is reliable and with comparable performance, I’d certainly rather do that.
I have no doubt ZFS is solid, too, FWIW. I leaned toward btrfs because it was simple, the commands straightforward and clear, nothing required more than one step - this is all super valuable to me because there are other things I want to spend my time on than fiddling with the filesystem.
@[email protected] said that btrfs is poor at software RAID.
You should check for yourself. I haven’t used software RAID in years - RAID 0+1 gives me no value - but the btrfs team and Arch wiki say 0, 1, and 10 are solid. You should not use 5 or 6, as they’re known to be buggy and even the btrfs man page tells you to not use it. So, yeah? btrfs is poor at RAID 5/6; to my understanding, it’s good at 0/1/10.
btrfs can do encryption, compression, snapshots, and some RAID. I found combining mdadm and lvm and FS built a jenga tower, of which if part failed, the entire end result was borked. I once did an OS upgrade and lost the mdadm config, and spent two days recreating it. I never used it on a new machine after that. Separation of concerns is great, but having an all-in-one that can self repair and boot into snapshots is better.
I can’t speak to performance. No doubt Toms of someone like that’s looked into that in detail.
Encryption in btrfs? https://wiki.archlinux.org/title/Btrfs#Encryption
You’re right, good catch. Now I’m going to have to go see how it’s set up; I thought it was being done in btrfs.
btrfs because it was simple
Personally I found ZFS far more simple. The userspace tools make more sense to me. Also I like, that volumes can have a default (relative) mount point attached. So in a recovery scenario, I simply have to open the zpool with a relative base path, and then have all my volumes ready to go. If I want to recover a btrfs system with multiple subvolumes, I typically need to know exactly which ones and where to I have to mount them (each individually).
Also I go really used to
zfsbootmenu.The issue last time I looked into btrfs mirrors was it’s poor reporting of disk problems and letting it boot with a borked drive. Might be fixed, but that was a 10 year old unresolved bug at that time. Seemed like a WONTFIX and I didn’t need that for a server OS drive.
For a workstation, btrfs is probably fine. It’s the shits at software RAID, but that’s rarely a thing on a workstation.
Look at
btrfs-assistantfor adminstration. That’s what Fedora ships with, I think it uses Snapper in the backend.It’s the shits at software RAID, but that’s rarely a thing on a workstation.
I am using a RAID 1 mirror over two disks. So that’s good to know. I’ll do a little research and see if it’s better to let mdadm handle that.
Look at
btrfs-assistantfor adminstration. That’s what Fedora ships with, I think it uses Snapper in the backend.Doesn’t look like that’s in the void repo. But that’s ok, I don’t mind learning the command line tools.
It’s not good at letting you know when a disk is borked. And normally if you reboot a mirror with a bad disk, it will complain so you know to fix it even if you missed the log entries about it being down. Btrfs will quietly just let you boot into a potentially lethal situation for a mirror with a bad slave.
And there was something about scrubs that was janky as well
I think btrfs ticks all your boxes. I would suggest yabsnap for snapshots. Then if you want a backup off-disk use borg or btrfs has a way to transmit (sync) to a remote. Yabsnap has a command which will make a script to restore from backup, which you can review and run.
Something to consider, advice given to me, is that ZFS support on Linux regularly breaks with newest kernels so if you go for ZFS long term, be prepared to run a lts kernel at least as a backup.
I use both. LUKS+btrfs being nice on the Arch desktops, and ZFS on a serverside pool, managed by a TrueNAS Scale VM.
Not answering your question but I had installed btrfs on my fedora install, thought I would use it for backups and system restores using snapshots.
But I felt there was always a performance tradeoff when doing a lot of writes like npm install and stuff.
Eventually replaced btrfs with ext4 and backup solution like timeshift.
I would say if you want system recovery then tools like timeshift make it really really simple and straightforward taking backups and restoring them.
Sometimes you just don’t need a Swiss army knife to do most basic stuff.
Using a filesystem which has NONE of modern features because you “noticed” performance tradeoffs…
I wasn’t familiar with timeshift so I took a look at it. My primary use case for snapshots is to take one before updates. So I can load from the snapshot if there’s issues. It doesn’t look like using it with ext4 would fulfill this use-case. But it looks like it also supports btrfs snapshots so could be useful as a UI to configure that.
The backups with time shift are incremental, hence most of the time the backup is taken within seconds and it only stores changes over time, something similar to git.
I used to do it exactly for that uses case, the backup was quick because there generally are not much changes outside the home directory.
I used to have Daily backups and monthly backup like 20 different dates stored in a relatively small space.
Like if my system is 30 gb then a 50 gb backup partition would store months of daily backups.
Ext4 on LVM can do both volume mirroring and snapshots. The is no COW support with ext4 though.
By the way I use BTRFS with LUKS on my workstation and have for 4 or 5 years. Primarily I like it for the snapshoting. I though I would like COW but frankly very mixed on that especially since there are cases you should not use COW and if you disable COW you loose snapshotting on that file. I have not used the raid capability. One thing I do not like about BTRFS is that I know of no way to track a bad block at the sector level to what file it is in if any. With Ext4 you can.
Another useful backup tool is restic.
deleted by creator
BTRFS for raid mirroring is fine, but any raid 5 you should steer away from.
I use is on VM’s and laptops all the time, wrapped with LUKS. I also like it for a home server, where I combine bcache (not bcachefs) with it, so I can have a 512g nvme cache in combination with two 16 TB spinning disks. Run tons of containers on it and it’s rock solid. The cache really helps the performance. It’s a file server, few web sites, a gitlab, a ghost host, nextcloud, media server, backup server, vm host, and more. And it’s chugged along great for 3 years.
