I really want to run ceph because it fits a number of criteria I have: gradually adding storage, mismatched disks, fault tolerance, erasure encoding, encryption, support out-of-the-box from other software (like Incus).
But then I look at the hardware suggestions, and they seem like an up-front investment and ongoing cost to keep at least three machines evenly matched on RAM and physical storage. I also want more of a single-box NAS.
Would it be idiotic to put a ceph setup all on one machine? I could run three mons on it with separate physical device backing each so I don’t lose everything from a disk failure with those. I’m not too concerned about speed or network partitioning, this would be lukewarm storage for me.
“As easy as buying four same-sized disks all at once” is kinda missing the point.
How do I migrate data from the existing z1 to the z2? And then how can I re-add the disks that were in z1 after I have moved the data? Buy yet another disk and add a z2 vdev with my now 4 disks, I guess. Unless it is possible to format and add them to the new z2?
If the vdevs are not all the same redundancy level am I right that there’s no guarantee which level of redundancy any particular file is getting?