@OP In my experience most people who frequent this list are professional or serious hobbyist who have chosen OpenBSD not because of the lack of the experience with other OSs but rather because of their poor experiences with other OSs including FreeBSD.
Comparing to FreeBSD, OpenBSD feels simpler, more strait-forward, more stable, and better documented. Short of high end (30-40K U.S. dollars) specialized hardware OpenBSD wins router contest with FreeBSD hands down for home or small office. As somebody pointed few years ago PF on FreeBSD feels like unwanted bastardized step child. FreeBSD comes without IPsec unlike OpenBSD which has its own OpenIKED. As you know even OpenSSH on FreeBSD comes from OpenBSD. Enough said for intelligent. Be warned though that some people consider OpenBSD just an OS, not a firewall... They even came up with the name for "real firewall". It is called pfSense. File server is a little longer story so continue to read this long message if you have a time. [email protected] wrote: >> At this point, the FreeBSD camp would point out that they have ZFS >> for infinite flexibility in building multi-terabyte storage pools, > >>That said, both modern SSDs and multi-terabyte spinning >>platters are handled quite well, thank you, by FFS2 on OpenBSD > >As an aside, people sometimes confuse ZFS with XFS or GlusterFS or other > No we don't. We compare apples to apples. So lets compare ZFS to fully functional HAMMER1 a default file system of DragonFly BSD circa 2008 which was forked of FreeBSD 4.8. by Matthew Dillon. Between HAMMER2 is more functional than Btrfs if you want to try. >stuff. ZFS is designed around extreme data reliability and integrity, >not huge array size or high end performance. ZFS is an all-in-one >disk+filesystem that incorporates partitions, multi-parity RAID, >backups, and directory structure into one unified thing. It features >raid-write-hole prevention, triple-redundant checksumming of both data >and metadata, built-in block duplication, advanced journaling, atomic >copy-on-write, and the ability to snapshot arbitrary parts of the system > >which can then be rolled back after a problem, among other things. ZFS >is far more than something that 'just handles multi-terabyte pools'. > I was not aware that FreeBSD foundation has a PR department. This is how I, an OpenBSD user who happen to use ZFS/FreeBSD at work (big data guy here) and HAMMER1 occasionally describe the difference to my 8 year olds. ZFS is a combined file system and logical volume manager originally designed by Sun Microsystem. Porting it to FreeBSD required reimplementation of the large part of Solaris kernel and importing huge amount of CDDL code. It is designed for large data center for which failover, redundancy and high availability is must. Redundancy means that the data is typically stored on a volume consisting of multiple physical HDDs in such a fashion that malfunction of a single or even several drives doesn't affect data consistency and availability of data. ZFS can for all practical purpose be thought as a software RAID. The following RAID disciplines are available for ZFS, mirror, RAID-Z, RAID-Z2, RAID-Z3. In the layman one picks up typically 6, 7 or more drives and combine them using ZFS into the single volume which in ZFS lingo is known as a ZFS pool. Those drives are physically attached to a computer with a Host Bus Adapter and exposed to the ZFS as JBOD (Just a Bunch of Disks). In typical deployment file servers with multiple ZFS pools as large as 40-50 TB are common. ZFS pools are pretty trivial to monitor and FreeBSD has excellent integration with S.M.A.R.T. daemon. ZFS pools are portable and easy to import from computer to computer and even across OS. It is possible to use ZFS Volume as a iSCSI Target on FreeBSD does support ZFS Volume growth. HAMMER is just a file system. BSD licensed file system! That means that if one wants a large logical volume or more importantly high redundancy and availability one should be using HAMMER in combination with hardware RAID. Two brands of hardware RAID cards come to mind: Areca and LSI MegaRAID cards. Areca cards are on FreeBSD supported by arcmsr() driver while newer LSI MegaRAID cards are supported by mfi() driver. The same is true for DF BSD. The immediate questions will be how does one monitor those cards and if it is possible to pass the status of HDD to SMART daemon. I am aware of the two set of tools to monitor LSI cards mfiutil and proprietary sysutils/storcli. Areca card should be supported even better than LSI as they are open hardware. There is proprietary tool sysutils/areca-cli for Areca for inquiry/monitoring of Areca cards. I am not aware of any special tool that can monitor HAMMER file system itself. DF uses /dev/serno for drives which enables volumes to be imported from one machine into another. HAMMER can't be grown. Personally as somebody who has a limited working knowledge of both with I actually prefer hardware RAID even though we use ZFS/FreeBSD for other reasons. Note that typical home user in an era of 3TB HDD which sell for less than $100 is unlikely to have a need for large logical volume and consequently also unlikely also to benefit from high redundancy and availability. Once you have a ZFS pool or HAMMER file system on the top of Hardware RAID you will need to create ZFS datasets or HAMMER pseudo file systems PFS for short. In that respect both systems are similar. A single ZFS pool might contain multiple ZFS datasets with different properties. The really cool feature of ZFS includes data compression. I personally like lz4. HAMMER volume can also contain multiple PFSs with different properties (master/slave) but no nested PFSs. I think that support for compression on HAMMER was in works if not already committed. Lets talk little bit about data protection. ZFS has copy-on-write, check-sums, and consistency but so does the HAMMER1. Depends on the type of the pool multiple HDD failures are permitted. RAID-Z3 discipline allows pool to remain fully functional even in the case when 3 HDD are dead. FreeBSD supports hot replacement of the failed HDDs as well as hot spare daemon on the selected hardware unlikely to be owned by an uninitiated person. Apart of the fact that hardware RAID6 has two disk redundancy everything else is available to users of hardware RAID. I like most ZFS people I know use only RAID-Z2 discipline so this is no winner contest. FreeBSD supports GELI full disk encryption when creating ZFS volumes so does DragonFly by means of device mapper target called dm_target_crypt (compatible with Linux dm-crypt) that provides transparent disk encryption. It makes best use of available cryptographic hardware, as well as multi-processor software crypto. DragonFly fully supports LUKS (cryptsetup) and TrueCrypt as disk encryption methods. tcplay, is a free (BSD-licensed) 100% compatible TrueCrypt implementation built on dm_target_crypt. I am not sure who wins this contest. I guess Oracle Solaris who has native ZFS crypto discipline. You mentioned log (ZIL) and L2ARC. Please check DragonFly SWAPCACHE. Advanced Journaling: Somebody pointed out to me that I don't use word journaling properly. A journal is purely a log of in-flight changes being made to a file system so it can be quickly made consistent after a unclean mount. An example of such file system which I would like to see on OpenBSD would be WAPBL. HAMMER support that. Personally I think of a Journal as built in version control system with data recovery support. ZFS supports journaling via periodic snapshots. Those are typically done as cronjobs. There are multitude of tools in FreeBSD ports which can be used to do snapshots I personally like sysutils/zfsnap but people might like others. HAMMER also supports snapshots. The default installation takes snapshot via daily periodic scripts and keeps them in /var/pfs for sixty days. On the top of it HAMMER support fine grained journaling via history. This is where HAMMER kicks ZFS rear end so bad that it is not even funny. Check out how this work with slider https://www.dragonflybsd.org/docs/docs/howtos/howtoslide/ Both ZFS and HAMMER journaling are both NFS and Samba aware which means that users directories exported via NFS or Samba and mounted on other OSs are fully supported by ZFS or HAMMER journal. One should mentioned though that DF people have given up on NFSv4 but we should also mention that their implementation of NFSv3 seems very robust and the fastest I am aware of. ZFS is performing continuous integrity checking and automatic repair. You have to use hammer history to do that on DF. DF wins this round handsomely Backup and Recovery: One typically uses ZFS replication to backup ZFS pools. Replication is of course network aware (uses OpenSSH to send data to remote ZFS server). It is done in deltas and is extremely efficient. One can do deduplication of blocks on the fly during ZFS replication. One can also use additional file system level compression to send deltas. Remote replicates of the file system are fully writable. However note that snapshots are needed before you can replicate your system. Multiple targets are allowed. Remote replications are fully functional remote clones. HAMMER uses hammer-stream for backup. It is network aware. One can have multiple targets (PFS slaves). Those are not writable. Note that a slave PFS can be promoted into the master. However one has to be aware of the problem with the time. Only t-time is preserved. PFS slaves are clones but not fully functional until promoted into the masters. I would be hard press to declare the winner of this round. Inquiry: Making inquiry about ZFS pool or HAMMER FS is very easy. zpool status or hammer pfs-status /data Monitoring: FreeBSD has enterprise level ZFS monitoring. IPMI, SNMP, S.M.A.R.T all work as expected. Tools like Nagios or Collectd have plugings for ZFS monitoring even the things like L2ARC. Monitoring in DragonFly BSD is challenging to say at least. I was appalled that net-mgmt/collectd5 fails to compile on DF. This is the reason I personally opted out of DF at work. But if you are home user please continue to read. FreeBSD wins this round. Alerting: Similar situation as Monitoring. Miscellaneous remarks: It is possible to use ZFS as boot environments. One could use ZFS mirror for a root partition. sysutils/beadm is a killer feature of FreeBSD and ZFS. It allows one to role to pre-update/up-grade fully functional version of OS in the case something goes wrong. DF BSD uses UFS for /boot. Boot is typically less than a 1GB. The rest of the system is HAMMER. DF installer doesn't support installation of PFS (master/slave) or for that matter on the pair of disks. Personally I hold a view that a large file server running DF will have OS installed on small SSD drive and use hardware RAID or physical drives for data. It is possible to use ZFS Volume as a iSCSI Target. I have no clue what is the state of iSCSI support on DF BSD. I have not seen any evidence of iSCSI support in HAMMER man pages. One of favorite FreeBSD are Jails. Tools as sysutils/iocage enable great integration of Jails and ZFS pools. Taking hot ZFS snapshot of a Jail and cloning it remotely is really cool. Similar tool is in works for Bhyve which will be really cool. DF Jail infrastructure has not being touched for a long time. I am not aware of DF Jails being able to take advantage of HAMMER. Since we are on the OpenBSD mailing list one should notice that Jail is flooded security concept from the point of view of mathematical provability as pointed out by our own Kristaps Dzonson https://www.youtube.com/watch?v=JaVnNllZxn4 Sorry I have to side with a fellow mathematician on this one. Final Remarks: ZFS Deduplication seems better more stable than HAMMER deduplication. However HAMMER boost off line deduplication. ZFS deduplications require tons of ECC RAM. ZFS is a no-brainer for large data centers (couple hundred servers). For home users in particularly those who have no more than 2-3 TB of data having data on the pair of PFS mirrors is very tempting and probably much more cost effective than similar FreeBSD set up. I will end this long write up with a story of a gentlemen who is doing humanitarian work in Africa Tanzania. He wrote an e-mail to DF mailing list after being directed there by few OpenBSD developers. He was trying to put together a network of electronic libraries in the part of Tanzania where power grid is barely existing and not very reliable. Those electronic libraries are now running DF BSD and use HAMMER1 as a default file system. >Now, whether a home user NEEDS all these reliability features is a >different question, but if you decide you do, OpenBSD (along with most >other *nixs) doesn't have anything remotely comparable. > In mathematics we call your last statement argumentum ad ignorantiam. Best, Predrag > >>That said for FreeBSD and ZFS you want at least 4GB of >>ram anyways. > >This is a common misconception. The ARC wants to cache your entire array > >in ram if it can, so it will expand to fill whatever's available. You >can run ZFS with limited ram, you'll just see a performance hit if you >try to do lots of random reads on things that aren't cached.
