cloud@kiwi ~ $

ZFS: The Last Word in Filesystems

tags
#Zfs #Freebsd #Macos #Storage #Backup #Sanoid #Syncoid
categories
Infrastructure Operations
published
reading time
12 minutes

A comprehensive guide to ZFS - the filesystem that combines volume management, RAID, and data integrity into one powerful solution. Learn why ZFS is awesome, how to manage snapshots with Sanoid/Syncoid on FreeBSD, and even run it on macOS.

Why ZFS is Awesome

ZFS (Zettabyte File System) isn’t just a filesystem - it’s a complete storage management solution that fundamentally changes how we think about data storage. Originally developed by Sun Microsystems for Solaris, ZFS has become the gold standard for data integrity and storage management.

Core Features That Make ZFS Special

1. Copy-on-Write (CoW) Architecture

ZFS never overwrites data in place. Every write creates a new block, and only after the write is complete does ZFS update the pointer. This means:

  • No write holes (partial writes can’t corrupt your filesystem)
  • Atomic operations - your filesystem is always in a consistent state
  • Free snapshots - since old data is kept around anyway, snapshots are nearly free

2. End-to-End Data Integrity

Every block in ZFS has a checksum. Every. Single. Block. This provides:

  • Silent corruption detection - ZFS knows when data has been corrupted
  • Self-healing with redundancy - if you have mirrors or RAID-Z, ZFS automatically repairs corrupted data
  • No trust in hardware - ZFS doesn’t trust your disk controller, cables, or RAM

3. Integrated Volume Management

Forget LVM, mdadm, and filesystem layers. ZFS does it all:

  • Create pools from multiple disks
  • Add disks to pools dynamically
  • Create filesystems instantly from pools
  • No need to pre-allocate space

4. Snapshots and Clones

ZFS snapshots are instantaneous and space-efficient:

  • Create thousands of snapshots with minimal overhead
  • Clone snapshots for testing or development
  • Roll back to any snapshot instantly
  • Send/receive snapshots for replication

5. Compression and Deduplication

Built-in compression that actually works:

  • LZ4 compression is so fast it often improves performance
  • Transparent to applications
  • Per-dataset configuration
  • Deduplication for truly redundant data (use carefully!)

ZFS on FreeBSD: The Native Experience

FreeBSD has first-class ZFS support. It’s not a bolt-on - it’s deeply integrated into the system. Many FreeBSD users run their root filesystem on ZFS.

Basic ZFS Setup on FreeBSD

Creating a Pool

# Simple mirror pool
zpool create tank mirror da0 da1

# RAID-Z2 pool (dual parity)
zpool create storage raidz2 da0 da1 da2 da3 da4 da5

# Pool with cache and log devices
zpool create tank \
  mirror da0 da1 \
  cache da2 \
  log mirror da3 da4

Creating Datasets

# Create datasets with different properties
zfs create tank/home
zfs create tank/data
zfs create tank/backups

# Set compression
zfs set compression=lz4 tank/data

# Set quotas
zfs set quota=100G tank/home/user1

# Enable deduplication (careful!)
zfs set dedup=on tank/backups

Pool Management

# Check pool status
zpool status

# Check pool I/O statistics
zpool iostat -v 1

# Scrub the pool (verify all data)
zpool scrub tank

# Check scrub progress
zpool status -v

# Export/import pools
zpool export tank
zpool import tank

ZFS Best Practices on FreeBSD

1. Use Appropriate RAID Levels

# For 2-4 disks: mirrors
zpool create tank mirror da0 da1

# For 5-8 disks: RAID-Z2
zpool create tank raidz2 da0 da1 da2 da3 da4

# For 9+ disks: RAID-Z3 or multiple RAID-Z2 vdevs
zpool create tank raidz3 da0 da1 da2 da3 da4 da5 da6 da7

2. Always Use Whole Disks

# Good - whole disk
zpool create tank mirror da0 da1

# Bad - partitions (unless you know what you're doing)
zpool create tank mirror da0p1 da1p1

3. Set Appropriate Record Sizes

# For databases (small random I/O)
zfs set recordsize=8K tank/postgres

# For large files (video, backups)
zfs set recordsize=1M tank/media

# Default (128K) is good for most workloads

4. Enable Compression

# LZ4 is almost always a win
zfs set compression=lz4 tank

# For already-compressed data
zfs set compression=off tank/media

5. Configure ARC (Adaptive Replacement Cache)

In /boot/loader.conf:

# Limit ARC to 8GB (on a 16GB system)
vfs.zfs.arc_max="8589934592"

# Minimum ARC size
vfs.zfs.arc_min="2147483648"

Sanoid and Syncoid: Snapshot Management Done Right

Sanoid is a policy-driven snapshot management tool, and Syncoid is its companion for replication. Together, they provide enterprise-grade backup capabilities with simple configuration.

Installing Sanoid/Syncoid on FreeBSD

# Install from ports
cd /usr/ports/sysutils/sanoid
make install clean

# Or use pkg
pkg install sanoid

Sanoid Configuration

Create /usr/local/etc/sanoid/sanoid.conf:

#############################
# Global Settings
#############################
[sanoid]
# Run sanoid every 15 minutes via cron

#############################
# Template Definitions
#############################

# Production template - aggressive retention
[template_production]
    frequently = 0
    hourly = 48
    daily = 90
    monthly = 12
    yearly = 5
    autosnap = yes
    autoprune = yes

# Development template - shorter retention
[template_development]
    frequently = 0
    hourly = 24
    daily = 7
    monthly = 3
    yearly = 0
    autosnap = yes
    autoprune = yes

# Backup template - long retention
[template_backup]
    frequently = 0
    hourly = 0
    daily = 30
    monthly = 12
    yearly = 10
    autosnap = yes
    autoprune = yes

# Frequent snapshots for critical data
[template_critical]
    frequently = 4    # Every 15 minutes
    hourly = 24
    daily = 30
    monthly = 6
    yearly = 2
    autosnap = yes
    autoprune = yes

#############################
# Dataset Configurations
#############################

# Production databases
[tank/postgres]
    use_template = production
    recursive = yes
    process_children_only = yes

# User home directories
[tank/home]
    use_template = production
    recursive = yes

# Development environments
[tank/dev]
    use_template = development
    recursive = yes

# Critical application data
[tank/app/critical]
    use_template = critical
    recursive = no

# Backup storage
[tank/backups]
    use_template = backup
    recursive = yes
    process_children_only = yes

# Media files (less frequent snapshots)
[tank/media]
    hourly = 0
    daily = 7
    monthly = 3
    yearly = 0
    autosnap = yes
    autoprune = yes

Setting Up Sanoid Cron Job

Add to /etc/crontab:

# Run sanoid every 15 minutes
*/15 * * * * root /usr/local/bin/sanoid --cron

Or create /usr/local/etc/cron.d/sanoid:

# Sanoid snapshot management
*/15 * * * * root /usr/local/bin/sanoid --cron --verbose >> /var/log/sanoid.log 2>&1

# Daily sanoid check
0 2 * * * root /usr/local/bin/sanoid --cron --verbose --force-prune >> /var/log/sanoid.log 2>&1

Syncoid: Replication Made Easy

Syncoid handles ZFS replication with resume support, compression, and bandwidth limiting.

Basic Syncoid Usage

# Replicate a dataset to another pool
syncoid tank/data backup/data

# Replicate to a remote host
syncoid tank/data [email protected]:backup/data

# Recursive replication
syncoid -r tank/home backup/home

# With compression (faster over network)
syncoid --compress=lz4 tank/data [email protected]:backup/data

# With bandwidth limiting
syncoid --mbuffer-size=1G --compress=lz4 \
  tank/data [email protected]:backup/data

# Resume interrupted transfers
syncoid --resume tank/data backup/data

Syncoid Configuration for Automated Backups

Create /usr/local/bin/syncoid-backup.sh:

#!/bin/sh

# Syncoid backup script
LOGFILE="/var/log/syncoid-backup.log"
REMOTE_HOST="backup.example.com"
REMOTE_USER="root"
REMOTE_POOL="backup"

log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOGFILE"
}

log "Starting syncoid backup"

# Replicate production datasets
syncoid -r --compress=lz4 \
    tank/postgres \
    ${REMOTE_USER}@${REMOTE_HOST}:${REMOTE_POOL}/postgres \
    2>&1 | tee -a "$LOGFILE"

syncoid -r --compress=lz4 \
    tank/home \
    ${REMOTE_USER}@${REMOTE_HOST}:${REMOTE_POOL}/home \
    2>&1 | tee -a "$LOGFILE"

syncoid -r --compress=lz4 \
    tank/app \
    ${REMOTE_USER}@${REMOTE_HOST}:${REMOTE_POOL}/app \
    2>&1 | tee -a "$LOGFILE"

log "Syncoid backup completed"

# Rotate logs
if [ $(stat -f%z "$LOGFILE") -gt 10485760 ]; then
    mv "$LOGFILE" "$LOGFILE.old"
    gzip "$LOGFILE.old"
fi

Make it executable and add to cron:

chmod +x /usr/local/bin/syncoid-backup.sh

# Add to crontab - run daily at 2 AM
0 2 * * * root /usr/local/bin/syncoid-backup.sh

Advanced Syncoid Patterns

Pull-Based Replication

On the backup server:

# Pull from production server
syncoid -r --compress=lz4 \
  [email protected]:tank/data \
  backup/prod-data

Multi-Destination Replication

#!/bin/sh
# Replicate to multiple backup servers

DATASET="tank/critical"

# Primary backup
syncoid -r --compress=lz4 \
  $DATASET \
  [email protected]:backup/critical

# Secondary backup (different location)
syncoid -r --compress=lz4 \
  $DATASET \
  [email protected]:backup/critical

# Cloud backup (with bandwidth limit)
syncoid -r --compress=lz4 --mbuffer-size=512M \
  $DATASET \
  [email protected]:backup/critical

Snapshot Management Best Practices

1. Layered Retention Strategy

Use different retention policies for different data types:

# Critical data - frequent snapshots, long retention
frequently = 4 (every 15 min)
hourly = 48
daily = 90
monthly = 12
yearly = 5

# Normal data - balanced approach
hourly = 24
daily = 30
monthly = 6
yearly = 2

# Bulk data - minimal snapshots
daily = 7
monthly = 3
yearly = 0

2. Monitor Snapshot Space Usage

# Check snapshot space usage
zfs list -t snapshot -o name,used,refer

# Check per-dataset snapshot usage
zfs list -o name,used,usedsnap,usedds

# Find datasets with most snapshot space
zfs list -t all -o name,usedsnap | sort -k2 -h

3. Automate Snapshot Verification

Create /usr/local/bin/verify-snapshots.sh:

#!/bin/sh

# Verify snapshots exist for critical datasets
CRITICAL_DATASETS="tank/postgres tank/app/critical"
MIN_SNAPSHOTS=24  # Expect at least 24 hourly snapshots

for dataset in $CRITICAL_DATASETS; do
    count=$(zfs list -t snapshot -r $dataset | wc -l)
    if [ $count -lt $MIN_SNAPSHOTS ]; then
        echo "WARNING: $dataset has only $count snapshots (expected $MIN_SNAPSHOTS)"
        # Send alert (email, Slack, etc.)
    fi
done

4. Snapshot Holds for Important Points

# Hold a snapshot before major changes
zfs snapshot tank/app@before-upgrade
zfs hold keep tank/app@before-upgrade

# Release hold when safe
zfs release keep tank/app@before-upgrade

5. Test Your Restores

# Clone a snapshot for testing
zfs clone tank/postgres@daily-2023-03-17 tank/postgres-test

# Mount it somewhere
zfs set mountpoint=/mnt/postgres-test tank/postgres-test

# Test your restore process
# ...

# Clean up
zfs destroy tank/postgres-test

ZFS on macOS: Yes, It’s Possible!

While not officially supported by Apple, OpenZFS on macOS brings ZFS to your Mac. It’s perfect for external storage, Time Machine alternatives, and development environments.

Installing OpenZFS on macOS

Using Homebrew:

# Install OpenZFS
brew install openzfs

# Load the kernel extension
sudo zpool import -a

Important Notes:

  • Requires disabling System Integrity Protection (SIP) for kernel extensions
  • Not recommended for boot drives
  • Perfect for external drives and storage pools

macOS-Specific Considerations

1. Disable Spotlight Indexing

# Prevent Spotlight from indexing ZFS datasets
sudo mdutil -i off /Volumes/tank

2. Set Appropriate Permissions

# Create a dataset for Time Machine-style backups
zfs create tank/backups
zfs set compression=lz4 tank/backups
zfs set atime=off tank/backups

# Set ownership
sudo chown $(whoami):staff /Volumes/tank/backups

3. Automatic Pool Import

Create a LaunchDaemon at /Library/LaunchDaemons/org.openzfs.zpool-import.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>org.openzfs.zpool-import</string>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/local/bin/zpool</string>
        <string>import</string>
        <string>-a</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
</dict>
</plist>

Load it:

sudo launchctl load /Library/LaunchDaemons/org.openzfs.zpool-import.plist

macOS Use Cases

1. External Backup Drive

# Create pool on external drive
zpool create -o ashift=12 backup /dev/disk2

# Create backup dataset
zfs create backup/mac
zfs set compression=lz4 backup/mac
zfs set atime=off backup/mac

# Use with Time Machine alternative
zfs snapshot backup/mac@$(date +%Y-%m-%d)

2. Development Environment

# Create development pool
zpool create -o ashift=12 dev /dev/disk3

# Create project datasets
zfs create dev/projects
zfs create dev/databases
zfs create dev/docker

# Snapshot before major changes
zfs snapshot -r dev@before-refactor

# Roll back if needed
zfs rollback -r dev@before-refactor

3. Media Storage

# Create media pool with multiple drives
zpool create -o ashift=12 media mirror /dev/disk2 /dev/disk3

# Create datasets for different media types
zfs create media/photos
zfs create media/videos
zfs create media/music

# Set appropriate properties
zfs set compression=off media/photos  # Already compressed
zfs set recordsize=1M media/videos    # Large sequential I/O
zfs set compression=lz4 media/music   # Can benefit from compression

Advanced ZFS Features

1. ZFS Send/Receive for Backups

# Initial full send
zfs snapshot tank/data@initial
zfs send tank/data@initial | ssh backup.example.com zfs receive backup/data

# Incremental sends
zfs snapshot tank/data@day2
zfs send -i tank/data@initial tank/data@day2 | \
  ssh backup.example.com zfs receive backup/data

# Recursive send with replication stream
zfs snapshot -r tank/data@backup-$(date +%Y%m%d)
zfs send -R tank/data@backup-$(date +%Y%m%d) | \
  ssh backup.example.com zfs receive -F backup/data

2. ZFS Encryption

# Create encrypted dataset (FreeBSD 12.2+, OpenZFS 2.0+)
zfs create -o encryption=on -o keyformat=passphrase tank/encrypted

# Load key
zfs load-key tank/encrypted

# Unload key (data becomes inaccessible)
zfs unload-key tank/encrypted

# Change encryption key
zfs change-key tank/encrypted

3. ZFS Delegation

# Allow user to create snapshots
zfs allow user1 snapshot,mount tank/home/user1

# Allow group to manage their dataset
zfs allow -g developers create,destroy,mount,snapshot tank/dev

# Remove permissions
zfs unallow user1 tank/home/user1

4. Performance Tuning

# For databases - disable prefetch, use smaller recordsize
zfs set primarycache=metadata tank/postgres
zfs set recordsize=8K tank/postgres
zfs set logbias=throughput tank/postgres

# For sequential workloads - larger recordsize
zfs set recordsize=1M tank/media
zfs set compression=off tank/media

# For random I/O - enable sync writes
zfs set sync=always tank/critical
zfs set logbias=latency tank/critical

Monitoring and Maintenance

Regular Maintenance Tasks

# Weekly scrub (verify all data)
zpool scrub tank

# Check pool health
zpool status -v

# Monitor I/O
zpool iostat -v 5

# Check ARC statistics
sysctl kstat.zfs.misc.arcstats

# List snapshots and space usage
zfs list -t snapshot -o name,used,refer

# Check fragmentation
zpool list -v

Alerting and Monitoring

Create /usr/local/bin/zfs-health-check.sh:

#!/bin/sh

# ZFS health check script
POOL="tank"
EMAIL="[email protected]"

# Check pool status
STATUS=$(zpool status $POOL | grep state: | awk '{print $2}')

if [ "$STATUS" != "ONLINE" ]; then
    echo "WARNING: Pool $POOL is $STATUS" | mail -s "ZFS Alert: Pool $POOL" $EMAIL
fi

# Check for errors
ERRORS=$(zpool status $POOL | grep -E "errors:" | grep -v "No known data errors")

if [ -n "$ERRORS" ]; then
    echo "WARNING: Pool $POOL has errors:\n$ERRORS" | mail -s "ZFS Alert: Errors on $POOL" $EMAIL
fi

# Check capacity
CAPACITY=$(zpool list -H -o capacity $POOL | sed 's/%//')

if [ $CAPACITY -gt 80 ]; then
    echo "WARNING: Pool $POOL is ${CAPACITY}% full" | mail -s "ZFS Alert: Capacity Warning" $EMAIL
fi

Conclusion

ZFS represents a paradigm shift in how we think about storage. Its combination of data integrity, snapshot capabilities, and integrated volume management makes it the ideal choice for everything from home servers to enterprise storage.

Key takeaways:

  1. ZFS provides unmatched data integrity - checksums on every block, self-healing with redundancy
  2. Snapshots are cheap and powerful - use them liberally for backups and testing
  3. Sanoid/Syncoid simplify snapshot management - policy-driven snapshots and easy replication
  4. FreeBSD offers the best ZFS experience - native support, excellent performance
  5. macOS can run ZFS too - perfect for external storage and development
  6. Regular maintenance is key - scrubs, monitoring, and testing restores

Whether you’re running a home NAS, a development environment, or production infrastructure, ZFS provides the reliability and features you need. Combined with Sanoid and Syncoid, you have an enterprise-grade storage solution that’s both powerful and manageable.

Start small, test thoroughly, and enjoy the peace of mind that comes with knowing your data is protected by the best filesystem available.

Have you deployed ZFS in production? Share your experiences and tips in the comments below!