How to speed up zfs resilver?

ZFS
Page content

In this article I will show you how speed up zfs.

First we need to understand there is two type of zfs the FreeBSD/Solaris based and Linux based cald Zfs On Linux or ZOL.

When a device is replaced, a resilvering operation is initiated to move data from the good copies to the new device. This action is a form of disk scrubbing. RAIDz resilvering is very slow in OpenZFS-based zpools. Basically, it starts with every transaction that’s ever happened in the pool and plays them back one-by-one to the new drive. This is very IO-intensive. Some say if you’re using hard drives larger than 1TB and you are using OpenZFS, use mirror, not RAIDz* or use only SSD for RAIDz*.

In ZOL 0.8.0 they changed to a new resilvering solution. The previous resilvering algorithm repairs blocks from oldest to newest, which can degrade into a lot of small random I/O. The new resilvering algorithm uses a two-step process to sort and resilver blocks in LBA order. If you have an older version of ZOL or want even better performance you can tweak with the zfs configuration.

Speed up resilvering

echo 0 > /sys/module/zfs/parameters/zfs_resilver_delay
echo 0 > /sys/module/zfs/parameters/zfs_scrub_delay
echo 512 > /sys/module/zfs/parameters/zfs_top_maxinflight
echo 8192 > /sys/module/zfs/parameters/zfs_top_maxinflight
echo 8000 > /sys/module/zfs/parameters/zfs_resilver_min_time_ms

Speed up on pool

zpool set autoreplace=on zpool

echo 0 > /sys/module/zfs/parameters/zfs_scan_idle
echo 24 > /sys/module/zfs/parameters/zfs_vdev_scrub_min_active
echo 64 > /sys/module/zfs/parameters/zfs_vdev_scrub_max_active
echo 64 > /sys/module/zfs/parameters/zfs_vdev_sync_write_min_active
echo 32 > /sys/module/zfs/parameters/zfs_vdev_sync_write_max_active
echo 8 > /sys/module/zfs/parameters/zfs_vdev_sync_read_min_active
echo 32 > /sys/module/zfs/parameters/zfs_vdev_sync_read_max_active
echo 8 > /sys/module/zfs/parameters/zfs_vdev_async_read_min_active
echo 32 > /sys/module/zfs/parameters/zfs_vdev_async_read_max_active
echo 8 > /sys/module/zfs/parameters/zfs_vdev_async_write_min_active