Although using RAID5/6 in BTRFS is still experimental at the moment, it can be used when certain things are taken into consideration. Most important is using RAID1c3 or RAID1c4 for metadata! But another post on that!

For now, I’m going to give some tips for speeding up the scrub speed of RAID5/6 in BTRFS. As many know the scrub speeds for the various RAID1 setups are all near full disk speed, but for RAID5/6 that’s different. BTRFS spawns a scrub thread per disk, with the different RAID1’s that’s no problem as no read from other disks is needed while scrubbing / checking the data blocks on a given disk. For RAID5/6 that’s different, the data block is striped together with one or two parity parts (depending on RAID5 or RAID6) over the different disks. If one block gets checked, the data and parity parts from all disks needs to be read and checked. The scrub utility isn’t aware of this and still spawns a thread per disk, thus we will get a lot of random IO, as each thread tries to check a block on a disk and needs to read the parts from the other disks. To mitigate the random IO a bit I’ve experimented with the IO schedulers available in Linux (from https://wiki.ubuntu.com/Kernel/Reference/IOSchedulers):

• – none, as you might expect, no re-ordering of IO…
• – mq-deadline, the default one
• – bfq, good for responsiveness, specially when running a desktop system
• – kyber, simple multiqueue scheduler with only two queues

In the end I got the best results with kyber. Furthermore I used the IO Nice options of scrub to make optimal use of the scheduler.

First check with lsscsi which disks needs to be set. I only set the disks in the RAID5/6 to the kyber scheduler. Setting the scheduler of the disks, replace the letters with the correct disk letters:

for i in {a,b,c,...}; do echo "kyber" > /sys/block/sd$i/queue/scheduler; done

After the scheduler on the disks is set the scrub command can be invoked:

btrfs scrub start -c 1 -n 0 <mountpoint>

The -c sets the IO class:

• 1: real time (highest priority)
• 2: best effort
• 3: idle (lowest priority)

The -n sets the level, if applicable. The level can be set ranging from 0 to 7, with 0 the highest priority and 7 the lowest priority.

With these settings I get about 50MB/s scrub speed on a BTRFS RAID6 with 6x 16TB TOSHIBA MG08ACA1 disks. And yes, the scrub still takes a long time, but at least at a bit better speed!