Comments on: ZFS Slow Performance Fix http://letsgetdugg.com/2009/10/21/zfs-slow-performance-fix/ Random tech jargon Mon, 06 Sep 2010 09:29:35 +0000 hourly 1 http://wordpress.org/?v=3.0.1 By: victori http://letsgetdugg.com/2009/10/21/zfs-slow-performance-fix/comment-page-1/#comment-208 victori Mon, 08 Feb 2010 22:05:20 +0000 http://letsgetdugg.com/?p=377#comment-208 I completely forgot about this post. I actually figured out the root cause of our performance problem. The issue was with a high TLER value defined on caviar black hard disks. See the link below for more information on TLER. http://en.wikipedia.org/wiki/Time-Limited_Error_Recovery Point of story is, don't go cheap when setting up a RAID system. Buy only RAID edition drives, they are tuned for RAID setups. Live and learn I guess. I completely forgot about this post. I actually figured out the root cause of our performance problem. The issue was with a high TLER value defined on caviar black hard disks. See the link below for more information on TLER.

http://en.wikipedia.org/wiki/Time-Limited_Error_Recovery

Point of story is, don’t go cheap when setting up a RAID system. Buy only RAID edition drives, they are tuned for RAID setups. Live and learn I guess.

]]> By: Brian Horn http://letsgetdugg.com/2009/10/21/zfs-slow-performance-fix/comment-page-1/#comment-206 Brian Horn Sun, 07 Feb 2010 17:57:31 +0000 http://letsgetdugg.com/?p=377#comment-206 Actually for SATA disks the ZFS folks required that only a single I/O be sent down to the driver at a time as they say a fall off in concurrent sequential I/O performance. The problem is likely is in an assumption in ZFS about the order of completion of the I/Os by the disk be at least roughly in the order that ZFS submitted them. This is not true with either SATA NCQ or SCSI/SAS simple tagged queueing. Nikolay was correct in pointing out that that something else is going on. Look at the size of the active queue ("actv"). It is 0, not 32. Actually for SATA disks the ZFS folks required that
only a single I/O be sent down to the driver at a time
as they say a fall off in concurrent sequential I/O
performance. The problem is likely is in an assumption
in ZFS about the order of completion of the I/Os by
the disk be at least roughly in the order that ZFS submitted
them. This is not true with either SATA NCQ or SCSI/SAS
simple tagged queueing. Nikolay was correct in pointing
out that that something else is going on. Look at the size
of the active queue (“actv”). It is 0, not 32.

]]> By: Nikolay Botev http://letsgetdugg.com/2009/10/21/zfs-slow-performance-fix/comment-page-1/#comment-152 Nikolay Botev Mon, 30 Nov 2009 04:45:19 +0000 http://letsgetdugg.com/?p=377#comment-152 Hey thanks for the info. I had the same problem as you: while writing to zpool reads would become impossible. I spent a whole week, going from zpool v6 running on FreeBSD 7.2, to OpenSolaris r127, upgraded pool to v13, added dedicated external log device, changed vdev_max_pending to 1, all to no avail... although, vdev_max_pending=1 did help a lot. In the end, however, I discovered the problem was not in vdev_max_pending (which on osol 127 was set to 10 by default for me, not 32, so they have probably fixed that now in the kernel to use 10 by default for SATA disks?). The real culprit was a bad hard disk - I discovered using iostat -xn 1 that zfs freezes when it gets stuck to writing to only one of the drives in the pool, which would show as 100% blocking in iostat => one of the drives was a lot slower, even though it never reported any errors, and scrubbing the pool actually went through fine (although slower too). I noticed that in your example also it is just one drive that is blocking at 100%, so I thought I'd let you know, just in case this might be the problem in your case too. Hey thanks for the info.

I had the same problem as you: while writing to zpool reads would become impossible.

I spent a whole week, going from zpool v6 running on FreeBSD 7.2, to OpenSolaris r127, upgraded pool to v13, added dedicated external log device, changed vdev_max_pending to 1, all to no avail… although, vdev_max_pending=1 did help a lot.

In the end, however, I discovered the problem was not in vdev_max_pending (which on osol 127 was set to 10 by default for me, not 32, so they have probably fixed that now in the kernel to use 10 by default for SATA disks?).

The real culprit was a bad hard disk – I discovered using iostat -xn 1 that zfs freezes when it gets stuck to writing to only one of the drives in the pool, which would show as 100% blocking in iostat => one of the drives was a lot slower, even though it never reported any errors, and scrubbing the pool actually went through fine (although slower too).

I noticed that in your example also it is just one drive that is blocking at 100%, so I thought I’d let you know, just in case this might be the problem in your case too.

]]>