ZFS Slow Performance Fix
Update: The following information could be beneficial to some, however my issues actually were with Caviar black drives shipping with TLER disabled. You need to pay Western Digital a premium for their “RAID” drives with TLER enabled. So for anyone reading this, avoid consumer Western Digital drives if you plan on using them for RAID.
zfs_vdev_max_pending
I can’t believe how long I have been tolerating horrible concurrent IO performance on OpenSolaris running ZFS. When I have any IO intensive writes happening the whole system slows down to a crawl for any further IO. Running “ls” on a uncached directory is just painful.
victori@opensolaris:/opt# iostat -xnz 1
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 87.0 0.0 2878.1 0.0 0.0 0.0 0.4 0 100 c4t0d0
0.0 83.0 0.0 2878.1 0.0 0.1 0.2 0.7 1 50 c4t1d0
1.0 0.0 28.0 0.0 0.0 0.0 0.0 5.4 0 1 c4t2d0
Notice c4t0d0 is blocking at 100%. If a disk is blocking at 100% good luck getting the disk to do any other operations such as reads.
SATA disks do Native Command Queuing while SAS disks do Tagged Command Queuing, this is an important distinction. Seems like OpenSolaris/Solaris is optimized for the latter with a 32 wide command queue set by default. This completely saturates the SATA disks with IO commands in turn making the system unusable for short periods of time.
Dynamically set the ZFS command queue to 1 to optimize for NCQ.
echo zfs_vdev_max_pending/W0t1 | mdb -kw
And add to /etc/system
set zfs:zfs_vdev_max_pending=1
Enjoy your OpenSolaris server on cheap SATA disks!
Hey thanks for the info.
I had the same problem as you: while writing to zpool reads would become impossible.
I spent a whole week, going from zpool v6 running on FreeBSD 7.2, to OpenSolaris r127, upgraded pool to v13, added dedicated external log device, changed vdev_max_pending to 1, all to no avail… although, vdev_max_pending=1 did help a lot.
In the end, however, I discovered the problem was not in vdev_max_pending (which on osol 127 was set to 10 by default for me, not 32, so they have probably fixed that now in the kernel to use 10 by default for SATA disks?).
The real culprit was a bad hard disk – I discovered using iostat -xn 1 that zfs freezes when it gets stuck to writing to only one of the drives in the pool, which would show as 100% blocking in iostat => one of the drives was a lot slower, even though it never reported any errors, and scrubbing the pool actually went through fine (although slower too).
I noticed that in your example also it is just one drive that is blocking at 100%, so I thought I’d let you know, just in case this might be the problem in your case too.
Actually for SATA disks the ZFS folks required that
only a single I/O be sent down to the driver at a time
as they say a fall off in concurrent sequential I/O
performance. The problem is likely is in an assumption
in ZFS about the order of completion of the I/Os by
the disk be at least roughly in the order that ZFS submitted
them. This is not true with either SATA NCQ or SCSI/SAS
simple tagged queueing. Nikolay was correct in pointing
out that that something else is going on. Look at the size
of the active queue (“actv”). It is 0, not 32.
I completely forgot about this post. I actually figured out the root cause of our performance problem. The issue was with a high TLER value defined on caviar black hard disks. See the link below for more information on TLER.
http://en.wikipedia.org/wiki/Time-Limited_Error_Recovery
Point of story is, don’t go cheap when setting up a RAID system. Buy only RAID edition drives, they are tuned for RAID setups. Live and learn I guess.
thank you for this help, it seems it was helpful, so far … 🙂
Actually, it’s only WD that charges you extra for the ability to set “TLER”. The command is a part of the SATA specification. Drives from other vendors support it (at least I have yet to see an exception; I’ve tried it on recent Seagate, Samsung and Hitachi drives so far).
AFAIK the *only* difference between “RAID Edition” and desktop edition WD drives is that the latter come with an intentionally crippled SATA command set.
The “%b” in iostat output is actually %busy and not %blocking?
My brother suggested I would possibly like this website.
He was once totally right. This publish truly made my day. You can not imagine simply how much time I had spent for this information!
Thank you!
Hey there I am so excited I found your blog, I really found you by error,
while I was browsing on Google for something else, Anyhow I
am here now and would just like to say kudos for
a remarkable post and a all round thrilling blog (I also love the theme/design), I don’t have time to go through it all at the minute but I have book-marked it and also included your RSS feeds, so when I have time I will be back to read a great deal more, Please do keep up the great job.
Thanks for one’s marvelous posting! I really enjoyed reading it, you will be a great author. I will ensure that I bookmark your blog and will often come back someday. I want to encourage continue your great work, have a nice day!
Have you ever thought about including a little bit more than just your articles?
I mean, what you say is valuable and all. Nevertheless
think about if you added some great graphics or video clips to give your posts
more, “pop”! Your content is excellent but with images and
videos, this website could certainly be one of the very best in its
niche. Excellent blog!
Hi, I do think this is a great blog. I stumbledupon it ;
) I am going to revisit yet again since i have book marked it.
Money and freedom is the best way to change, may
you be rich and continue to help other people.
michael kors bags
Good respond in return of this question with firm arguments and
explaining the whole thing regarding that.
When someone writes an piece of writing he/she keeps the image of a user in his/her mind that
how a user can be aware of it. Thus that’s why this
piece of writing is great. Thanks!
Sweet blog! I found it while searching on Yahoo News. Do you have any tips on how to get listed in Yahoo News?
I’ve been trying for a while but I never seem to get
there! Many thanks
Thanks for your marvelous posting! I quite enjoyed reading it, you will be a great author.I will be sure to bookmark your
blog and may come back in the future. I want to encourage continue
your great writing, have a nice afternoon!
This design is wicked! You definitely know how to keep
a reader entertained. Between your wit and your videos, I was almost moved to start
my own blog (well, almost…HaHa!) Great
job. I really enjoyed what you had to say, and more
than that, how you presented it. Too cool!
I’m really impressed with your writing skills as well as with the layout on your weblog.
Is this a paid theme or did you customize it yourself?
Either way keep up the excellent quality writing, it’s rare to see a great
blog like this one these days.
Howdy! I could have sworn I’ve been to this site before but
after browsing through some of the post I realized
it’s new to me. Nonetheless, I’m definitely happy I found
it and I’ll be book-marking and checking back frequently!
Hello to all, the contents present at this web site are genuinely remarkable for people knowledge, well, keep
up the nice work fellows.
Hi there this is kind of of off topic but I was wondering if blogs use WYSIWYG editors or
if you have to manually code with HTML. I’m starting a blog soon but have no coding skills so
I wanted to get advice from someone with experience.
Any help would be enormously appreciated!
What’s up to all, how is all, I think every one is getting more from this website,
and your views are pleasant designed for new
visitors.
I’m curious to find out what blog system you have been using?
I’m experiencing some minor security problems with my latest blog and I’d like to find something more safeguarded.
Do you have any recommendations?
I need to to thank you for this good read!! I certainly loved every little bit of it. I have you bookmarked to look at new stuff you post…
Hello there I am so thrilled I found your web site, I really
found you by mistake, while I was researching on Askjeeve for something else, Anyhow I am here now and would
just like to say kudos for a marvelous post and a all round
enjoyable blog (I also love the theme/design), I don’t have time to look over it all at the moment but I have bookmarked it and
also added in your RSS feeds, so when I have time I will be back to read
more, Please do keep up the excellent jo.
Simply wish to say your article is as astonishing. The
clarity in your post is just spectacular and i
could assume you’re an expert on this subject. Fine with your permission allow me to grab your RSS feed to keep up to date with forthcoming post.
Thanks a million and please carry on the gratifying work.
I loved as much as you’ll receive carried out right here.
The sketch is tasteful, your authored subject matter stylish.
nonetheless, you command get got an shakiness over that you wish be delivering
the following. unwell unquestionably come further formerly again as exactly the same nearly
a lot often inside case you shield this hike.
It’s hard to find your page in google. I found it
on 21 spot, you should build quality backlinks , it will help you to get more
visitors. I know how to help you, just type in google – k2 seo tricks
It’s really a nice and useful piece of information. I’m glad that you shared this helpful info with us.
Please keep us up to date like this. Thanks for sharing.
I’m extremely impressed with your writing skills as well as with the layout on your blog.
Is this a paid theme or did you modify it yourself? Either
way keep up the excellent quality writing, it’s rare to see a great blog like this one today.
Excellent post however I was wondering if you could write
a litte more on this topic? I’d be very thankful if you could elaborate a little bit more.
Cheers!
This article is truly a pleasant one it helps new internet users, who
are wishing in favor of blogging.
This post is worth everyone’s attention. When can I find
out more?
Hello I am so happy I found your blog, I really found you by
mistake, while I was looking on Askjeeve for something
else, Anyways I am here now and would just like to say
thanks a lot for a marvelous post and a all round interesting blog (I also love the theme/design), I don’t have time to browse it all at
the minute but I have book-marked it and also included your RSS feeds, so when I have time I will be back to read a great deal more, Please do
keep up the excellent jo.
Hmm it appears like your website ate my first comment (it was
super long) so I guess I’ll just sum it up what I wrote and say, I’m thoroughly enjoying your blog.
I as well am an aspiring blog writer but I’m still
new to the whole thing. Do you have any tips and hints for beginner
blog writers? I’d genuinely appreciate it.
I know this website provides quality dependent articles
or reviews and extra material, is there any other site which gives such things in quality?
What’s up colleagues, its fantastic article on the topic of teachingand fully explained, keep it
up all the time.
Wow that was unusual. I just wrote an extremely long
comment but after I clicked submit my comment didn’t show up.
Grrrr… well I’m not writing all that over again. Anyhow, just wanted to say great blog!
My spouse and I stumbled over here by a different website and thought
I might check things out. I like what I see so now i’m following you.
Look forward to exploring your web page yet again.
Quality content is the important to attract the people to pay a quick visit the site, that’s what this web site ( Maryellen – http://ge.tt/82FRmEr1/v/0 ) is providing.
I think this is among the such a lot vital information for me.
And i am satisfied studying your article. However want to commentary on few common issues, The website taste is ideal, the articles is truly excellent : D.
Good process, cheers
Very rapidly this web site will be famous among all blogging
viewers, due to it’s nice posts
Can I just say what a relief to discover an individual who genuinely knows
what they’re talking about online. You actually know
how to bring a problem to light and make it important. A lot more people need to
read this and understand this side of the story. I was surprised that you’re not more popular since
you surely possess the gift.
Highly descriptive article, I liked that bit.
Will there be a part 2?
Magnificent goods from you, man. I have understand your stuff previous to and you’re just too fantastic.
I really like what you’ve acquired here, really like what you’re stating and the way in which you say it.
You make it enjoyable and you still care for to keep it sensible.
I can’t wait to read far more from you. This is actually a wonderful website.
Just wish to say your article is as surprising.
The clarity for your post is simply cool and that i
could assume you are a professional in this subject.
Well together with your permission let me to
seize your RSS feed to keep up to date with impending post.
Thanks a million and please carry on the rewarding
work.
You actually make it seem so easy with your presentation but I find this matter to be actually something which
I think I would never understand. It seems too complex and extremely broad for
me. I am looking forward for your next post, I’ll try to get
the hang of it!
If you are going for best contents like me, just go to see this site every day for
the reason that it provides feature contents,
thanks
Hello there, just became aware of your blog through Google, and found that
it is really informative. I am going to watch out for brussels.
I will appreciate if you continue this in future. Lots of people will be benefited from your writing.
Cheers!
I read a lot of interesting content here. Probably you spend a lot of time
writing, i know how to save you a lot of time, there
is an online tool that creates unique, google friendly posts in seconds, just type in google – laranitas free content source
Thankfulness to my father who stated to me about this website, this webpage is
genuinely awesome.