I have not voiced my opinion on anything in the past year but I just have to make my opinion known on our transition to OpenSolaris from FreeBSD at Fabulously40. It has been absolutely freaking spectacular!

First off, some admins might want to flog me for using the bleeding edge OpenSolaris Indiana install for a production setup. However, I feel comfortable enough to use it in our production setting. Its not like we are a banking site or something ridiculously mission critical. I am sure the OpenSolaris community could use some early adopter reports from production server use, if there is even any to report at all!

First off why the switch and what went wrong with our old setup

For about 3 years we happily hosted Fabulously40 and other sites including this blog on an initial FreeBSD-5.2 install, which has been upgraded to 6.0,6.1 and 7.0 respectively. I gradually saw performance get better with FreeBSD with each new release. During this time frame it all ran fine. We had little traffic and all was good until we started pick up on more traffic. I would have monthly or bi-weekly kernel panics. I filled out tickets for a techie to run some diagnostic tests on the hardware from where we were renting out the server from. They told me all was okay with the hardware. Enough was enough so we decided to go with our own dedicated hardware.

So we purchased a 2 socket quad core xeon 5355 (8 cores total), 4 gigs of ram , 500×2 gig drive 1u rackmount. The thing is a beast.

Since now we own the hardware, I can go with any OS install that I want. Oh, it feels like being a fat kid in a candy shop. I could try our luck again with FreeBSD but after all these stability issues I am not sure I want to risk it again. So I took a look at my options; Linux (debian), NetBSD or OpenSolaris. I have had extremely good uptime with NetBSD at home on a Soekris router. However, NetBSD does not scale well with many-core systems and does not have Jails. Linux looked like a good candidate but after looking at the software RAID setup, nope not for me, just seemed really arcane and complex and no Jail support. OpenSolaris seems to have everything that I want in a server OS, everything "just works." Mirror raid support works out of the box thanks to ZFS and Solaris takes the Jails concept to a whole new level with Zones.

NetBSD

pros:
Good stability from my experience

cons:
Does not scale well to many cpu cores (hear say)
No Jail support

Linux

pros:
Scales to many cpu cores
apt-get, large package repository

cons:
Lack of Jail support, chroot is primitive.
Complex RAID setup

OpenSolaris:

pros:
Stability (hear say, lets see what happens)
Scales well to many cpu cores
Zones - Jails taken to the next level
ZFS - file system taken to the next level

cons:
pkg, great starting point but small repository.
Using Sun Studio 11 to compile perl and ruby (makes cpan/gem modules require it, note: roll your own ruby/perl/python in a custom prefix instead)
Software packages such as Varnish and PowerDNS need patching to get them to run on Solaris.

So I decided on OpenSolaris. I absolutely love the system. The observability in the OS beats the crap out of anything out there. DTRACE is king, I am already addicted to dtrace one liners. Adding dtrace SDT probes to software such as nginx is brain dead simple. Took me about 30 minutes to figure out how to add dtrace probes into nginx, and I have no C/C++ knowledge! ZFS, this is what we should be expecting from modern filesystems. I am hooked on snapshots, I am already contemplating on using it for postgresql hot-backups. Zones, I love it. I wish I had zoneadm on my FreeBSD boxes at work to control jails. Zone management is a whole lot easier than working with Jails. Furthermore, I have no issues using perl's cpan command utility in a zone unlike within a FreeBSD Jail due to not having access to /dev/tty*.

The one thing I really enjoy about OpenSolaris, it is tweaked out of the box to work optimally on GOOD hardware and not on the lowest common denominator-ware. It seems like every Linux distribution out there caters to the low-end hardware crowd. Sorry, but this is 2008 and we all have hardware well above Pentium 3 class systems.

So to sum it up, if you have modest needs with a single CPU,single hard drive server stick with FreeBSD, Debian or even CentOS. If you need the advance features that Solaris provides then by all means go with it. I am loving the performance and observability of the OS! Next time we need to pick up an extra server I will be taking a look at sun.com and coolthreads servers, they have won a new customer over here thanks to project Indiana.