Here is my script that I use to snapshot our MySQL database. It uses my zBackup.rb script for the automated backup rotation.

smb.conf

As Charles Heston would say, You can have my ZFS when you pry it from my cold dead hands.

viva ZFS on OSX!

Second, CouchDB works hand-in-hand with ZFS’s block level compression. ZFS can compress blocks of data as they are being written out to the disk. However, it only does it for new blocks and not retroactively. Now, the awesome part, CouchDB on compaction writes out a brand new database file which can utilize the new gzip compression settings on ZFS. This means you can try out different gzip compression settings just by compacting your CouchDB.

Some tips on running CouchDB on ZFS:

1. Use automated snapshots to prevent $admin error, it is painless with ZFS and CouchDB loves being snapshot

You can give my little ruby script a try for daily snapshots; I use it both on Mac OSX and Solaris for automated ZFS snapshot goodness.

2. Try out various gzip compression schemes on your CouchDB workload, re-compact the database to use the new gzip compression settings. I personally use the gzip-4 compression for our workload which strikes the perfect balance between space and cpu utilization.

3. Set the ZFS dataset to 4k block record size and turn off atime. Yes the B-Tree append only approach is elastic on writes but you can have near perfect tiny writes with a small 4k block record size.

With that lesson in mind, I reconfigured our backup system to do it correctly after re-reading the PostgreSQL documentation.

Prerequisite: You must have WAL archiving on and have the archive directory under your database directory. For example if your database is under /rpool/pgdata/db1 configure your archive directory under /rpool/pgdata/db1/archives

Completely optional but I highly suggest you automate your backups; My zbackup ruby script is pretty simple to setup.

This is how my /rpool/pgdata/db1 Looks like:

victori@opensolaris:/# ls /rpool/data/db1
archives      pg_clog	     pg_multixact  pg_twophase	    postmaster.log
backup_label  pg_hba.conf    pg_stat_tmp   PG_VERSION	    postmaster.opts
base	      pg_ident.conf  pg_subtrans   pg_xlog	    postmaster.pid
global	      pg_log	     pg_tblspc	   postgresql.conf

Source for my pgsnap.sh script.

The process is quite simple. You issue a command to initiate the backup process so PostgreSQL goes into “backup mode.” Second, you do the ZFS snapshot, in this case I am using my zbackup ruby script. Third, you issue another SQL command to PostgrSQL to get out of backup mode. Lastly, since you have the database snapshot you can safely delete your previous WAL archives.

Now, this is all nice and dandy but you should *TEST* your backups, before assuming your backup strategy actually worked.

Basically you clone the snapshot and test it by running it under PostgreSQL in single user mode. Once in singleuser mode, test out your backup to make sure it is readable, you can issue a SQL queries to confirm that all is fine with the backup.

ZFS you rock my world

With zbackup.rb you can define what to snapshot and how many rotation days you want to go back.

So say you want a month of snapshots:

Simple, no?

There for I recently open sourced the jetty-session-store to solve this problem. With the jetty-session-store you can save your session state to Ehcache, Memcached or the database. State should not be bound to a single JVM, Viva Shared Session Stores!

So now that jetty-session-store is out in the wild you can technically cluster Wicket using just the HttpSessionStore. However, it isn’t very efficient with the way Memcached allocates data in fixed sized cache buckets.

1. Wicket sessions under the HttpSessionStore can get quite large, well over 1Mb in size. A Wicket session not only stores the session state but also the previous serialized pages the user has visited.

2. Serializing and de-serializing a large data structure can get expensive. The HttpSessionStore retains an AccessStackPageMap, which is a list data structure consisting of multiple page map revisions.

So instead of saving one large AccessStackPageMap, I wrote a SecondLevelCacheSessionStore that saves a page map revision per cache entry. This leads to much better cache utilization and a whole lot less serialization on the wire. Not to mention this avoids the whole 1Mb Memcached size limit.

Before you go willy nilly with clustering, read the Wicket render strategies page. Wicket requires session affinity for buffered responses with the default rendering strategy.

Clustering Wicket has never been easier.

Here is an example on how to offload page maps to a hybrid EhCache/Memcached cache. Memcached for long term shared storage while EhCache for short-lived fast cache look ups.

Here is an example on how to offload page maps to the database.

Here is my CachePageStore;

Well yet another Solaris specific bug/issue to report. HAProxy resets long running connections. Meaning users on slow bandwidth connections are affected by this. I have sent tcpdumps and logs to the author of HAProxy, hopefully this bug/issue would be resolved. I am writing this as a precautionary warning to other Solaris admins out there.

Here the way to trigger this, see if your service is affected by this.

Result:

/Raging, why are there so many Solaris TCP issues? First Varnish? now HAProxy? ARGHHHHH!@#!@

Once our dataset grew into the gigabytes, it took a very long time to do database dumps. Today, I stumbled upon yet another awesome blog post done by Ted Dzibua mentioning two useful parallel compression utilities. So why not try parallel compression with PostgreSQL dumps?

pbzip2 – Parallel BZIP2: Parallel implementation of BZIP2. BZIP2 is well known for being balls slow, so speed it up using multiple CPUs.

pigz – Parallel GZIP: Parallel implementation of GZIP written by Mark Adler.

Time to try this out with our PostgreSQL dump, here are the result times.

• This was done on a quad core xeon 2.66ghz machine.

The original database was 1.6gigs. The compressed files came out to….

And just to make this post complete, to pipe the SQL dump back into PostgreSQL

The motivation behind HTTP health checks was because once a month or so at Fabulously40 our ActiveMQ would break down while still accepting connections, the only way to figure out if it was zombified was to check the HTTP administrator interface. If the ActiveMQ instance was actually knelled over, the administrator interface would come back with a HTTP/500 response code, hence the birth of HTTP health checks.

Here is our Satan configuration file that we use at Fabulously40.

The “args” property might be a bit confusing, it is a snippet of text that Satan looks for in the arguments passed to your application to identify the running process. So for example, if you start your ActiveMQ instance with the following arguments; “java -jar activemq.jar -Dactivemq=8161 -XXXXX” Placing “8161″ in args property would be a good unique identifier for Satan to pick up on.

Satan.watch do |s|
  s.name = "jvm instances"                # name of job
  s.user = "webservd"                     # under what user
  s.group = "webservd"                    # under what group
  s.deamon = "java"                       # deamon binary name to grep for
  s.args = nil                            # globally look for specific arguments, optional
  s.debug = true                          # if to write out debug information
  s.safe_mode = false                     # If in safe mode, satan will not kill ;-(
  s.interval = 10.seconds                 # interval to run at to collect statistics
  s.sleep_after_kill = 1.minute           # sleep after killing, satan is tired!
  s.contact = "victori@fabulously40.com"  # admin contact, optional if you want email alerts

  s.kill_if do |process|
    process.condition(:cpu) do |cpu|      # on cpu condition
      cpu.name  = "50% CPU limit"         # name for job
      cpu.args  = "jetty"                 # make sure this is a jetty process, optional
      cpu.above = 48.percent              # if above certain percentage
      cpu.times = 5                       # how many times we can hit this condition before killing
    end

    process.condition(:memory) do |memory|  # on memory condition
      memory.name  = "850MB limit"          # name for job
      memory.args  = "jetty"                # make sure this is a jetty process, optional
      memory.above = 850.megabytes          # limit for memory use
      memory.times = 5                      # how many times we can hit this condition before killing
    end

    # ActiveMQ tends to die on us under heavy load so we need the power of satan!
    process.condition(:http) do |http|                        # on http condition
      http.name   = "HTTP ActiveMQ Check"                     # name for job
      http.args   = "8161"                                    # look for specific app arguments
                                                              # to associate app to URI
      http.uri    = "http://localhost:8161/admin/queues.jsp"  # the URI
      http.times  = 5                                         # how many times before kill
    end
  end
end

What I hate is fording endless rivers of horseshit that are in the way
of seemingly simple tasks. And I hate it even more when I have to explain
to a non-programmer what I am doing, "building LXML against a different
version of libiconv because I think it might be the source of a crash". 

"But all I asked you to do was parse some documents."

Good times.