Browsing the topic open source
MySQL ZFS Snapshot Backups
1 Comment | Filed under administration open source
(4 votes, average: 4.00 out of 5)
Loading ...This is a followup on my previous post concerning how to correctly snapshot databases on ZFS. Snapshotting MySQL any other way will just lead to corrupt database states, essentially making your backups useless.
Here is my script that I use to snapshot our MySQL database. It uses my zBackup.rb script for the automated backup rotation.
mysql -h fab2 -u usr -ppass -e ‘flush tables;flush tables with read lock;’
/usr/bin/ruby /opt/zbackup.rb rpool/mydata 7
mysql -h fab2 -u usr -ppass -e ‘unlock tables;’
Ruby LocaleTranslator
Leave a comment | Filed under main open source programming
(2 votes, average: 3.50 out of 5) Loading ...Today, I am open sourcing my Ruby LocaleTranslator; the translator uses google’s translator API to translate a primary seed locale into various other languages. This eases the creation of multi-lingual sites. Not only can the LocaleTranslator translate your main seed locale into different languages but it can also recursively merge in differences, this comes in handy if you have hand-optimized your translated locales.
Viva Localization!
My projects that use the LocaleTranslator; UploadBooth, PasteBooth and ShrinkBooth.
LocaleTranslator Examples
en.yml
site: hello_world: Hello World! home: Home statement: Localization should be simple!
Batch Conversion of your English locale.
require ‘monkey-patches.rb’
require ‘locale_translator.rb’
en_yml = YAML::load(File.open(‘en.yml’))
[:de,:ru].each do |lang|
lang_yml = LocaleTranslator.translate(en_yml,
:to=>lang,
:html=>true,
:key=>‘GOOGLE API KEY’)
f = File.new("#{lang.to_s.downcase}.yml","w")
f.puts(lang_yml.ya2yaml(:syck_compatible => true))
f.close
p "Translated to #{lang.to_s}"
end
Merge in new locale keys from your English Locale into your already translated Russian locale.
require ‘monkey-patches.rb’
require ‘locale_translator.rb’
en_yml = YAML::load(File.open(‘en.yml’))
ru_yml = YAML::load(File.open(‘ru.yml’))
ru_new_yml = LocaleTranslator.translate(en_yml,
:to=>:ru,
:html=>true,
:merge=>ru_yml,
:key=>‘GOOGLE API KEY’)
puts ru_new_yml.ya2yaml(:syck_compatible => true)
The Implementation Code
Support Monkey Patches
monkey-patches.rb
def to_list
h2l(self)
end
def diff(hash)
hsh = {}
this = self
hash.each do |k,v|
if v.kind_of?Hash and this.key?k
tmp = this[k].diff(v)
hsh[k] = tmp if tmp.size > 0
else
hsh[k] = v unless this.key?k
end
end
hsh
end
def merge_r(hash)
hsh = {}
this = self
hash.each do |k,v|
if v.kind_of?Hash
hsh[k] = this[k].merge_r(v)
else
hsh[k] = v
end
end
self.merge(hsh)
end
private
def h2l(hash)
list = []
hash.each {|k,v| list = (v.kind_of?Hash) ? list.merge_with_dups(h2l(v)) : list << v }
list
end
end
class Array
def chunk(p=2)
return [] if p.zero?
p_size = (length.to_f / p).ceil
[first(p_size), *last(length - p_size).chunk(p - 1)]
end
def to_hash(hash)
l2h(hash,self)
end
def merge(arr)
self | arr
end
def merge_with_dups(arr)
temp = []
self.each {|a| temp << a }
arr.each {|a| temp << a }
temp
end
def merge!(arr)
temp = self.clone
self.clear
temp.each {|a| self << a }
arr.each {|a| self << a unless temp.include?a }
true
end
def merge_with_dups!(arr)
temp = self.clone
self.clear
temp.each {|a| self << a }
arr.each {|a| self << a }
true
end
private
def l2h(hash,lst)
hsh = {}
hash.each {|k,v| hsh[k] = (v.kind_of?Hash) ? l2h(v,lst) : lst.shift }
hsh
end
end
The LocaleTranslator Implementation
You need the ya2yaml and easy_translate gems. Ya2YAML can export locales in UTF-8 unlike the standard yaml implementation that can only export in binary for non-standard ascii.
locale-translator.rb
require ‘rubygems’
require ‘ya2yaml’
require ‘yaml’
require ‘easy_translate’
class LocaleTranslator
def self.translate(text,opts)
opts[:to] = [opts[:to]] if opts[:to] and !opts[:to].kind_of?Array
if opts[:merge].kind_of?Hash and text.kind_of?Hash
diff = opts[:merge].diff(text)
diff_hsh = LocaleTranslator.translate(diff,:to=>opts[:to],:html=>true)
return opts[:merge].merge_r(diff_hsh)
end
if text.kind_of?Hash
t_arr = text.to_list
t_arr = t_arr.first if t_arr.size == 1
tout_arr = LocaleTranslator.translate(t_arr,:to=>opts[:to],:html=>true)
tout_arr = [tout_arr] if tout_arr.kind_of?String
tout_arr.to_hash(text)
elsif text.kind_of?Array
if text.size > 50
out = []
text.chunk.each {|l| out.merge_with_dups!(EasyTranslate.translate(l,opts).first) }
out
else
text = text.first if text.size == 1
EasyTranslate.translate(text,opts).first
end
else
EasyTranslate.translate(text,opts).first
end
end
end
MacOS X ZFS Samba Transfer Fix
Leave a comment | Filed under administration main open source (2 votes, average: 4.00 out of 5) Loading ...I have a very non-standard storage setup at home. The setup is made up of a 3x500G raidz array on ZFS hosted by OSX. For the longest time I could not get files to copy over samba on ZFS. The files would stream just fine but not copy over, they would abort at the 99% transfer point. Well, I have finally found the fix for it; turn off extended attributes!
smb.conf
vfs objects = notify_kqueue,darwinacl
; The darwin_streams module gives us named streams support.
stream support = no
ea support = no
; Enable locking coherency with AFP.
darwin_streams:brlm = no
As Charles Heston would say, You can have my ZFS when you pry it from my cold dead hands.
viva ZFS on OSX!
CouchDB on ZFS
6 Comments | Filed under administration open source (4 votes, average: 4.75 out of 5) Loading ...CouchDB was made for next generation filesystems such as ZFS and BTRFS. First off, unlike PostgreSQL or MySQL, CouchDB can be snapshot while in production without any flushing or locking trickery since it uses an append only B-Tree storage approach. That alone makes it a compelling database choice on ZFS/BTRFS.
Second, CouchDB works hand-in-hand with ZFS’s block level compression. ZFS can compress blocks of data as they are being written out to the disk. However, it only does it for new blocks and not retroactively. Now, the awesome part, CouchDB on compaction writes out a brand new database file which can utilize the new gzip compression settings on ZFS. This means you can try out different gzip compression settings just by compacting your CouchDB.
Some tips on running CouchDB on ZFS:
1. Use automated snapshots to prevent $admin error, it is painless with ZFS and CouchDB loves being snapshot 
You can give my little ruby script a try for daily snapshots; I use it both on Mac OSX and Solaris for automated ZFS snapshot goodness.
2. Try out various gzip compression schemes on your CouchDB workload, re-compact the database to use the new gzip compression settings. I personally use the gzip-4 compression for our workload which strikes the perfect balance between space and cpu utilization.
3. Set the ZFS dataset to 4k block record size and turn off atime. Yes the B-Tree append only approach is elastic on writes but you can have near perfect tiny writes with a small 4k block record size.
zfs set atime=off rpool/couchdb
PostgreSQL ZFS Snapshot Backups
2 Comments | Filed under administration open source (1 votes, average: 5.00 out of 5) Loading ...I recently had a WD Raptor drive die in a server that hosted our PostgreSQL database. I had a ZFS snapshot strategy setup that sent over ZFS snapshots of the live database to a ZFS mirror for backup purposes. Looked good in theory right? Except, I forgot to do one critical thing, test my backups. Long story short, I had a bunch of snapshots that were useless. Luckily I had offsite nightly PostgreSQL dumps that I did test which were used to seed my development database. So in the end I avoided catastrophic data failure.
With that lesson in mind, I reconfigured our backup system to do it correctly after re-reading the PostgreSQL documentation.
Prerequisite: You must have WAL archiving on and have the archive directory under your database directory. For example if your database is under /rpool/pgdata/db1 configure your archive directory under /rpool/pgdata/db1/archives
Completely optional but I highly suggest you automate your backups; My zbackup ruby script is pretty simple to setup.
This is how my /rpool/pgdata/db1 Looks like:
victori@opensolaris:/# ls /rpool/data/db1 archives pg_clog pg_multixact pg_twophase postmaster.log backup_label pg_hba.conf pg_stat_tmp PG_VERSION postmaster.opts base pg_ident.conf pg_subtrans pg_xlog postmaster.pid global pg_log pg_tblspc postgresql.conf
Source for my pgsnap.sh script.
PGPASSWORD=”mypass” psql -h fab2 postgres -U myuser -c “select pg_start_backup(‘nightly’,true);”
/usr/bin/ruby /opt/zbackup.rb rpool/pgdata 7
PGPASSWORD=”mypass” psql -h fab2 postgres -U myuser -c “select pg_stop_backup();”
rm /rpool/pgdata/db1/archives/*
The process is quite simple. You issue a command to initiate the backup process so PostgreSQL goes into “backup mode.” Second, you do the ZFS snapshot, in this case I am using my zbackup ruby script. Third, you issue another SQL command to PostgrSQL to get out of backup mode. Lastly, since you have the database snapshot you can safely delete your previous WAL archives.
Now, this is all nice and dandy but you should *TEST* your backups, before assuming your backup strategy actually worked.
postgres –singleuser mydb -D /rpool/pgtest/db1
Basically you clone the snapshot and test it by running it under PostgreSQL in single user mode. Once in singleuser mode, test out your backup to make sure it is readable, you can issue a SQL queries to confirm that all is fine with the backup.
ZFS you rock my world
Scala yEnc Decoder
Leave a comment | Filed under open source programming (1 votes, average: 5.00 out of 5) Loading ...SYEnc is a Scala decoder for the yEnc format that is based on Alex Russ’s Java yEnc Decoder. SYEnc was designed to be used as a library by applications needing to use yEnc to decode data. It should be thread-safe, so don’t worry about using it in a threaded context.
Uploaded to github: http://github.com/victori/syenc
Example:
Automated ZFS Snapshots
1 Comment | Filed under administration open source programming (1 votes, average: 5.00 out of 5) Loading ...I needed something like zfs-auto-snapshot written by Tim Foster but portable so it works on all systems that support ZFS. I reviewed a few scripts on github and was unhappy with what was out there so I decided to write my own.
With zbackup.rb you can define what to snapshot and how many rotation days you want to go back.
So say you want a month of snapshots:
Simple, no?
# Create snapshots for a 7 day rotation.
# ./zbackup.rb iraidz/zWork 7
#
# Add to crontab
# crontab -e
# 0 2 * * * /usr/bin/zbackup.rb iraidz/zWork 7
pool = ARGV[0]
days_back = ARGV[1].to_i
if pool.nil? or pool.empty?
puts "\nDefine the pool you want to snapshot:"
puts "\tex: zbackup.rb iraidz/zWork 7\n\n"
exit 0
end
if days_back.nil? or days_back < 1
puts "\nDefine how many days for your rotation:"
puts "\tex: zbackup.rb iraidz/zWork 7\n\n"
exit 0
end
# response from zfs list
curr_snaps = `zfs list -t snapshot -o name`
# days back limit variable
date_back = Time.now - (86400*days_back)
curr_snaps.split(/\n/).each do |pline|
if m = pline.match(/#{pool}\@([0-9]+)\-([0-9]+)\-([0-9]+)/)
if date_back >= Time.local(m[1],m[2],m[3])
`zfs destroy #{pline}`
end
end
end
# take snapshot for this run if needed.
month = Time.now.month
day = Time.now.day
year = Time.now.year
if curr_snaps !~ /#{pool}\@#{year}\-#{month}\-#{day}/
`zfs snapshot -r #{pool}@#{year}-#{month}-#{day}`
end
Clustering Wicket for fun and profit!
2 Comments | Filed under administration main open source programming (11 votes, average: 4.73 out of 5) Loading ...I hate expired sessions, death to all expired sessions. Traditionally a Java servlet container has a fixed session time, a flood of traffic can potentially cause JVM OOM errors if the session time is set too high. I wanted a smart session container that can hold onto sessions for as long as possible and expire sessions only when it is absolutely necessary; A Memcached store would be perfect for this.
There for I recently open sourced the jetty-session-store to solve this problem. With the jetty-session-store you can save your session state to Ehcache, Memcached or the database. State should not be bound to a single JVM, Viva Shared Session Stores!
So now that jetty-session-store is out in the wild you can technically cluster Wicket using just the HttpSessionStore. However, it isn’t very efficient with the way Memcached allocates data in fixed sized cache buckets.
1. Wicket sessions under the HttpSessionStore can get quite large, well over 1Mb in size. A Wicket session not only stores the session state but also the previous serialized pages the user has visited.
2. Serializing and de-serializing a large data structure can get expensive. The HttpSessionStore retains an AccessStackPageMap, which is a list data structure consisting of multiple page map revisions.
So instead of saving one large AccessStackPageMap, I wrote a SecondLevelCacheSessionStore that saves a page map revision per cache entry. This leads to much better cache utilization and a whole lot less serialization on the wire. Not to mention this avoids the whole 1Mb Memcached size limit.
Before you go willy nilly with clustering, read the Wicket render strategies page. Wicket requires session affinity for buffered responses with the default rendering strategy.
Clustering Wicket has never been easier.
Here is an example on how to offload page maps to a hybrid EhCache/Memcached cache. Memcached for long term shared storage while EhCache for short-lived fast cache look ups.
@Override
protected ISessionStore newSessionStore() {
// localhost:11211 — memcached server
// "fabpagestore" — unique appender to avoid key clashes.
// 300 — 5 minute TTL for local ehcache.
return new SecondLevelCacheSessionStore(this,
new CachePageStore(Arrays.asList("localhost:11211"),"fabpagestore",300));
}
}
Here is an example on how to offload page maps to the database.
@Override
protected ISessionStore newSessionStore() {
// "fabpagestore" — unique appender to avoid key clashes.
return new SecondLevelCacheSessionStore(this,new CachePageStore(
new DBCache("jdbc:mysql://foo/mydb", "myname", "mypass", "com.driver.Name", "fabpagestore")));
}
}
Here is my CachePageStore;
import com.base.cache.AsyncMemcache;
import com.base.cache.ICache;
import org.apache.wicket.Page;
import org.apache.wicket.protocol.http.SecondLevelCacheSessionStore.IClusteredPageStore;
import org.apache.wicket.protocol.http.pagestore.AbstractPageStore;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.List;
public class CachePageStore extends AbstractPageStore implements IClusteredPageStore {
private ICache cache;
private Logger logger = LoggerFactory.getLogger(CachePageStore.class);
public CachePageStore(final List<String> servers, final String poolName, final int ttl) {
this(servers, poolName, true, ttl);
}
public CachePageStore(final List<String> servers, final String poolName, boolean async, final int ttl) {
this(new AsyncMemcache(servers, poolName, async, ttl));
}
public CachePageStore(final ICache cache) {
this.cache = cache;
}
// If pageVersion -1 must return highest page version.
protected String getKey(final String sessId, final String pageMapName, final int pageId, final int pageVersion) {
int pageVer = (pageVersion == -1) ? 0 : pageVersion;
if(pageVersion == -1) {
String[] meta = getMeta(sessId, pageMapName, pageId);
pageVer = Integer.valueOf(meta[0]);
}
return sessId + ":" + pageMapName + ":" + pageId + ":" + pageVer;
}
// If pageVersion -1 must return highest page version.
// If ajaxVersion -1 must return highest version.
public String getKey(final String sessId, final String pageMapName, final int pageId, final int pageVersion, final int ajaxVersion) {
// Default it to 0 initially
int ajaxVer = (ajaxVersion == -1) ? 0 : ajaxVersion;
int pageVer = (pageVersion == -1) ? 0 : pageVersion;
if(pageVersion == -1 || ajaxVersion == -1) {
String[] meta = getMeta(sessId, pageMapName, pageId);
if(pageVersion == -1) {
pageVer = Integer.valueOf(meta[0]);
}
if(ajaxVersion == -1) {
ajaxVer = Integer.valueOf(meta[1]);
}
}
return sessId + ":" + pageMapName + ":" + pageId + ":" + pageVer + ":" + ajaxVer;
}
protected String storeKey(final String sessionId, final Page page) {
return sessionId + ":" + page.getPageMapName() + ":" + page.getId() + ":" + page.getCurrentVersionNumber() + ":" + page.getAjaxVersionNumber();
}
protected String getBaseKey(String sessionId, Page page) {
return sessionId + ":" + page.getPageMapName() + ":" + page.getId();
}
protected String getMetaKey(String sessionId, String pageMap, int id) {
return getBaseKey(sessionId,pageMap,id)+"_meta";
}
protected String getMetaKey(String sessionId, Page page) {
return getBaseKey(sessionId,page)+"_meta";
}
protected String getBaseKey(String sessionId, String pageMap, int id) {
if(id == -1) {
return sessionId + ":" + pageMap;
} else {
return sessionId + ":" + pageMap + ":" + id;
}
}
public boolean containsPage(final String sessionId, final String pageMapName, final int pageId, final int pageVersion) {
String key = getKey(sessionId, pageMapName, pageId, pageVersion, -1);
if (logger.isDebugEnabled()) {
logger.debug("CheckExists: " + key);
}
return cache.keyExists(key);
}
public void destroy() {
}
public <T> Page getPage(final String sessionId, final String pagemap, final int id, final int versionNumber, final int ajaxVersionNumber) {
String key = getKey(sessionId, pagemap, id, versionNumber, ajaxVersionNumber);
if (logger.isDebugEnabled()) {
logger.debug("GetPage: " + key);
}
return (Page) cache.get(key);
}
public void pageAccessed(final String sessionId, final Page page) {
}
// If ID == -1 remove the entire pagemap; getBaseKey() takes care of this.
public void removePage(final String sessionId, final String pagemap, final int id) {
String key = getBaseKey(sessionId, pagemap, id);
if (logger.isDebugEnabled()) {
logger.debug("RemovePage: " + key);
}
cache.remove(getMetaKey(sessionId, pagemap, id));
for (String k : cache.getKeys()) {
if (k.startsWith(key)) {
cache.remove(k);
}
}
}
protected String[] getMeta(final String sessionId, String pageMap, int pageId) {
String metaKey = getMetaKey(sessionId,pageMap,pageId);
Object ret = cache.get(metaKey);
if (logger.isDebugEnabled()) {
logger.debug("GetMeta: " + metaKey);
}
if(ret == null) {
return new String[] {"0","0"};
} else {
return String.valueOf(ret).split(":");
}
}
protected void storeMeta(final String sessionId, final Page page) {
String metaKey = getMetaKey(sessionId, page);
Object ret = cache.get(metaKey);
if (logger.isDebugEnabled()) {
logger.debug("StoreMeta: " + metaKey);
}
if(ret == null) {
cache.put(metaKey,page.getCurrentVersionNumber()+":"+page.getAjaxVersionNumber());
} else {
String[] vals = String.valueOf(ret).split(":");
int currPage = Integer.valueOf(vals[0]);
int currAjax = Integer.valueOf(vals[1]);
if(page.getCurrentVersionNumber() > currPage) {
currPage = page.getCurrentVersionNumber();
}
if(page.getAjaxVersionNumber() > currAjax) {
currAjax = page.getAjaxVersionNumber();
}
cache.put(metaKey,currPage+":"+currAjax);
}
}
public void storePage(final String sessionId, final Page page) {
String sKey = storeKey(sessionId, page);
if (logger.isDebugEnabled()) {
logger.debug("StorePage: " + sKey);
}
cache.put(sKey, page);
storeMeta(sessionId,page);
}
public void unbind(final String sessionId) {
if (logger.isDebugEnabled()) {
logger.debug("Unbind: " + sessionId);
}
for (String key : cache.getKeys()) {
if (key.startsWith(sessionId)) {
cache.remove(key);
}
}
}
}
HAProxy 1.4dev6 broken on Solaris
2 Comments | Filed under administration open source programming (1 votes, average: 5.00 out of 5) Loading ...Update: I feel like a jackass now, I thought I was running this against the stable haproxy build, but in reality this was against haproxy-1.4dev6. DOH! Well on the bright-side, I am helping out the author fix a potentially critical bug. Here is the truss and tcp dump if anyone cares.
Well yet another Solaris specific bug/issue to report. HAProxy resets long running connections. Meaning users on slow bandwidth connections are affected by this. I have sent tcpdumps and logs to the author of HAProxy, hopefully this bug/issue would be resolved. I am writing this as a precautionary warning to other Solaris admins out there.
Here the way to trigger this, see if your service is affected by this.
Result:
–2010-01-20 11:19:29– http://somesite.com/onebigfile.txt
Resolving somesite.com (somesite.com)… 72.11.142.91
Connecting to somesite.com (somesite.com)|72.11.142.91|:84… connected.
HTTP request sent, awaiting response… 200 OK
Length: 3806025 (3.6M)
Saving to: “onebigfile.txt”
7% [====> ] 269,008 20.1K/s in 13s
2010-01-20 11:19:42 (20.1 KB/s) – Read error at byte 269008/3806025 (Connection reset by peer). Retrying.
–2010-01-20 11:19:43– (try: 2) http://somesite.com/onebigfile.txt
Connecting to somesite.com (somesite.com)|72.11.142.91|:84… connected.
HTTP request sent, awaiting response… 200 OK
Length: 3806025 (3.6M)
Saving to: “onebigfile.txt”
4% [==> ] 186,016 20.0K/s eta
/Raging, why are there so many Solaris TCP issues? First Varnish? now HAProxy? ARGHHHHH!@#!@
Speedy PostgreSQL Parallel Compression Dumps
2 Comments | Filed under administration main open source (5 votes, average: 5.00 out of 5) Loading ...I used to backup our database using the following statement;
Once our dataset grew into the gigabytes, it took a very long time to do database dumps. Today, I stumbled upon yet another awesome blog post done by Ted Dzibua mentioning two useful parallel compression utilities. So why not try parallel compression with PostgreSQL dumps?
pbzip2 – Parallel BZIP2: Parallel implementation of BZIP2. BZIP2 is well known for being balls slow, so speed it up using multiple CPUs.
pigz – Parallel GZIP: Parallel implementation of GZIP written by Mark Adler.
Time to try this out with our PostgreSQL dump, here are the result times.
• This was done on a quad core xeon 2.66ghz machine.
real 2m7.332s
user 1m16.414s
sys 0m8.233s
# time pg_dump -U secret -h fab2 somedb | pbzip2 -c > somedb.bz2
real 4m14.253s
user 10m35.879s
sys 0m10.904s
The original database was 1.6gigs. The compressed files came out to….
147M somedb.bz2
194M somedb.gz
And just to make this post complete, to pipe the SQL dump back into PostgreSQL
# createdb somedb
# gzip -d -c somedb.gz | psql somedb
