Browsing the tag open source
MySQL ZFS Snapshot Backups
1 Comment | Filed under administration open source
(4 votes, average: 4.00 out of 5)
Loading ...This is a followup on my previous post concerning how to correctly snapshot databases on ZFS. Snapshotting MySQL any other way will just lead to corrupt database states, essentially making your backups useless.
Here is my script that I use to snapshot our MySQL database. It uses my zBackup.rb script for the automated backup rotation.
mysql -h fab2 -u usr -ppass -e ‘flush tables;flush tables with read lock;’
/usr/bin/ruby /opt/zbackup.rb rpool/mydata 7
mysql -h fab2 -u usr -ppass -e ‘unlock tables;’
Ruby LocaleTranslator
Leave a comment | Filed under main open source programming
(2 votes, average: 3.50 out of 5) Loading ...Today, I am open sourcing my Ruby LocaleTranslator; the translator uses google’s translator API to translate a primary seed locale into various other languages. This eases the creation of multi-lingual sites. Not only can the LocaleTranslator translate your main seed locale into different languages but it can also recursively merge in differences, this comes in handy if you have hand-optimized your translated locales.
Viva Localization!
My projects that use the LocaleTranslator; UploadBooth, PasteBooth and ShrinkBooth.
LocaleTranslator Examples
en.yml
site: hello_world: Hello World! home: Home statement: Localization should be simple!
Batch Conversion of your English locale.
require ‘monkey-patches.rb’
require ‘locale_translator.rb’
en_yml = YAML::load(File.open(‘en.yml’))
[:de,:ru].each do |lang|
lang_yml = LocaleTranslator.translate(en_yml,
:to=>lang,
:html=>true,
:key=>‘GOOGLE API KEY’)
f = File.new("#{lang.to_s.downcase}.yml","w")
f.puts(lang_yml.ya2yaml(:syck_compatible => true))
f.close
p "Translated to #{lang.to_s}"
end
Merge in new locale keys from your English Locale into your already translated Russian locale.
require ‘monkey-patches.rb’
require ‘locale_translator.rb’
en_yml = YAML::load(File.open(‘en.yml’))
ru_yml = YAML::load(File.open(‘ru.yml’))
ru_new_yml = LocaleTranslator.translate(en_yml,
:to=>:ru,
:html=>true,
:merge=>ru_yml,
:key=>‘GOOGLE API KEY’)
puts ru_new_yml.ya2yaml(:syck_compatible => true)
The Implementation Code
Support Monkey Patches
monkey-patches.rb
def to_list
h2l(self)
end
def diff(hash)
hsh = {}
this = self
hash.each do |k,v|
if v.kind_of?Hash and this.key?k
tmp = this[k].diff(v)
hsh[k] = tmp if tmp.size > 0
else
hsh[k] = v unless this.key?k
end
end
hsh
end
def merge_r(hash)
hsh = {}
this = self
hash.each do |k,v|
if v.kind_of?Hash
hsh[k] = this[k].merge_r(v)
else
hsh[k] = v
end
end
self.merge(hsh)
end
private
def h2l(hash)
list = []
hash.each {|k,v| list = (v.kind_of?Hash) ? list.merge_with_dups(h2l(v)) : list << v }
list
end
end
class Array
def chunk(p=2)
return [] if p.zero?
p_size = (length.to_f / p).ceil
[first(p_size), *last(length - p_size).chunk(p - 1)]
end
def to_hash(hash)
l2h(hash,self)
end
def merge(arr)
self | arr
end
def merge_with_dups(arr)
temp = []
self.each {|a| temp << a }
arr.each {|a| temp << a }
temp
end
def merge!(arr)
temp = self.clone
self.clear
temp.each {|a| self << a }
arr.each {|a| self << a unless temp.include?a }
true
end
def merge_with_dups!(arr)
temp = self.clone
self.clear
temp.each {|a| self << a }
arr.each {|a| self << a }
true
end
private
def l2h(hash,lst)
hsh = {}
hash.each {|k,v| hsh[k] = (v.kind_of?Hash) ? l2h(v,lst) : lst.shift }
hsh
end
end
The LocaleTranslator Implementation
You need the ya2yaml and easy_translate gems. Ya2YAML can export locales in UTF-8 unlike the standard yaml implementation that can only export in binary for non-standard ascii.
locale-translator.rb
require ‘rubygems’
require ‘ya2yaml’
require ‘yaml’
require ‘easy_translate’
class LocaleTranslator
def self.translate(text,opts)
opts[:to] = [opts[:to]] if opts[:to] and !opts[:to].kind_of?Array
if opts[:merge].kind_of?Hash and text.kind_of?Hash
diff = opts[:merge].diff(text)
diff_hsh = LocaleTranslator.translate(diff,:to=>opts[:to],:html=>true)
return opts[:merge].merge_r(diff_hsh)
end
if text.kind_of?Hash
t_arr = text.to_list
t_arr = t_arr.first if t_arr.size == 1
tout_arr = LocaleTranslator.translate(t_arr,:to=>opts[:to],:html=>true)
tout_arr = [tout_arr] if tout_arr.kind_of?String
tout_arr.to_hash(text)
elsif text.kind_of?Array
if text.size > 50
out = []
text.chunk.each {|l| out.merge_with_dups!(EasyTranslate.translate(l,opts).first) }
out
else
text = text.first if text.size == 1
EasyTranslate.translate(text,opts).first
end
else
EasyTranslate.translate(text,opts).first
end
end
end
Scala memcached implementation
Leave a comment | Filed under open source programming (4 votes, average: 3.50 out of 5) Loading ...I finally got around to open sourcing our scala memcached implementation that we use at fabulously40 for session storage.
Since wicket sessions can vary greatly in size, using the standard memcached server implementation became impractical due to the slab allocator.
The current code on github lacks the ehcache store and an Actor IoHandler adapter. The internal SMemcached application at fabulously40 uses a private caching API so we can hook up various caching backend storage implementations such as mysql, postgresql, ehcache or even another memcached server. You can grab the TCache project on github that SMemcached uses to unify caching under a single API. This gives SMemcached a lot of flexibility when it comes to caching your data.
fyi. TCache stands for “Tanek” Cache, Tanek means cache in russian.
The project works quite well, but don’t use it in production just yet since there is no data expiration for cached data in the HashMap storage implementation. This is just a technical preview. Do use it in production, this is what we use at Fabulously40 
