Browsing the topic programming
Ruby LocaleTranslator
Leave a comment | Filed under main open source programming
(1 votes, average: 5.00 out of 5)
Loading ...Today, I am open sourcing my Ruby LocaleTranslator; the translator uses google’s translator API to translate a primary seed locale into various other languages. This eases the creation of multi-lingual sites. Not only can the LocaleTranslator translate your main seed locale into different languages but it can also recursively merge in differences, this comes in handy if you have hand-optimized your translated locales.
Viva Localization!
My projects that use the LocaleTranslator; UploadBooth, PasteBooth and ShrinkBooth.
LocaleTranslator Examples
en.yml
site: hello_world: Hello World! home: Home statement: Localization should be simple!
Batch Conversion of your English locale.
require ‘monkey-patches.rb’
require ‘locale_translator.rb’
en_yml = YAML::load(File.open(‘en.yml’))
[:de,:ru].each do |lang|
lang_yml = LocaleTranslator.translate(en_yml,
:to=>lang,
:html=>true,
:key=>‘GOOGLE API KEY’)
f = File.new("#{lang.to_s.downcase}.yml","w")
f.puts(lang_yml.ya2yaml(:syck_compatible => true))
f.close
p "Translated to #{lang.to_s}"
end
Merge in new locale keys from your English Locale into your already translated Russian locale.
require ‘monkey-patches.rb’
require ‘locale_translator.rb’
en_yml = YAML::load(File.open(‘en.yml’))
ru_yml = YAML::load(File.open(‘ru.yml’))
ru_new_yml = LocaleTranslator.translate(en_yml,
:to=>:ru,
:html=>true,
:merge=>ru_yml,
:key=>‘GOOGLE API KEY’)
puts ru_new_yml.ya2yaml(:syck_compatible => true)
The Implementation Code
Support Monkey Patches
monkey-patches.rb
def to_list
h2l(self)
end
def diff(hash)
hsh = {}
this = self
hash.each do |k,v|
if v.kind_of?Hash and this.key?k
tmp = this[k].diff(v)
hsh[k] = tmp if tmp.size > 0
else
hsh[k] = v unless this.key?k
end
end
hsh
end
def merge_r(hash)
hsh = {}
this = self
hash.each do |k,v|
if v.kind_of?Hash
hsh[k] = this[k].merge_r(v)
else
hsh[k] = v
end
end
self.merge(hsh)
end
private
def h2l(hash)
list = []
hash.each {|k,v| list = (v.kind_of?Hash) ? list.merge_with_dups(h2l(v)) : list << v }
list
end
end
class Array
def chunk(p=2)
return [] if p.zero?
p_size = (length.to_f / p).ceil
[first(p_size), *last(length - p_size).chunk(p - 1)]
end
def to_hash(hash)
l2h(hash,self)
end
def merge(arr)
self | arr
end
def merge_with_dups(arr)
temp = []
self.each {|a| temp << a }
arr.each {|a| temp << a }
temp
end
def merge!(arr)
temp = self.clone
self.clear
temp.each {|a| self << a }
arr.each {|a| self << a unless temp.include?a }
true
end
def merge_with_dups!(arr)
temp = self.clone
self.clear
temp.each {|a| self << a }
arr.each {|a| self << a }
true
end
private
def l2h(hash,lst)
hsh = {}
hash.each {|k,v| hsh[k] = (v.kind_of?Hash) ? l2h(v,lst) : lst.shift }
hsh
end
end
The LocaleTranslator Implementation
You need the ya2yaml and easy_translate gems. Ya2YAML can export locales in UTF-8 unlike the standard yaml implementation that can only export in binary for non-standard ascii.
locale-translator.rb
require ‘rubygems’
require ‘ya2yaml’
require ‘yaml’
require ‘easy_translate’
class LocaleTranslator
def self.translate(text,opts)
opts[:to] = [opts[:to]] if opts[:to] and !opts[:to].kind_of?Array
if opts[:merge].kind_of?Hash and text.kind_of?Hash
diff = opts[:merge].diff(text)
diff_hsh = LocaleTranslator.translate(diff,:to=>opts[:to],:html=>true)
return opts[:merge].merge_r(diff_hsh)
end
if text.kind_of?Hash
t_arr = text.to_list
t_arr = t_arr.first if t_arr.size == 1
tout_arr = LocaleTranslator.translate(t_arr,:to=>opts[:to],:html=>true)
tout_arr = [tout_arr] if tout_arr.kind_of?String
tout_arr.to_hash(text)
elsif text.kind_of?Array
if text.size > 50
out = []
text.chunk.each {|l| out.merge_with_dups!(EasyTranslate.translate(l,opts).first) }
out
else
text = text.first if text.size == 1
EasyTranslate.translate(text,opts).first
end
else
EasyTranslate.translate(text,opts).first
end
end
end
Scala yEnc Decoder
Leave a comment | Filed under open source programming
(No Ratings Yet) Loading ...SYEnc is a Scala decoder for the yEnc format that is based on Alex Russ’s Java yEnc Decoder. SYEnc was designed to be used as a library by applications needing to use yEnc to decode data. It should be thread-safe, so don’t worry about using it in a threaded context.
Uploaded to github: http://github.com/victori/syenc
Example:
Automated ZFS Snapshots
1 Comment | Filed under administration open source programming (1 votes, average: 5.00 out of 5) Loading ...I needed something like zfs-auto-snapshot written by Tim Foster but portable so it works on all systems that support ZFS. I reviewed a few scripts on github and was unhappy with what was out there so I decided to write my own.
With zbackup.rb you can define what to snapshot and how many rotation days you want to go back.
So say you want a month of snapshots:
Simple, no? 
# Create snapshots for a 7 day rotation.
# ./zbackup.rb iraidz/zWork 7
#
# Add to crontab
# crontab -e
# 0 2 * * * /usr/bin/zbackup.rb iraidz/zWork 7
pool = ARGV[0]
days_back = ARGV[1].to_i
if pool.nil? or pool.empty?
puts "\nDefine the pool you want to snapshot:"
puts "\tex: zbackup.rb iraidz/zWork 7\n\n"
exit 0
end
if days_back.nil? or days_back < 1
puts "\nDefine how many days for your rotation:"
puts "\tex: zbackup.rb iraidz/zWork 7\n\n"
exit 0
end
# response from zfs list
curr_snaps = `zfs list -t snapshot -o name`
# days back limit variable
date_back = Time.now - (86400*days_back)
curr_snaps.split(/\n/).each do |pline|
if m = pline.match(/#{pool}\@([0-9]+)\-([0-9]+)\-([0-9]+)/)
if date_back >= Time.local(m[1],m[2],m[3])
`zfs destroy #{pline}`
end
end
end
# take snapshot for this run if needed.
month = Time.now.month
day = Time.now.day
year = Time.now.year
if curr_snaps !~ /#{pool}\@#{year}\-#{month}\-#{day}/
`zfs snapshot -r #{pool}@#{year}-#{month}-#{day}`
end
Clustering Wicket for fun and profit!
Leave a comment | Filed under administration main open source programming
(7 votes, average: 4.57 out of 5) Loading ...I hate expired sessions, death to all expired sessions. Traditionally a Java servlet container has a fixed session time, a flood of traffic can potentially cause JVM OOM errors if the session time is set too high. I wanted a smart session container that can hold onto sessions for as long as possible and expire sessions only when it is absolutely necessary; A Memcached store would be perfect for this.
There for I recently open sourced the jetty-session-store to solve this problem. With the jetty-session-store you can save your session state to Ehcache, Memcached or the database. State should not be bound to a single JVM, Viva Shared Session Stores!
So now that jetty-session-store is out in the wild you can technically cluster Wicket using just the HttpSessionStore. However, it isn’t very efficient with the way Memcached allocates data in fixed sized cache buckets.
1. Wicket sessions under the HttpSessionStore can get quite large, well over 1Mb in size. A Wicket session not only stores the session state but also the previous serialized pages the user has visited.
2. Serializing and de-serializing a large data structure can get expensive. The HttpSessionStore retains an AccessStackPageMap, which is a list data structure consisting of multiple page map revisions.
So instead of saving one large AccessStackPageMap, I wrote a SecondLevelCacheSessionStore that saves a page map revision per cache entry. This leads to much better cache utilization and a whole lot less serialization on the wire. Not to mention this avoids the whole 1Mb Memcached size limit.
Before you go willy nilly with clustering, read the Wicket render strategies page. Wicket requires session affinity for buffered responses with the default rendering strategy.
Clustering Wicket has never been easier.
Here is an example on how to offload page maps to a hybrid EhCache/Memcached cache. Memcached for long term shared storage while EhCache for short-lived fast cache look ups.
@Override
protected ISessionStore newSessionStore() {
// localhost:11211 — memcached server
// "fabpagestore" — unique appender to avoid key clashes.
// 300 — 5 minute TTL for local ehcache.
return new SecondLevelCacheSessionStore(this,
new CachePageStore(Arrays.asList("localhost:11211"),"fabpagestore",300));
}
}
Here is an example on how to offload page maps to the database.
@Override
protected ISessionStore newSessionStore() {
// "fabpagestore" — unique appender to avoid key clashes.
return new SecondLevelCacheSessionStore(this,new CachePageStore(
new DBCache("jdbc:mysql://foo/mydb", "myname", "mypass", "com.driver.Name", "fabpagestore")));
}
}
Here is my CachePageStore;
import com.base.cache.AsyncMemcache;
import com.base.cache.ICache;
import org.apache.wicket.Page;
import org.apache.wicket.protocol.http.SecondLevelCacheSessionStore.IClusteredPageStore;
import org.apache.wicket.protocol.http.pagestore.AbstractPageStore;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.List;
public class CachePageStore extends AbstractPageStore implements IClusteredPageStore {
private ICache cache;
private Logger logger = LoggerFactory.getLogger(CachePageStore.class);
public CachePageStore(final List<String> servers, final String poolName, final int ttl) {
this(servers, poolName, true, ttl);
}
public CachePageStore(final List<String> servers, final String poolName, boolean async, final int ttl) {
this(new AsyncMemcache(servers, poolName, async, ttl));
}
public CachePageStore(final ICache cache) {
this.cache = cache;
}
// If pageVersion -1 must return highest page version.
protected String getKey(final String sessId, final String pageMapName, final int pageId, final int pageVersion) {
int pageVer = (pageVersion == -1) ? 0 : pageVersion;
if(pageVersion == -1) {
String[] meta = getMeta(sessId, pageMapName, pageId);
pageVer = Integer.valueOf(meta[0]);
}
return sessId + ":" + pageMapName + ":" + pageId + ":" + pageVer;
}
// If pageVersion -1 must return highest page version.
// If ajaxVersion -1 must return highest version.
public String getKey(final String sessId, final String pageMapName, final int pageId, final int pageVersion, final int ajaxVersion) {
// Default it to 0 initially
int ajaxVer = (ajaxVersion == -1) ? 0 : ajaxVersion;
int pageVer = (pageVersion == -1) ? 0 : pageVersion;
if(pageVersion == -1 || ajaxVersion == -1) {
String[] meta = getMeta(sessId, pageMapName, pageId);
if(pageVersion == -1) {
pageVer = Integer.valueOf(meta[0]);
}
if(ajaxVersion == -1) {
ajaxVer = Integer.valueOf(meta[1]);
}
}
return sessId + ":" + pageMapName + ":" + pageId + ":" + pageVer + ":" + ajaxVer;
}
protected String storeKey(final String sessionId, final Page page) {
return sessionId + ":" + page.getPageMapName() + ":" + page.getId() + ":" + page.getCurrentVersionNumber() + ":" + page.getAjaxVersionNumber();
}
protected String getBaseKey(String sessionId, Page page) {
return sessionId + ":" + page.getPageMapName() + ":" + page.getId();
}
protected String getMetaKey(String sessionId, String pageMap, int id) {
return getBaseKey(sessionId,pageMap,id)+"_meta";
}
protected String getMetaKey(String sessionId, Page page) {
return getBaseKey(sessionId,page)+"_meta";
}
protected String getBaseKey(String sessionId, String pageMap, int id) {
if(id == -1) {
return sessionId + ":" + pageMap;
} else {
return sessionId + ":" + pageMap + ":" + id;
}
}
public boolean containsPage(final String sessionId, final String pageMapName, final int pageId, final int pageVersion) {
String key = getKey(sessionId, pageMapName, pageId, pageVersion, -1);
if (logger.isDebugEnabled()) {
logger.debug("CheckExists: " + key);
}
return cache.keyExists(key);
}
public void destroy() {
}
public <T> Page getPage(final String sessionId, final String pagemap, final int id, final int versionNumber, final int ajaxVersionNumber) {
String key = getKey(sessionId, pagemap, id, versionNumber, ajaxVersionNumber);
if (logger.isDebugEnabled()) {
logger.debug("GetPage: " + key);
}
return (Page) cache.get(key);
}
public void pageAccessed(final String sessionId, final Page page) {
}
// If ID == -1 remove the entire pagemap; getBaseKey() takes care of this.
public void removePage(final String sessionId, final String pagemap, final int id) {
String key = getBaseKey(sessionId, pagemap, id);
if (logger.isDebugEnabled()) {
logger.debug("RemovePage: " + key);
}
cache.remove(getMetaKey(sessionId, pagemap, id));
for (String k : cache.getKeys()) {
if (k.startsWith(key)) {
cache.remove(k);
}
}
}
protected String[] getMeta(final String sessionId, String pageMap, int pageId) {
String metaKey = getMetaKey(sessionId,pageMap,pageId);
Object ret = cache.get(metaKey);
if (logger.isDebugEnabled()) {
logger.debug("GetMeta: " + metaKey);
}
if(ret == null) {
return new String[] {"0","0"};
} else {
return String.valueOf(ret).split(":");
}
}
protected void storeMeta(final String sessionId, final Page page) {
String metaKey = getMetaKey(sessionId, page);
Object ret = cache.get(metaKey);
if (logger.isDebugEnabled()) {
logger.debug("StoreMeta: " + metaKey);
}
if(ret == null) {
cache.put(metaKey,page.getCurrentVersionNumber()+":"+page.getAjaxVersionNumber());
} else {
String[] vals = String.valueOf(ret).split(":");
int currPage = Integer.valueOf(vals[0]);
int currAjax = Integer.valueOf(vals[1]);
if(page.getCurrentVersionNumber() > currPage) {
currPage = page.getCurrentVersionNumber();
}
if(page.getAjaxVersionNumber() > currAjax) {
currAjax = page.getAjaxVersionNumber();
}
cache.put(metaKey,currPage+":"+currAjax);
}
}
public void storePage(final String sessionId, final Page page) {
String sKey = storeKey(sessionId, page);
if (logger.isDebugEnabled()) {
logger.debug("StorePage: " + sKey);
}
cache.put(sKey, page);
storeMeta(sessionId,page);
}
public void unbind(final String sessionId) {
if (logger.isDebugEnabled()) {
logger.debug("Unbind: " + sessionId);
}
for (String key : cache.getKeys()) {
if (key.startsWith(sessionId)) {
cache.remove(key);
}
}
}
}
HAProxy 1.4dev6 broken on Solaris
2 Comments | Filed under administration open source programming (1 votes, average: 5.00 out of 5) Loading ...Update: I feel like a jackass now, I thought I was running this against the stable haproxy build, but in reality this was against haproxy-1.4dev6. DOH! Well on the bright-side, I am helping out the author fix a potentially critical bug. Here is the truss and tcp dump if anyone cares.
Well yet another Solaris specific bug/issue to report. HAProxy resets long running connections. Meaning users on slow bandwidth connections are affected by this. I have sent tcpdumps and logs to the author of HAProxy, hopefully this bug/issue would be resolved. I am writing this as a precautionary warning to other Solaris admins out there.
Here the way to trigger this, see if your service is affected by this.
Result:
–2010-01-20 11:19:29– http://somesite.com/onebigfile.txt
Resolving somesite.com (somesite.com)… 72.11.142.91
Connecting to somesite.com (somesite.com)|72.11.142.91|:84… connected.
HTTP request sent, awaiting response… 200 OK
Length: 3806025 (3.6M)
Saving to: “onebigfile.txt”
7% [====> ] 269,008 20.1K/s in 13s
2010-01-20 11:19:42 (20.1 KB/s) – Read error at byte 269008/3806025 (Connection reset by peer). Retrying.
–2010-01-20 11:19:43– (try: 2) http://somesite.com/onebigfile.txt
Connecting to somesite.com (somesite.com)|72.11.142.91|:84… connected.
HTTP request sent, awaiting response… 200 OK
Length: 3806025 (3.6M)
Saving to: “onebigfile.txt”
4% [==> ] 186,016 20.0K/s eta
/Raging, why are there so many Solaris TCP issues? First Varnish? now HAProxy? ARGHHHHH!@#!@
Relax, Satan is on your side.
2 Comments | Filed under administration main open source programming (1 votes, average: 5.00 out of 5) Loading ...I just pushed up a new version of Satan to GitHub. For the uniformed uninformed Satan is my process reaper for run away unix processes. Satan was designed to work with Solaris’ SMF self-healing properties. Basically, Satan kills while SMF revives. The new version that was pushed up contains HTTP health checks, so Satan now has the ability to kill processes that are not responding back with a HTTP/200 response code.
The motivation behind HTTP health checks was because once a month or so at Fabulously40 our ActiveMQ would break down while still accepting connections, the only way to figure out if it was zombified was to check the HTTP administrator interface. If the ActiveMQ instance was actually knelled over, the administrator interface would come back with a HTTP/500 response code, hence the birth of HTTP health checks.
Here is our Satan configuration file that we use at Fabulously40.
The “args” property might be a bit confusing, it is a snippet of text that Satan looks for in the arguments passed to your application to identify the running process. So for example, if you start your ActiveMQ instance with the following arguments; “java -jar activemq.jar -Dactivemq=8161 -XXXXX” Placing “8161″ in args property would be a good unique identifier for Satan to pick up on.
Satan.watch do |s| s.name = "jvm instances" # name of job s.user = "webservd" # under what user s.group = "webservd" # under what group s.deamon = "java" # deamon binary name to grep for s.args = nil # globally look for specific arguments, optional s.debug = true # if to write out debug information s.safe_mode = false # If in safe mode, satan will not kill ;-( s.interval = 10.seconds # interval to run at to collect statistics s.sleep_after_kill = 1.minute # sleep after killing, satan is tired! s.contact = "victori@fabulously40.com" # admin contact, optional if you want email alerts s.kill_if do |process| process.condition(:cpu) do |cpu| # on cpu condition cpu.name = "50% CPU limit" # name for job cpu.args = "jetty" # make sure this is a jetty process, optional cpu.above = 48.percent # if above certain percentage cpu.times = 5 # how many times we can hit this condition before killing end process.condition(:memory) do |memory| # on memory condition memory.name = "850MB limit" # name for job memory.args = "jetty" # make sure this is a jetty process, optional memory.above = 850.megabytes # limit for memory use memory.times = 5 # how many times we can hit this condition before killing end # ActiveMQ tends to die on us under heavy load so we need the power of satan! process.condition(:http) do |http| # on http condition http.name = "HTTP ActiveMQ Check" # name for job http.args = "8161" # look for specific app arguments # to associate app to URI http.uri = "http://localhost:8161/admin/queues.jsp" # the URI http.times = 5 # how many times before kill end end end
Why deadlines go to hell
Leave a comment | Filed under administration main programming (2 votes, average: 5.00 out of 5) Loading ...Ted Dziuba beautifully articulated why deadlines go to crap and seemingly straight forward tasks go out the window. You sir have done a public service for us all, thank you.
What I hate is fording endless rivers of horseshit that are in the way of seemingly simple tasks. And I hate it even more when I have to explain to a non-programmer what I am doing, "building LXML against a different version of libiconv because I think it might be the source of a crash". "But all I asked you to do was parse some documents." Good times.
Squid Headers Normalization Patch
Leave a comment | Filed under administration programming (2 votes, average: 5.00 out of 5) Loading ...Since Varnish did not work out on Solaris yet again. I have decided to bite the bullet and write a headers normalization patch for Squid 2.7. This patch should produce much better cache hit rates with Squid. Efficiency++
What the patch does:
1. Removes Cache-Control request headers, don’t let clients by-pass cache if it is primed.
2. Normalize Accept-Encoding Headers for a higher cache hit rate.
3. Clear Accept-Encoding Headers for content that should not be compressed such as image,video and audio.
and the patch: squid-headers-normalization.patch
Update: Fixed a minor memory leak, all good now.
Update 2: Added audio exception to strip accept-encoding.
--- src/client_side.c.og 2010-01-20 12:00:56.000000000 -0800 +++ src/client_side.c 2010-01-19 20:35:31.000000000 -0800 @@ -3983,6 +3983,7 @@ errorAppendEntry(http->entry, err); return -1; } + /* compile headers */ /* we should skip request line! */ if ((http->http_ver.major >= 1) && !httpMsgParseRequestHeader(request, &msg)) { @@ -3992,10 +3993,59 @@ err->url = xstrdup(http->uri); http->al.http.code = err->http_status; http->log_type = LOG_TCP_DENIED; + http->entry = clientCreateStoreEntry(http, method, null_request_flags); errorAppendEntry(http->entry, err); return -1; } + + /* + * Normalize Request Cache-Control / If-Modified-Since Headers + * Don't let client by-pass the cache if there is cached content. + */ + if(httpHeaderHas(&request->header,HDR_CACHE_CONTROL)) { + httpHeaderDelByName(&request->header,"cache-control"); + } + + /* + * Un-comment this if you want Squid to always respond with the request + * instead of returning back with a 304 if the cache has not changed. + */ + /* + if(httpHeaderHas(&request->header,HDR_IF_MODIFIED_SINCE)) { + httpHeaderDelByName(&request->header,"if-modified-since"); + }*/ + + /* + * Normalize Accept-Encoding Headers sent from client + */ + if(httpHeaderHas(&request->header,HDR_ACCEPT_ENCODING)) { + String val = httpHeaderGetByName(&request->header,"accept-encoding"); + if(val.buf) { + if(strstr(val.buf,"gzip") != NULL) { + httpHeaderDelByName(&request->header,"accept-encoding"); + httpHeaderPutStr(&request->header,HDR_ACCEPT_ENCODING,"gzip"); + } else if(strstr(val.buf,"deflate") != NULL) { + httpHeaderDelByName(&request->header,"accept-encoding"); + httpHeaderPutStr(&request->header,HDR_ACCEPT_ENCODING,"deflate"); + } else { + httpHeaderDelByName(&request->header,"accept-encoding"); + } + } + stringClean(&val); + } + + /* + * Normalize Accept-Encoding Headers for video/image content + */ + char *mime_type = mimeGetContentType(http->uri); + if(mime_type) { + if(strstr(mime_type,"image") != NULL || strstr(mime_type,"video") != NULL || strstr(mime_type,"audio") != NULL) { + httpHeaderDelByName(&request->header,"accept-encoding"); + } + } + + /* * If we read past the end of this request, move the remaining * data to the beginning
MogileFS Client on JRuby!
Leave a comment | Filed under programming (1 votes, average: 5.00 out of 5) Loading ...Are you running JRuby in production? Do you want distributed file storage for your “enterprise” application? Look no further, MogileFS is here.
MogileFS-Client has compatibility issues with JRuby due to it’s use of the low level Socket class. JRuby 1.5-dev does not yet support all the Socket methods, so here is a monkey patch to get the ruby mogilefs client working on JRuby. Yes it blocks, but who cares JRuby has native threads.
This is exactly why I love Ruby; monkey patching.
def self.mogilefs_new(host,port,timeout=5.0)
TCPSocket.open(host,port,timeout)
end
end
class TCPSocket
attr_accessor :mogilefs_addr, :mogilefs_connected, :mogilefs_size, :mogilefs_tcp_cork
def self.open(host,port,timeout = 5.0)
super(host,port.to_i)
end
def readable?
true
end
def write_nonblock(data)
write(data)
end
def recv_nonblock(size,arg)
recv(size,arg)
end
def mogilefs_init(host = nil, port = nil)
true
end
end
Here is an example test case on how to get it all to work.
require ‘mogilefs’
# jmogilefs.rb is the monkey patch above
# load it after loading mogilefs client.
require ‘jmogilefs.rb’
mg = MogileFS::MogileFS.new(:domain=>‘testserv’,:hosts=>[‘xxx.xxx.xxx.xxx:6001′])
p mg.get_file_data ‘video:100:default.jpg’
p mg.get_paths ‘video:100:default.jpg’,true
mg.list_keys(‘video:100′)[0].each do |f|
p f
end
MogileFS on Solaris
1 Comment | Filed under main programming (1 votes, average: 5.00 out of 5) Loading ...*Update* Patches got accepted into MogileFS Trunk
Just go check out trunk, it has all my patches already included.
http://code.sixapart.com/svn/mogilefs/trunk/
The only thing you need is my mogstored disk patch which is still pending. All the issues revolving around postgresql and solaris have been already included in trunk.
I fixed a few issues with MogileFS and Solaris. MogileFS should run wonderfully on Solaris with my patches applied.
Directory for all my patches: http://victori.uploadbooth.com/patches
http://victori.uploadbooth.com/patches/solaris-disk-du.patch
This patch fixes mogstored to work with solaris’s df utility.
http://victori.uploadbooth.com/patches/store-max-requests.patch
This patch adds a new feature to the MogileFS Tracker – max_requests.
The default is 0, but it is suggested you set it to 1000 max_requests, to avoid memory leaks.
The tracker will give out the database handle up to the max_requests limit before expiring the connection for a new one. This avoids memory leaks with long running persistent connections. PostgreSQL has issues with long persistent connections, it accumulates a lot of ram and does not let go until the process/connection is killed off. This patch makes sure that the connection is expired after so many dbh handle requests.
http://victori.uploadbooth.com/patches/mogilefs-sunos-pg.patch
This patch applies the InactiveDestroy argument to avoid the MogileFS Tracker locking up with the PostgreSQL store on Solaris.
http://victori.uploadbooth.com/patches/solaris-mogilefs-full.patch
This is the full patch for all my fixes.
I am slowly migrating our fab40 static asset data to MogileFS. I have imported >300,000 images, no issues with my patches so far.
/ PLUG go make an account on uploadbooth!
Enjoy
