Letsgetdugg» programming http://letsgetdugg.com Random tech jargon Tue, 31 Aug 2010 22:18:45 +0000 en hourly 1 http://wordpress.org/?v=3.0.1 Ruby LocaleTranslator http://letsgetdugg.com/2010/07/20/ruby-locale-translator/ http://letsgetdugg.com/2010/07/20/ruby-locale-translator/#comments Tue, 20 Jul 2010 21:05:55 +0000 victori http://letsgetdugg.com/?p=741 Today, I am open sourcing my Ruby LocaleTranslator; the translator uses google’s translator API to translate a primary seed locale into various other languages. This eases the creation of multi-lingual sites. Not only can the LocaleTranslator translate your main seed locale into different languages but it can also recursively merge in differences, this comes in handy if you have hand-optimized your translated locales.

Viva Localization!

My projects that use the LocaleTranslator; UploadBooth, PasteBooth and ShrinkBooth.

LocaleTranslator Examples

en.yml

site:
  hello_world: Hello World!
  home: Home
  statement: Localization should be simple!

Batch Conversion of your English locale.

#!/opt/local/bin/ruby

require ‘monkey-patches.rb’
require ‘locale_translator.rb’

en_yml = YAML::load(File.open(‘en.yml’))

[:de,:ru].each do |lang|
  lang_yml = LocaleTranslator.translate(en_yml,
    :to=>lang,
    :html=>true,
    :key=>‘GOOGLE API KEY’)
  f = File.new("#{lang.to_s.downcase}.yml","w")
  f.puts(lang_yml.ya2yaml(:syck_compatible => true))
  f.close
  p "Translated to #{lang.to_s}"
end

Merge in new locale keys from your English Locale into your already translated Russian locale.

#!/opt/local/bin/ruby

require ‘monkey-patches.rb’
require ‘locale_translator.rb’

en_yml = YAML::load(File.open(‘en.yml’))
ru_yml = YAML::load(File.open(‘ru.yml’))

ru_new_yml = LocaleTranslator.translate(en_yml,
  :to=>:ru,
  :html=>true,
  :merge=>ru_yml,
  :key=>‘GOOGLE API KEY’)
puts ru_new_yml.ya2yaml(:syck_compatible => true)

The Implementation Code

Support Monkey Patches

monkey-patches.rb

class Hash
  def to_list
    h2l(self)
  end
 
  def diff(hash)
    hsh = {}
    this = self
    hash.each do |k,v|
      if v.kind_of?Hash and this.key?k
        tmp = this[k].diff(v)
        hsh[k] = tmp if tmp.size > 0
      else
        hsh[k] = v unless this.key?k
      end
    end
    hsh
  end
 
  def merge_r(hash)
    hsh = {}
    this = self
    hash.each do |k,v|
      if v.kind_of?Hash
        hsh[k] = this[k].merge_r(v)
      else
        hsh[k] = v
      end
    end
    self.merge(hsh)
  end
 
  private
    def h2l(hash)
      list = []
      hash.each {|k,v| list = (v.kind_of?Hash) ? list.merge_with_dups(h2l(v)) : list << v }
      list
    end
end

class Array
  def chunk(p=2)
    return [] if p.zero?
    p_size = (length.to_f / p).ceil
    [first(p_size), *last(length - p_size).chunk(p - 1)]
  end
  def to_hash(hash)
    l2h(hash,self)
  end
  def merge(arr)
    self | arr
  end
  def merge_with_dups(arr)
    temp = []
    self.each {|a| temp << a }
    arr.each {|a| temp << a }
    temp
  end
  def merge!(arr)
    temp = self.clone
    self.clear
    temp.each {|a| self << a }
    arr.each {|a| self << a unless temp.include?a }
    true
  end
  def merge_with_dups!(arr)
    temp = self.clone
    self.clear
    temp.each {|a| self << a }
    arr.each {|a| self << a }
    true
  end
  private
    def l2h(hash,lst)
      hsh = {}
      hash.each {|k,v| hsh[k] = (v.kind_of?Hash) ? l2h(v,lst) : lst.shift }
      hsh
    end
end

The LocaleTranslator Implementation

You need the ya2yaml and easy_translate gems. Ya2YAML can export locales in UTF-8 unlike the standard yaml implementation that can only export in binary for non-standard ascii.

locale-translator.rb

$KCODE = ‘UTF8′ if RUBY_VERSION < ’1.9.0′
require ‘rubygems’
require ‘ya2yaml’
require ‘yaml’
require ‘easy_translate’

class LocaleTranslator
  def self.translate(text,opts)
    opts[:to] = [opts[:to]] if opts[:to] and !opts[:to].kind_of?Array
   
    if opts[:merge].kind_of?Hash and text.kind_of?Hash
      diff = opts[:merge].diff(text)
      diff_hsh = LocaleTranslator.translate(diff,:to=>opts[:to],:html=>true)
      return opts[:merge].merge_r(diff_hsh)
    end
   
    if text.kind_of?Hash
      t_arr = text.to_list
      t_arr = t_arr.first if t_arr.size == 1
      tout_arr = LocaleTranslator.translate(t_arr,:to=>opts[:to],:html=>true)
      tout_arr = [tout_arr] if tout_arr.kind_of?String
      tout_arr.to_hash(text)
    elsif text.kind_of?Array
      if text.size > 50
        out = []
        text.chunk.each {|l| out.merge_with_dups!(EasyTranslate.translate(l,opts).first) }
        out
      else
        text = text.first if text.size == 1
        EasyTranslate.translate(text,opts).first
      end
    else
      EasyTranslate.translate(text,opts).first
    end
  end
end

Share and Enjoy: Digg Sphinn del.icio.us Facebook Mixx Google Bookmarks Design Float DZone Reddit Slashdot TwitThis

]]> http://letsgetdugg.com/2010/07/20/ruby-locale-translator/feed/ 0 Scala yEnc Decoder http://letsgetdugg.com/2010/06/22/scala-yenc-decoder/ http://letsgetdugg.com/2010/06/22/scala-yenc-decoder/#comments Tue, 22 Jun 2010 22:07:17 +0000 victori http://letsgetdugg.com/?p=671 SYEnc is a Scala decoder for the yEnc format that is based on Alex Russ’s Java yEnc Decoder. SYEnc was designed to be used as a library by applications needing to use yEnc to decode data. It should be thread-safe, so don’t worry about using it in a threaded context.

Uploaded to github: http://github.com/victori/syenc

Example:

val baos = new ByteArrayOutputStream
SYEnc.decode(new ByteArrayInputStream(stringFromStream.getBytes("UTF-8")), baos)
println(baos.toString)
Share and Enjoy: Digg Sphinn del.icio.us Facebook Mixx Google Bookmarks Design Float DZone Reddit Slashdot TwitThis

]]> http://letsgetdugg.com/2010/06/22/scala-yenc-decoder/feed/ 0 Automated ZFS Snapshots http://letsgetdugg.com/2010/03/30/automated-zfs-snapshots/ http://letsgetdugg.com/2010/03/30/automated-zfs-snapshots/#comments Wed, 31 Mar 2010 00:04:33 +0000 victori http://letsgetdugg.com/?p=653 I needed something like zfs-auto-snapshot written by Tim Foster but portable so it works on all systems that support ZFS. I reviewed a few scripts on github and was unhappy with what was out there so I decided to write my own.

With zbackup.rb you can define what to snapshot and how many rotation days you want to go back.

So say you want a month of snapshots:

/usr/bin/zbackup.rb iraidz/zWork 30

Simple, no? ;-)

#!/usr/bin/env ruby

# Create snapshots for a 7 day rotation.
# ./zbackup.rb iraidz/zWork 7
#
# Add to crontab
# crontab -e
# 0 2 * * * /usr/bin/zbackup.rb iraidz/zWork 7

pool = ARGV[0]
days_back = ARGV[1].to_i

if pool.nil? or pool.empty?
  puts "\nDefine the pool you want to snapshot:"
  puts "\tex: zbackup.rb iraidz/zWork 7\n\n"
  exit 0
end

if days_back.nil? or days_back < 1
  puts "\nDefine how many days for your rotation:"
  puts "\tex: zbackup.rb iraidz/zWork 7\n\n"
  exit 0
end

# response from zfs list
curr_snaps = `zfs list -t snapshot -o name`
# days back limit variable
date_back = Time.now - (86400*days_back)

curr_snaps.split(/\n/).each do |pline|
  if m = pline.match(/#{pool}\@([0-9]+)\-([0-9]+)\-([0-9]+)/)
    if date_back >= Time.local(m[1],m[2],m[3])
      `zfs destroy #{pline}`
    end
  end
end

# take snapshot for this run if needed.
month = Time.now.month
day = Time.now.day
year = Time.now.year
if curr_snaps !~ /#{pool}\@#{year}\-#{month}\-#{day}/
  `zfs snapshot -r #{pool}@#{year}-#{month}-#{day}`
end

Share and Enjoy: Digg Sphinn del.icio.us Facebook Mixx Google Bookmarks Design Float DZone Reddit Slashdot TwitThis

]]> http://letsgetdugg.com/2010/03/30/automated-zfs-snapshots/feed/ 1 Clustering Wicket for fun and profit! http://letsgetdugg.com/2010/02/07/clustering-wicket-for-fun-and-profit/ http://letsgetdugg.com/2010/02/07/clustering-wicket-for-fun-and-profit/#comments Sun, 07 Feb 2010 17:55:09 +0000 victori http://letsgetdugg.com/?p=615 I hate expired sessions, death to all expired sessions. Traditionally a Java servlet container has a fixed session time, a flood of traffic can potentially cause JVM OOM errors if the session time is set too high. I wanted a smart session container that can hold onto sessions for as long as possible and expire sessions only when it is absolutely necessary; A Memcached store would be perfect for this.

There for I recently open sourced the jetty-session-store to solve this problem. With the jetty-session-store you can save your session state to Ehcache, Memcached or the database. State should not be bound to a single JVM, Viva Shared Session Stores!

So now that jetty-session-store is out in the wild you can technically cluster Wicket using just the HttpSessionStore. However, it isn’t very efficient with the way Memcached allocates data in fixed sized cache buckets.

1. Wicket sessions under the HttpSessionStore can get quite large, well over 1Mb in size. A Wicket session not only stores the session state but also the previous serialized pages the user has visited.

2. Serializing and de-serializing a large data structure can get expensive. The HttpSessionStore retains an AccessStackPageMap, which is a list data structure consisting of multiple page map revisions.

So instead of saving one large AccessStackPageMap, I wrote a SecondLevelCacheSessionStore that saves a page map revision per cache entry. This leads to much better cache utilization and a whole lot less serialization on the wire. Not to mention this avoids the whole 1Mb Memcached size limit.

Before you go willy nilly with clustering, read the Wicket render strategies page. Wicket requires session affinity for buffered responses with the default rendering strategy.

Clustering Wicket has never been easier.

Here is an example on how to offload page maps to a hybrid EhCache/Memcached cache. Memcached for long term shared storage while EhCache for short-lived fast cache look ups.

public class WebApp extends WebApplication {
        @Override
        protected ISessionStore newSessionStore() {
                // localhost:11211 — memcached server
                // "fabpagestore" — unique appender to avoid key clashes.
                // 300 — 5 minute TTL for local ehcache.
                return new SecondLevelCacheSessionStore(this,
                                new CachePageStore(Arrays.asList("localhost:11211"),"fabpagestore",300));
        }
}

Here is an example on how to offload page maps to the database.

public class WebApp extends WebApplication {
        @Override
        protected ISessionStore newSessionStore() {
                // "fabpagestore" — unique appender to avoid key clashes.
                return new SecondLevelCacheSessionStore(this,new CachePageStore(
                                new DBCache("jdbc:mysql://foo/mydb", "myname", "mypass", "com.driver.Name", "fabpagestore")));
        }
}

Here is my CachePageStore;

package com.base.pagestore;

import com.base.cache.AsyncMemcache;
import com.base.cache.ICache;
import org.apache.wicket.Page;
import org.apache.wicket.protocol.http.SecondLevelCacheSessionStore.IClusteredPageStore;
import org.apache.wicket.protocol.http.pagestore.AbstractPageStore;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.List;

public class CachePageStore extends AbstractPageStore implements IClusteredPageStore {
    private ICache cache;
    private Logger logger = LoggerFactory.getLogger(CachePageStore.class);

    public CachePageStore(final List<String> servers, final String poolName, final int ttl) {
        this(servers, poolName, true, ttl);
    }

    public CachePageStore(final List<String> servers, final String poolName, boolean async, final int ttl) {
        this(new AsyncMemcache(servers, poolName, async, ttl));
    }

    public CachePageStore(final ICache cache) {
        this.cache = cache;
    }

    // If pageVersion -1 must return highest page version.
    protected String getKey(final String sessId, final String pageMapName, final int pageId, final int pageVersion) {
        int pageVer = (pageVersion == -1) ? 0 : pageVersion;
        if(pageVersion == -1) {
            String[] meta = getMeta(sessId, pageMapName, pageId);
            pageVer = Integer.valueOf(meta[0]);
        }

        return sessId + ":" + pageMapName + ":" + pageId + ":" + pageVer;
    }

    // If pageVersion -1 must return highest page version.
    // If ajaxVersion -1 must return highest version.
    public String getKey(final String sessId, final String pageMapName, final int pageId, final int pageVersion, final int ajaxVersion) {
        // Default it to 0 initially
        int ajaxVer = (ajaxVersion == -1) ? 0 : ajaxVersion;
        int pageVer = (pageVersion == -1) ? 0 : pageVersion;

        if(pageVersion == -1 || ajaxVersion == -1) {
            String[] meta = getMeta(sessId, pageMapName, pageId);
            if(pageVersion == -1) {
                pageVer = Integer.valueOf(meta[0]);
            }
            if(ajaxVersion == -1) {
                ajaxVer = Integer.valueOf(meta[1]);
            }
        }

        return sessId + ":" + pageMapName + ":" + pageId + ":" + pageVer + ":" + ajaxVer;
    }

    protected String storeKey(final String sessionId, final Page page) {
        return sessionId + ":" + page.getPageMapName() + ":" + page.getId() + ":" + page.getCurrentVersionNumber() + ":" + page.getAjaxVersionNumber();
    }

    protected String getBaseKey(String sessionId, Page page) {
        return sessionId + ":" + page.getPageMapName() + ":" + page.getId();
    }

    protected String getMetaKey(String sessionId, String pageMap, int id) {
        return getBaseKey(sessionId,pageMap,id)+"_meta";
    }

    protected String getMetaKey(String sessionId, Page page) {
        return getBaseKey(sessionId,page)+"_meta";
    }

    protected String getBaseKey(String sessionId, String pageMap, int id) {
        if(id == -1) {
            return sessionId + ":" + pageMap;
        } else {
            return sessionId + ":" + pageMap + ":" + id;
        }
    }
    public boolean containsPage(final String sessionId, final String pageMapName, final int pageId, final int pageVersion) {
        String key = getKey(sessionId, pageMapName, pageId, pageVersion, -1);
        if (logger.isDebugEnabled()) {
            logger.debug("CheckExists: " + key);
        }
        return cache.keyExists(key);
    }

    public void destroy() {
    }

    public <T> Page getPage(final String sessionId, final String pagemap, final int id, final int versionNumber, final int ajaxVersionNumber) {
        String key = getKey(sessionId, pagemap, id, versionNumber, ajaxVersionNumber);
        if (logger.isDebugEnabled()) {
            logger.debug("GetPage: " + key);
        }
        return (Page) cache.get(key);
    }

    public void pageAccessed(final String sessionId, final Page page) {
    }

    // If ID == -1 remove the entire pagemap; getBaseKey() takes care of this.
    public void removePage(final String sessionId, final String pagemap, final int id) {
        String key = getBaseKey(sessionId, pagemap, id);

        if (logger.isDebugEnabled()) {
            logger.debug("RemovePage: " + key);
        }

        cache.remove(getMetaKey(sessionId, pagemap, id));
        for (String k : cache.getKeys()) {
            if (k.startsWith(key)) {
                cache.remove(k);
            }
        }
    }

    protected String[] getMeta(final String sessionId, String pageMap, int pageId) {
        String metaKey = getMetaKey(sessionId,pageMap,pageId);
        Object ret = cache.get(metaKey);

        if (logger.isDebugEnabled()) {
            logger.debug("GetMeta: " + metaKey);
        }

        if(ret == null) {
            return new String[] {"0","0"};
        } else {
            return String.valueOf(ret).split(":");
        }
    }

    protected void storeMeta(final String sessionId, final Page page) {
        String metaKey = getMetaKey(sessionId, page);
        Object ret = cache.get(metaKey);

        if (logger.isDebugEnabled()) {
            logger.debug("StoreMeta: " + metaKey);
        }

        if(ret == null) {
            cache.put(metaKey,page.getCurrentVersionNumber()+":"+page.getAjaxVersionNumber());
        } else {
            String[] vals = String.valueOf(ret).split(":");
            int currPage = Integer.valueOf(vals[0]);
            int currAjax = Integer.valueOf(vals[1]);

            if(page.getCurrentVersionNumber() > currPage) {
                currPage = page.getCurrentVersionNumber();
            }
            if(page.getAjaxVersionNumber() > currAjax) {
                currAjax = page.getAjaxVersionNumber();
            }
            cache.put(metaKey,currPage+":"+currAjax);
        }
    }

    public void storePage(final String sessionId, final Page page) {
        String sKey = storeKey(sessionId, page);

        if (logger.isDebugEnabled()) {
            logger.debug("StorePage: " + sKey);
        }

        cache.put(sKey, page);
        storeMeta(sessionId,page);
    }

    public void unbind(final String sessionId) {
        if (logger.isDebugEnabled()) {
            logger.debug("Unbind: " + sessionId);
        }

        for (String key : cache.getKeys()) {
            if (key.startsWith(sessionId)) {
                cache.remove(key);
            }
        }
    }

}

Share and Enjoy: Digg Sphinn del.icio.us Facebook Mixx Google Bookmarks Design Float DZone Reddit Slashdot TwitThis

]]> http://letsgetdugg.com/2010/02/07/clustering-wicket-for-fun-and-profit/feed/ 0 HAProxy 1.4dev6 broken on Solaris http://letsgetdugg.com/2010/01/20/haproxy-broken-on-solaris/ http://letsgetdugg.com/2010/01/20/haproxy-broken-on-solaris/#comments Wed, 20 Jan 2010 19:45:16 +0000 victori http://letsgetdugg.com/?p=567 Update: I feel like a jackass now, I thought I was running this against the stable haproxy build, but in reality this was against haproxy-1.4dev6. DOH! Well on the bright-side, I am helping out the author fix a potentially critical bug. Here is the truss and tcp dump if anyone cares.

Well yet another Solaris specific bug/issue to report. HAProxy resets long running connections. Meaning users on slow bandwidth connections are affected by this. I have sent tcpdumps and logs to the author of HAProxy, hopefully this bug/issue would be resolved. I am writing this as a precautionary warning to other Solaris admins out there.

Here the way to trigger this, see if your service is affected by this.

wget –limit-rate=2k http://somesite.com/onebigfile.txt

Result:

syris:~ victori$ wget –limit-rate=20k http://somesite.com/onebigfile.txt
–2010-01-20 11:19:29– http://somesite.com/onebigfile.txt
Resolving somesite.com (somesite.com)… 72.11.142.91
Connecting to somesite.com (somesite.com)|72.11.142.91|:84… connected.
HTTP request sent, awaiting response… 200 OK
Length: 3806025 (3.6M)
Saving to: “onebigfile.txt”

7% [====> ] 269,008 20.1K/s in 13s

2010-01-20 11:19:42 (20.1 KB/s) – Read error at byte 269008/3806025 (Connection reset by peer). Retrying.

–2010-01-20 11:19:43– (try: 2) http://somesite.com/onebigfile.txt
Connecting to somesite.com (somesite.com)|72.11.142.91|:84… connected.
HTTP request sent, awaiting response… 200 OK
Length: 3806025 (3.6M)
Saving to: “onebigfile.txt”

4% [==> ] 186,016 20.0K/s eta

/Raging, why are there so many Solaris TCP issues? First Varnish? now HAProxy? ARGHHHHH!@#!@

Share and Enjoy: Digg Sphinn del.icio.us Facebook Mixx Google Bookmarks Design Float DZone Reddit Slashdot TwitThis

]]> http://letsgetdugg.com/2010/01/20/haproxy-broken-on-solaris/feed/ 2 Relax, Satan is on your side. http://letsgetdugg.com/2010/01/05/relax-satan-is-on-your-side/ http://letsgetdugg.com/2010/01/05/relax-satan-is-on-your-side/#comments Tue, 05 Jan 2010 22:17:20 +0000 victori http://letsgetdugg.com/?p=540 I just pushed up a new version of Satan to GitHub. For the uniformed uninformed Satan is my process reaper for run away unix processes. Satan was designed to work with Solaris’ SMF self-healing properties. Basically, Satan kills while SMF revives. The new version that was pushed up contains HTTP health checks, so Satan now has the ability to kill processes that are not responding back with a HTTP/200 response code.

The motivation behind HTTP health checks was because once a month or so at Fabulously40 our ActiveMQ would break down while still accepting connections, the only way to figure out if it was zombified was to check the HTTP administrator interface. If the ActiveMQ instance was actually knelled over, the administrator interface would come back with a HTTP/500 response code, hence the birth of HTTP health checks.

Here is our Satan configuration file that we use at Fabulously40.

The “args” property might be a bit confusing, it is a snippet of text that Satan looks for in the arguments passed to your application to identify the running process. So for example, if you start your ActiveMQ instance with the following arguments; “java -jar activemq.jar -Dactivemq=8161 -XXXXX” Placing “8161″ in args property would be a good unique identifier for Satan to pick up on.

Satan.watch do |s|
  s.name = "jvm instances"                # name of job
  s.user = "webservd"                     # under what user
  s.group = "webservd"                    # under what group
  s.deamon = "java"                       # deamon binary name to grep for
  s.args = nil                            # globally look for specific arguments, optional
  s.debug = true                          # if to write out debug information
  s.safe_mode = false                     # If in safe mode, satan will not kill ;-(
  s.interval = 10.seconds                 # interval to run at to collect statistics
  s.sleep_after_kill = 1.minute           # sleep after killing, satan is tired!
  s.contact = "victori@fabulously40.com"  # admin contact, optional if you want email alerts

  s.kill_if do |process|
    process.condition(:cpu) do |cpu|      # on cpu condition
      cpu.name  = "50% CPU limit"         # name for job
      cpu.args  = "jetty"                 # make sure this is a jetty process, optional
      cpu.above = 48.percent              # if above certain percentage
      cpu.times = 5                       # how many times we can hit this condition before killing
    end

    process.condition(:memory) do |memory|  # on memory condition
      memory.name  = "850MB limit"          # name for job
      memory.args  = "jetty"                # make sure this is a jetty process, optional
      memory.above = 850.megabytes          # limit for memory use
      memory.times = 5                      # how many times we can hit this condition before killing
    end

    # ActiveMQ tends to die on us under heavy load so we need the power of satan!
    process.condition(:http) do |http|                        # on http condition
      http.name   = "HTTP ActiveMQ Check"                     # name for job
      http.args   = "8161"                                    # look for specific app arguments
                                                              # to associate app to URI
      http.uri    = "http://localhost:8161/admin/queues.jsp"  # the URI
      http.times  = 5                                         # how many times before kill
    end
  end
end
Share and Enjoy: Digg Sphinn del.icio.us Facebook Mixx Google Bookmarks Design Float DZone Reddit Slashdot TwitThis

]]> http://letsgetdugg.com/2010/01/05/relax-satan-is-on-your-side/feed/ 2 Why deadlines go to hell http://letsgetdugg.com/2009/12/16/why-deadlines-go-to-hell/ http://letsgetdugg.com/2009/12/16/why-deadlines-go-to-hell/#comments Wed, 16 Dec 2009 21:55:27 +0000 victori http://letsgetdugg.com/?p=527 Ted Dziuba beautifully articulated why deadlines go to crap and seemingly straight forward tasks go out the window. You sir have done a public service for us all, thank you.

What I hate is fording endless rivers of horseshit that are in the way
of seemingly simple tasks. And I hate it even more when I have to explain
to a non-programmer what I am doing, "building LXML against a different
version of libiconv because I think it might be the source of a crash". 

"But all I asked you to do was parse some documents."

Good times.
Share and Enjoy: Digg Sphinn del.icio.us Facebook Mixx Google Bookmarks Design Float DZone Reddit Slashdot TwitThis

]]> http://letsgetdugg.com/2009/12/16/why-deadlines-go-to-hell/feed/ 0 Squid Headers Normalization Patch http://letsgetdugg.com/2009/12/06/squid-headers-normalization-patch/ http://letsgetdugg.com/2009/12/06/squid-headers-normalization-patch/#comments Mon, 07 Dec 2009 01:48:38 +0000 victori http://letsgetdugg.com/?p=503 Since Varnish did not work out on Solaris yet again. I have decided to bite the bullet and write a headers normalization patch for Squid 2.7. This patch should produce much better cache hit rates with Squid. Efficiency++

What the patch does:

1. Removes Cache-Control request headers, don’t let clients by-pass cache if it is primed.
2. Normalize Accept-Encoding Headers for a higher cache hit rate.
3. Clear Accept-Encoding Headers for content that should not be compressed such as image,video and audio.

and the patch: squid-headers-normalization.patch

Update: Fixed a minor memory leak, all good now.
Update 2: Added audio exception to strip accept-encoding.

--- src/client_side.c.og	2010-01-20 12:00:56.000000000 -0800
+++ src/client_side.c	2010-01-19 20:35:31.000000000 -0800
@@ -3983,6 +3983,7 @@
 	    errorAppendEntry(http->entry, err);
 	    return -1;
 	}
+
 	/* compile headers */
 	/* we should skip request line! */
 	if ((http->http_ver.major >= 1) && !httpMsgParseRequestHeader(request, &msg)) {
@@ -3992,10 +3993,59 @@
 	    err->url = xstrdup(http->uri);
 	    http->al.http.code = err->http_status;
 	    http->log_type = LOG_TCP_DENIED;
+
 	    http->entry = clientCreateStoreEntry(http, method, null_request_flags);
 	    errorAppendEntry(http->entry, err);
 	    return -1;
 	}
+
+	/*
+	 *  Normalize Request Cache-Control / If-Modified-Since Headers
+	 *  Don't let client by-pass the cache if there is cached content.
+	 */
+	if(httpHeaderHas(&request->header,HDR_CACHE_CONTROL)) {
+		httpHeaderDelByName(&request->header,"cache-control");
+	}
+
+	/*
+	 * Un-comment this if you want Squid to always respond with the request
+	 * instead of returning back with a 304 if the cache has not changed.
+	 */
+	/*
+	if(httpHeaderHas(&request->header,HDR_IF_MODIFIED_SINCE)) {
+		httpHeaderDelByName(&request->header,"if-modified-since");
+	}*/
+
+	/*
+	 * Normalize Accept-Encoding Headers sent from client
+	 */
+	if(httpHeaderHas(&request->header,HDR_ACCEPT_ENCODING)) {
+		String val = httpHeaderGetByName(&request->header,"accept-encoding");
+		if(val.buf) {
+			if(strstr(val.buf,"gzip") != NULL) {
+				httpHeaderDelByName(&request->header,"accept-encoding");
+				httpHeaderPutStr(&request->header,HDR_ACCEPT_ENCODING,"gzip");
+			} else if(strstr(val.buf,"deflate") != NULL) {
+				httpHeaderDelByName(&request->header,"accept-encoding");
+				httpHeaderPutStr(&request->header,HDR_ACCEPT_ENCODING,"deflate");
+			} else {
+				httpHeaderDelByName(&request->header,"accept-encoding");
+			}
+		}
+		stringClean(&val);
+	}
+
+	/*
+	 * Normalize Accept-Encoding Headers for video/image content
+	 */
+	char *mime_type = mimeGetContentType(http->uri);
+	if(mime_type) {
+		if(strstr(mime_type,"image") != NULL || strstr(mime_type,"video") != NULL || strstr(mime_type,"audio") != NULL) {
+			httpHeaderDelByName(&request->header,"accept-encoding");
+		}
+	}
+
+
 	/*
 	 * If we read past the end of this request, move the remaining
 	 * data to the beginning
Share and Enjoy: Digg Sphinn del.icio.us Facebook Mixx Google Bookmarks Design Float DZone Reddit Slashdot TwitThis

]]> http://letsgetdugg.com/2009/12/06/squid-headers-normalization-patch/feed/ 0 MogileFS Client on JRuby! http://letsgetdugg.com/2009/11/20/mogilefs-client-on-jruby/ http://letsgetdugg.com/2009/11/20/mogilefs-client-on-jruby/#comments Sat, 21 Nov 2009 04:19:11 +0000 victori http://letsgetdugg.com/?p=445 Are you running JRuby in production? Do you want distributed file storage for your “enterprise” application? Look no further, MogileFS is here.

MogileFS-Client has compatibility issues with JRuby due to it’s use of the low level Socket class. JRuby 1.5-dev does not yet support all the Socket methods, so here is a monkey patch to get the ruby mogilefs client working on JRuby. Yes it blocks, but who cares JRuby has native threads.

This is exactly why I love Ruby; monkey patching.

class Socket
  def self.mogilefs_new(host,port,timeout=5.0)
    TCPSocket.open(host,port,timeout)
  end
end
class TCPSocket
  attr_accessor :mogilefs_addr, :mogilefs_connected, :mogilefs_size, :mogilefs_tcp_cork
  def self.open(host,port,timeout = 5.0)
    super(host,port.to_i)
  end
  def readable?
    true
  end
  def write_nonblock(data)
    write(data)
  end
  def recv_nonblock(size,arg)
    recv(size,arg)
  end
  def mogilefs_init(host = nil, port = nil)
    true
  end
end

Here is an example test case on how to get it all to work.

require ‘rubygems’
require ‘mogilefs’

# jmogilefs.rb is the monkey patch above
# load it after loading mogilefs client.
require ‘jmogilefs.rb’

mg = MogileFS::MogileFS.new(:domain=>‘testserv’,:hosts=>[‘xxx.xxx.xxx.xxx:6001′])

p mg.get_file_data ‘video:100:default.jpg’

p mg.get_paths ‘video:100:default.jpg’,true

mg.list_keys(‘video:100′)[0].each do |f|
 p f
end

Share and Enjoy: Digg Sphinn del.icio.us Facebook Mixx Google Bookmarks Design Float DZone Reddit Slashdot TwitThis

]]> http://letsgetdugg.com/2009/11/20/mogilefs-client-on-jruby/feed/ 0 MogileFS on Solaris http://letsgetdugg.com/2009/11/02/mogilefs-on-solaris/ http://letsgetdugg.com/2009/11/02/mogilefs-on-solaris/#comments Mon, 02 Nov 2009 23:11:52 +0000 victori http://letsgetdugg.com/?p=407 *Update* Patches got accepted into MogileFS Trunk ;-)

Just go check out trunk, it has all my patches already included.

http://code.sixapart.com/svn/mogilefs/trunk/

The only thing you need is my mogstored disk patch which is still pending. All the issues revolving around postgresql and solaris have been already included in trunk.

I fixed a few issues with MogileFS and Solaris. MogileFS should run wonderfully on Solaris with my patches applied.

Directory for all my patches: http://victori.uploadbooth.com/patches

http://victori.uploadbooth.com/patches/solaris-disk-du.patch

This patch fixes mogstored to work with solaris’s df utility.

http://victori.uploadbooth.com/patches/store-max-requests.patch

This patch adds a new feature to the MogileFS Tracker – max_requests.

The default is 0, but it is suggested you set it to 1000 max_requests, to avoid memory leaks.

The tracker will give out the database handle up to the max_requests limit before expiring the connection for a new one. This avoids memory leaks with long running persistent connections. PostgreSQL has issues with long persistent connections, it accumulates a lot of ram and does not let go until the process/connection is killed off. This patch makes sure that the connection is expired after so many dbh handle requests.

http://victori.uploadbooth.com/patches/mogilefs-sunos-pg.patch

This patch applies the InactiveDestroy argument to avoid the MogileFS Tracker locking up with the PostgreSQL store on Solaris.

http://victori.uploadbooth.com/patches/solaris-mogilefs-full.patch

This is the full patch for all my fixes.

I am slowly migrating our fab40 static asset data to MogileFS. I have imported >300,000 images, no issues with my patches so far.

/ PLUG go make an account on uploadbooth!

Enjoy ;-)

Share and Enjoy: Digg Sphinn del.icio.us Facebook Mixx Google Bookmarks Design Float DZone Reddit Slashdot TwitThis

]]> http://letsgetdugg.com/2009/11/02/mogilefs-on-solaris/feed/ 1