Browsing the topic programming
Twitter4r Twitter::User patch fix
1 Comment | Filed under programming
(1 votes, average: 5.00 out of 5)
Loading ...I am working on a little twitter project that uses twitter4r as the client API. As of recently twitter pulled some strings on their API and broke compatibility.
/opt/local/lib/ruby/gems/1.8/gems/mbbx6spp-twitter4r-0.4.0/lib/twitter/client/base.rb:43:in `raise_rest_error’: Not Found (Twitter::RESTError)
from /opt/local/lib/ruby/gems/1.8/gems/mbbx6spp-twitter4r-0.4.0/lib/twitter/client/base.rb:48:in `handle_rest_response’
from /opt/local/lib/ruby/gems/1.8/gems/mbbx6spp-twitter4r-0.4.0/lib/twitter/client/base.rb:20:in `http_connect’
from /opt/local/lib/ruby/1.8/net/http.rb:543:in `start’
from /opt/local/lib/ruby/gems/1.8/gems/mbbx6spp-twitter4r-0.4.0/lib/twitter/client/base.rb:16:in `http_connect’
from /opt/local/lib/ruby/gems/1.8/gems/mbbx6spp-twitter4r-0.4.0/lib/twitter/client/user.rb:37:in `user’
from somebot.rb:5
Curse you twitter!
Luckly Ruby has the concept of monkey patching, here is the fix to get it all working correctly.
@@USER_URIS = {
:info => ‘/users/show.json’,
:friends => ‘/statuses/friends.json’,
:followers => ‘/statuses/followers.json’,
}
end
Shazzam… it works…
Scala memcached implementation
Leave a comment | Filed under open source programming
(4 votes, average: 3.50 out of 5) Loading ...I finally got around to open sourcing our scala memcached implementation that we use at fabulously40 for session storage.
Since wicket sessions can vary greatly in size, using the standard memcached server implementation became impractical due to the slab allocator.
The current code on github lacks the ehcache store and an Actor IoHandler adapter. The internal SMemcached application at fabulously40 uses a private caching API so we can hook up various caching backend storage implementations such as mysql, postgresql, ehcache or even another memcached server. You can grab the TCache project on github that SMemcached uses to unify caching under a single API. This gives SMemcached a lot of flexibility when it comes to caching your data.
fyi. TCache stands for “Tanek” Cache, Tanek means cache in russian.
The project works quite well, but don’t use it in production just yet since there is no data expiration for cached data in the HashMap storage implementation. This is just a technical preview. Do use it in production, this is what we use at Fabulously40 
I would be sitting on a gold mine.
SLOC Directory SLOC-by-Language (Sorted) 37890 src java=37890 5026 contrib-utilities java=5026 4457 contrib-crud java=4457 2259 contrib-mootools java=2259 1235 contrib-generaldao java=1235 1185 contrib-emailmanager java=1185 986 jetty-memcache java=986 961 contrib-cache java=961 787 jmemcached java=786,sh=1 640 contrib-thumbnail java=640 503 contrib-blueprint java=503 181 contrib-snapshot java=181 32 contrib-nicedit java=32 Totals grouped by language (dominant language first): java: 56141 (100.00%) sh: 1 (0.00%) Total Physical Source Lines of Code (SLOC) = 56,142 Development Effort Estimate, Person-Years (Person-Months) = 13.73 (164.80) (Basic COCOMO model, Person-Months = 2.4 * (KSLOC**1.05)) Schedule Estimate, Years (Months) = 1.45 (17.39) (Basic COCOMO model, Months = 2.5 * (person-months**0.38)) Estimated Average Number of Developers (Effort/Schedule) = 9.47 Total Estimated Cost to Develop = $ 1,855,214 (average salary = $56,286/year, overhead = 2.40). SLOCCount, Copyright (C) 2001-2004 David A. Wheeler SLOCCount is Open Source Software/Free Software, licensed under the GNU GPL. SLOCCount comes with ABSOLUTELY NO WARRANTY, and you are welcome to redistribute it under certain conditions as specified by the GNU GPL license; see the documentation for details. Please credit this data as "generated using David A. Wheeler's 'SLOCCount'."
Wicket Stateless pagination
3 Comments | Filed under programming (7 votes, average: 5.00 out of 5) Loading ...One of the first few things I have tackled while creating fabulously40 was stateless pagination. I wanted to keep all the public facing pages stateless to avoid session overhead. Wicket makes pagination brain dead simple at the cost of session use. I wanted to keep the simple programming model yet be completely stateless. Here is my solution to the problem.
Wicket a stateful framework, is actually really good at being stateless.
Example code on how to use it.
update: I would suggest using MixedParamHybridCodingStrategy with the stateless paging navigation to keep links clean.
add(dv = new StatelessDataView<Photo>("pics", getPhotoProvider(), amount, pageParams) {
private static final long serialVersionUID = 1L;
@Override
protected void populateItem(final Item<Photo> arg0) {
}
});
// call back page class, page paramaters passed, the dataview and 12 pagination links.
add(new StatelessSimplePagingNavigator("nav", PhotosPage.class, pageParams, dataView, 12));
The dataview
import org.apache.wicket.PageParameters;
import org.apache.wicket.markup.repeater.data.DataView;
import org.apache.wicket.markup.repeater.data.IDataProvider;
public abstract class StatelessDataView<T> extends DataView<T> {
private static final long serialVersionUID = 1L;
private PageParameters pp;
@Override
protected boolean getStatelessHint() {
return true;
}
protected int getPageNumber(final String param) {
String numResult = param;
if(numResult.contains(".wicket-")) {
numResult = numResult.substring(0,numResult.indexOf(".wicket-"));
}
return Integer.valueOf(numResult);
}
public StatelessDataView(final String id, final IDataProvider dataProvider, final PageParameters pp) {
super(id, dataProvider);
this.pp = pp;
if (pp.getString(id) != null) {
int pageNum = getPageNumber(pp.getString(id));
if(pageNum != -1 && pageNum >= 0 && pageNum <= getPageCount()) {
setCurrentPage(getPageNumber(pp.getString(id)));
}
}
}
public StatelessDataView(final String id, final IDataProvider dataProvider, final int itemsPerPage,
final PageParameters pp) {
super(id, dataProvider, itemsPerPage);
this.pp = pp;
if (pp.getString(id) != null) {
int pageNum = getPageNumber(pp.getString(id));
if(pageNum != -1 && pageNum >= 0 && pageNum <= getPageCount()) {
setCurrentPage(getPageNumber(pp.getString(id)));
}
}
}
}
The paging navigator
import org.apache.wicket.Component;
import org.apache.wicket.PageParameters;
import org.apache.wicket.markup.ComponentTag;
import org.apache.wicket.markup.html.link.BookmarkablePageLink;
import org.apache.wicket.markup.html.link.Link;
import org.apache.wicket.markup.html.navigation.paging.IPageable;
import org.apache.wicket.markup.html.navigation.paging.IPagingLabelProvider;
import org.apache.wicket.markup.html.navigation.paging.PagingNavigation;
public class StatelessSimplePagingNavigator extends SimplePagingNavigator {
private static final long serialVersionUID = 1L;
private Class clazz;
private PageParameters pp;
private int current_page;
private int page_count;
private String pageable_id;
private String anchor = null;
@Override
protected boolean getStatelessHint() {
return true;
}
public StatelessSimplePagingNavigator(final String id, final Class clazz, final PageParameters pp,
final IPageable pageable, final int viewsize) {
this(id, clazz, pp, pageable, viewsize, false);
}
public StatelessSimplePagingNavigator(final String id, final Class clazz, final PageParameters pp,
final IPageable pageable, final int viewsize, final boolean anchorSelf) {
super(id, pageable, viewsize, anchorSelf);
this.clazz = clazz;
this.pp = pp;
this.current_page = pageable.getCurrentPage();
this.page_count = pageable.getPageCount();
this.pageable_id = ((Component) pageable).getId();
}
public StatelessSimplePagingNavigator(final String id, final Class clazz, final PageParameters pp,
final IPageable pageable, final int viewsize, final String anchor) {
super(id, pageable, viewsize, false);
this.clazz = clazz;
this.pp = pp;
this.current_page = pageable.getCurrentPage();
this.page_count = pageable.getPageCount();
this.pageable_id = ((Component) pageable).getId();
this.anchor = anchor;
}
@Override
protected void onBeforeRender() {
super.onBeforeRender();
addOrReplace(newStatelessPagingNavigationLink("next", pageable_id, current_page, 1));
addOrReplace(newStatelessPagingNavigationLink("prev", pageable_id, current_page, -1));
}
@Override
protected PagingNavigation newNavigation(final IPageable pageable, final IPagingLabelProvider labelProvider) {
PagingNavigation pg = new PagingNavigation("navigation", pageable, labelProvider) {
private static final long serialVersionUID = 1L;
@Override
protected Link newPagingNavigationLink(final String id, final IPageable pageable, final int pageIndex) {
Component c = (Component) pageable;
// PageParameters p1 = new PageParameters();
// p1.putAll(pp);
pp.put(c.getId(), String.valueOf(pageIndex));
BookmarkablePageLink<Object> lnk = new BookmarkablePageLink<Object>(id, clazz, pp) {
private static final long serialVersionUID = 1L;
@Override
protected void onComponentTag(final ComponentTag arg0) {
super.onComponentTag(arg0);
if (anchor != null) {
if(arg0.getString("href") != null) {
String href = arg0.getString("href").toString();
String atag = anchor.contains("#") ? anchor : "#" + anchor;
arg0.put("href", href + atag);
}
}
}
@Override
public boolean isEnabled() {
return (current_page != pageIndex);
}
};
if (isAnchorSelf()) {
lnk.setAnchor(StatelessSimplePagingNavigator.this);
}
return lnk;
}
};
pg.setViewSize(getViewsize());
return pg;
}
@Override
protected Link<Object> newPagingNavigationIncrementLink(final String id, final IPageable pageable, final int increment) {
Component c = (Component) pageable;
final int page = Integer.valueOf(pp.getString(c.getId()));
pp.put(c.getId(), String.valueOf(page + increment));
return new BookmarkablePageLink<Object>(id, clazz, pp) {
private static final long serialVersionUID = 1L;
@Override
public boolean isVisible() {
return (page + increment) >= 0;
}
};
}
protected Link<Object> newStatelessPagingNavigationLink(final String id, final String pageableId, final int currentPage,
final int increment) {
// PageParameters p1 = new PageParameters();
// p1.putAll(pp);
pp.put(pageableId, String.valueOf(currentPage + increment));
return new BookmarkablePageLink<Object>(id, clazz, pp) {
private static final long serialVersionUID = 1L;
@Override
public boolean isVisible() {
return (currentPage + increment) < page_count && (currentPage + increment) >= 0;
}
};
}
@Override
protected Link<Object> newPagingNavigationLink(final String id, final IPageable pageable, final int pageNumber) {
Component c = (Component) pageable;
// PageParameters p1 = new PageParameters();
// p1.putAll(pp);
pp.put(c.getId(), String.valueOf(pageNumber));
return new BookmarkablePageLink(id, clazz, pp);
}
}
Which inherits from
import org.apache.wicket.markup.html.link.Link;
import org.apache.wicket.markup.html.navigation.paging.IPageable;
import org.apache.wicket.markup.html.navigation.paging.IPagingLabelProvider;
import org.apache.wicket.markup.html.navigation.paging.PagingNavigation;
import org.apache.wicket.markup.html.navigation.paging.PagingNavigator;
public class SimplePagingNavigator extends PagingNavigator {
private static final long serialVersionUID = 1L;
private int viewsize = 0;
private boolean anchorSelf = false;
public SimplePagingNavigator(final String id, final IPageable pageable, final int viewsize) {
this(id, pageable, viewsize, false);
}
public SimplePagingNavigator(final String id, final IPageable pageable, final int viewsize, final boolean anchorSelf) {
super(id, pageable);
setOutputMarkupId(true);
this.setViewsize(viewsize);
this.setAnchorSelf(anchorSelf);
}
@Override
protected void onBeforeRender() {
if (get("navigation") != null) {
remove("navigation");
}
if (get("prev") != null) {
remove("prev");
}
if (get("next") != null) {
remove("next");
}
super.onBeforeRender();
if (get("first") != null) {
remove("first");
}
if (get("last") != null) {
remove("last");
}
if (getViewsize() != 0) {
getPagingNavigation().setViewSize(getViewsize());
}
}
@Override
protected PagingNavigation newNavigation(final IPageable pageable, final IPagingLabelProvider labelProvider) {
PagingNavigation pg = new PagingNavigation("navigation", pageable, labelProvider) {
private static final long serialVersionUID = 1L;
@Override
protected Link newPagingNavigationLink(final String id, final IPageable pageable, final int pageIndex) {
Link lnk = (Link) super.newPagingNavigationLink(id, pageable, pageIndex);
if (isAnchorSelf()) {
lnk.setAnchor(SimplePagingNavigator.this);
}
return lnk;
}
};
pg.setViewSize(getViewsize());
return pg;
}
@Override
public boolean isVisible() {
if (getPageable() != null) {
return (getPageable().getPageCount() > 1);
}
return true;
}
public void setAnchorSelf(final boolean anchorSelf) {
this.anchorSelf = anchorSelf;
}
public boolean isAnchorSelf() {
return anchorSelf;
}
public void setViewsize(final int viewsize) {
this.viewsize = viewsize;
}
public int getViewsize() {
return viewsize;
}
}
Wicket MixedParamHybridUrlCodingStrategy
1 Comment | Filed under programming (2 votes, average: 5.00 out of 5) Loading ...I am about to run out the house for the Labor Day weekend, quickie post on my new coding url strategy.
MixedParamHybridUrlCodingStrategy lets you keep stateful multi-pagemap URLs clean while using mixed parameters.
Example…
This will mount “/questions/stupid-category” and convert it to…
import java.util.HashSet;
import java.util.Iterator;
import java.util.Map;
import java.util.Set;
import org.apache.wicket.IRequestTarget;
import org.apache.wicket.PageParameters;
import org.apache.wicket.protocol.http.request.WebRequestCodingStrategy;
import org.apache.wicket.request.target.coding.HybridUrlCodingStrategy;
import org.apache.wicket.util.string.AppendingStringBuffer;
import org.apache.wicket.util.value.ValueMap;
public class MixedParamHybridUrlCodingStrategy extends HybridUrlCodingStrategy {
private final String[] parameterNames;
public MixedParamHybridUrlCodingStrategy(final String mountPath, final Class pageClass) {
super(mountPath, pageClass);
this.parameterNames = new String[] {};
}
public MixedParamHybridUrlCodingStrategy(final String mountPath, final Class pageClass,final boolean refresh) {
super(mountPath,pageClass,refresh);
this.parameterNames = new String[] {};
}
public MixedParamHybridUrlCodingStrategy(final String mountPath, final Class pageClass,final String[] params) {
super(mountPath,pageClass,false);
this.parameterNames = params;
}
public MixedParamHybridUrlCodingStrategy(final String mountPath, final Class pageClass,final boolean refresh,final String[] params) {
super(mountPath,pageClass,refresh);
this.parameterNames = params;
}
@Override
protected void appendParameters(final AppendingStringBuffer url, final Map parameters)
{
Set parameterNamesToAdd = new HashSet(parameters.keySet());
// Find index of last specified parameter
boolean foundParameter = false;
int lastSpecifiedParameter = parameterNames.length;
while (lastSpecifiedParameter != 0 && !foundParameter)
{
foundParameter = parameters.containsKey(parameterNames[–lastSpecifiedParameter]);
}
if (foundParameter)
{
for (int i = 0; i <= lastSpecifiedParameter; i++)
{
String parameterName = parameterNames[i];
final Object param = parameters.get(parameterName);
String value = param instanceof String[] ? ((String[])param)[0] : (String)param;
if (value == null)
{
value = "";
}
if (!url.endsWith("/"))
{
url.append("/");
}
url.append(urlEncodePathComponent(value));
parameterNamesToAdd.remove(parameterName);
}
}
if (!parameterNamesToAdd.isEmpty())
{
boolean first = true;
final Iterator iterator = parameterNamesToAdd.iterator();
while (iterator.hasNext())
{
url.append(first ? ‘?’ : ‘&’);
String parameterName = (String)iterator.next();
final Object param = parameters.get(parameterName);
String value = param instanceof String[] ? ((String[])param)[0] : (String)param;
url.append(urlEncodeQueryComponent(parameterName)).append("=").append(
urlEncodeQueryComponent(value));
first = false;
}
}
String pageMap = (String)parameters.get(WebRequestCodingStrategy.PAGEMAP);
if (pageMap != null)
{
pageMap = WebRequestCodingStrategy.encodePageMapName(pageMap);
if (!url.endsWith("/"))
{
url.append("/");
}
url.append(WebRequestCodingStrategy.PAGEMAP).append("/").append(
urlEncodePathComponent(pageMap));
}
}
@Override
public CharSequence encode(final IRequestTarget requestTarget) {
String url = String.valueOf(super.encode(requestTarget));
if(url.contains(".wicket")) {
// Rewrite URL from.. /foo?bar=5.wicket-xxx to /foo.wicket-xxx?bar=5
return url.replaceAll("(.*)\\?(.*)\\.(wicket-[0-9]+)$", "$1.$3?$2");
}
return url;
}
@Override
protected ValueMap decodeParameters(final String urlFragment, final Map urlParameters)
{
PageParameters params = new PageParameters();
if(urlFragment == null) {
return params;
}
// Add all url parameters and normalize
for(Iterator<Map.Entry> itr = urlParameters.entrySet().iterator();itr.hasNext();) {
Map.Entry<String,String[]> e = itr.next();
if(e.getValue() != null) {
String val = (e.getValue())[0];
if(val != null) {
if(val.contains(".wicket")) {
val = val.substring(0,val.indexOf(".wicket"));
urlParameters.put(e.getKey(), val);
}
}
}
}
params.putAll(urlParameters);
String urlPath = urlFragment;
if (urlPath.startsWith("/"))
{
urlPath = urlPath.substring(1);
}
if (urlPath.length() > 0)
{
String[] pathParts = urlPath.split("/");
if (pathParts.length > parameterNames.length)
{
throw new IllegalArgumentException(
"Too many path parts, please provide sufficient number of path parameter names");
}
for (int i = 0; i < pathParts.length; i++)
{
if(pathParts[i].contains(".wicket")) {
pathParts[i].substring(0,pathParts[i].indexOf(".wicket"));
}
if (WebRequestCodingStrategy.PAGEMAP.equals(pathParts[i]))
{
params.put(WebRequestCodingStrategy.PAGEMAP,
WebRequestCodingStrategy.decodePageMapName(urlDecodePathComponent(pathParts[i])));
}
params.put(parameterNames[i], urlDecodePathComponent(pathParts[i]));
}
}
return params;
}
}
GrizzlyConnector patch for Jetty to work with QueuedThreadPool
1 Comment | Filed under administration programming (No Ratings Yet) Loading ...This is a late night post, so I am just going to make it short. This patch lets you use QueuedThreadPool with the Grizzly Connector. This is a monkey patch, getMaxThreads() should be moved up into the Thread Interface.
--- GrizzlyConnection-old.java Sat May 2 01:08:02 2009 +++ GrizzlyConnector.java Sat May 2 00:56:37 2009 @@ -51,6 +51,8 @@ import org.mortbay.jetty.webapp.WebAppContext; import org.mortbay.log.Log; import org.mortbay.thread.BoundedThreadPool; +import org.mortbay.thread.QueuedThreadPool; +import org.mortbay.thread.ThreadPool; /* ------------------------------------------------------------------------------- */ /** @@ -178,8 +180,13 @@ controller.setProtocolChainInstanceHandler(instanceHandler); Pipeline pipeline = new DefaultPipeline(); - pipeline.setMaxThreads( - ((BoundedThreadPool)getServer().getThreadPool()).getMaxThreads()); + if(getServer().getThreadPool() instanceof BoundedThreadPool) { + pipeline.setMaxThreads( + ((BoundedThreadPool)getServer().getThreadPool()).getMaxThreads()); + } else { + pipeline.setMaxThreads( + ((QueuedThreadPool)getServer().getThreadPool()).getMaxThreads()); + } controller.setPipeline(pipeline); }
You might find this plugin nifty if you have multiple application servers processing requests. The Perlbal BackendHeaders plugin appends X-Backend headers with which backend served the request.
Update 06/26/09 Now on github perlbal-plugin-backendheaders
syris:~ victori$ curl -I http://fabulously40.com/questions HTTP/1.1 200 OK Server: nginx/0.7.52 Content-Type: text/html; charset=utf-8 Expires: Thu, 01 Jan 1970 00:00:00 GMT Content-Language: en X-Backend: 72.11.142.91:8880 X-Dilbert: If you have any trouble sounding condescending, find a Unix user to show you how it's done Content-Length: 48046
use Perlbal;
use strict;
use warnings;
#
# Add $self->{service}->run_hook(‘modify_response_headers’, $self);
# To sub handle_response in BackendHTTP after Content-Length is set.
#
# LOAD BackendHeaders
# SET plugins = backendheaders
sub load {
my $class = shift;
return 1;
}
sub unload {
my $class = shift;
return 1;
}
# called when we’re being added to a service
sub register {
my ( $class, $svc ) = @_;
my $modify_response_headers_hook = sub {
my Perlbal::BackendHTTP $be = shift;
my Perlbal::HTTPHeaders $hds = $be->{res_headers};
my Perlbal::Service $svc = $be->{service};
return 0 unless defined $hds && defined $svc;
$hds->header( ‘X-Backend’, $be->{ipport} );
return 0;
};
$svc->register_hook( ‘BackendHeaders’, ‘modify_response_headers’,
$modify_response_headers_hook );
return 1;
}
# called when we’re no longer active on a service
sub unregister {
my ( $class, $svc ) = @_;
$svc->unregister_hooks(‘BackendHeaders’);
$svc->unregister_setters(‘BackendHeaders’);
return 1;
}
1;
Try this fun perl benchmark, to test your dual core, SMP or hyperthreaded system.
Before running, make sure you have perl 5.8 with threading support compiled in.
Perl has native ithreads as of perl 5.8.
use threads;
use strict;
my $y1=Bench->new();
print "Bencmarking multi-threadedn";
$y1->benchmark();
print "Benchmarking single-threadedn";
$y1->ncpu(1);
$y1->benchmark();
package Bench;
sub new ()
{
my $self = {result => 0,ncpu=>0};
my $cpus =`sysctl hw.ncpu`;
$cpus =~ /: (.*)/g;
$self->{ncpu}=$1;
my $class = shift;
bless ($self,$class);
return $self;
}
sub ncpu {
my ($self,$num) = @_;
if(defined $num) { $self->{ncpu}=$num; } else { return $self->{ncpu}; }
}
sub benchmark ()
{
my ($self)=@_;
my @thr;
for(my $i=0;$i < $self->{ncpu};$i++)
{
print "Starting thread $in";
push @thr, threads->create(‘benchmark_thread’);
}
my $total=0;
foreach my $t (@thr)
{
$total=$total+$t->join();
}
print "Total number of insane floating point divisions in 10 seconds is ". $total . "n";
}
sub benchmark_thread()
{
my ($y,$x)=0;
my $time1 = time();
my ($self)=@_;
while(1){
#$time2 = time();
if((time() - $time1)>= 10){last;}
else {
$x=1.00/24000000000.001;
$y++;
}
}
return $y;
}
Finding Circular Reference leaks in Perl
5 Comments | Filed under programming (4 votes, average: 3.75 out of 5) Loading ...Dynamic type languages such as Perl, Ruby, PHP, and Python free you as the developer from managing memory in your application. However, it isn’t a fool proof solution that you won’t have memory leaks in your application. You as the developer should be aware of how the underlying garbage collector of your preferred language works to accommodate for the inadequacies of its garbage collection algorithm.
Currently there are two ways of doing garbage collection; mark and sweep and reference counting. The Perl interpreter uses the latter. Reference counting is a fairly simple garbage collection technique. Each time you declare an instance, the reference count increments by one. When your program reaches the end of scope, objects with a reference of one get collected. However, if your object has a reference count of two it is kept. The one main draw back of reference counting is the fact it can’t deal with circular references. This is when both objects point to each other and they never get garbage collected.
On the other hand, Ruby and Java use the mark and sweep garbage collector. I personally have mixed feelings about it, since I don’t know exactly when my objects will be collected. The way mark and sweep garbage collection works, is it does not collect anything for a period of time. At intervals when the heap gets full, it runs its garbage collection. The downside to this is you don’t know exactly know when this happens and if there are lots of objects to be collected this leads to “stutters” and unresponsiveness of the application. If you have ever used a Java swing application you might have noticed these stutters, this is when garbage collection is taking place. However, it’s not as gloomy as I set the pretense to be with the mark and sweep garbage collection. Mark and sweep garbage collection can handle cyclic references unlike with reference counting, which is a huge boon to its usefulness. There has been much work done on mark and sweep garbage collection, specifically with generational mark and sweep collectors that try to fix the unresponsiveness issue. Java currently uses a generation GC, and Ruby hopes to obtain a generational GC for the Ruby 2.0 interpreter. Ideally a generational garbage collector would be the preferred GC for a long-standing process.
With that little garbage collection background out of the way, lets look at the life cycle of a instance in reference counting garbage collector.
Here is an example of how reference counting works ideally:
my $i = 5;
$i + 5;
print $i . ‘n‘;
} # $i should be garbage collected when it goes out of scope.
Unlike mark and sweep garbage collection with reference counting, you know exactly when your instance gets collected.
Here is a very simple problematic case for reference counting:
my $a;
my $b;
$a->{b} = $b;
$b->{a} = $a;
} # since both are pointing to each other they will never get collected.
This is a fairly simple case of where reference counting falls right on its face. Usually this isn’t a problem since most Perl scripting revolves around short-lived scripts. However, with frameworks such as Catalyst that are long running perl scripts this becomes an issue quickly. Thankfully, with Perl it is extremely easily to nail memory leaks, more so than with Ruby or Java. Enter Devel::Cycle and Devel::Peek, both of these modules can be installed from cpan. Both Devel::Cycle and Devel::Peek can assist you in tracking down the memory leak in a relatively short time.
use Devel::Peek;
foreach (1) {
my $parent = {name => ‘victor’ };
my $child = {name => ‘victor jr’ };
$parent->{child} = $child;
$child->{parent} = $parent;
find_cycle($parent);
# find_cycle belongs to Devel::Cycle
# which prints out the
# circular reference to STDOUT
Dump($parent);
# Dump belongs to Devel::Peek , its extra verbose
# which prints out the reference count to STDOUT
}
# Sample output # ibook:~/Desktop victori$ perl blah.pl # Cycle (1): <-- find_cycle tells you literly where the cyclic reference leak is at. # $A->{'child'} => %B # $B->{'parent'} => %A # # SV = RV(0x1817898) at 0x1800ec8 # REFCNT = 1 # FLAGS = (PADBUSY,PADMY,ROK) # RV = 0x18006dc # SV = PVHV(0x1830980) at 0x18006dc # REFCNT = 2 <-- Notice the reference count of 2 , we know we have a leak # FLAGS = (SHAREKEYS) # IV = 2 # NV = 0 # ARRAY = 0x404e60 (0:6, 1:2) # hash quality = 125.0% # KEYS = 2 # FILL = 2 # MAX = 7 # RITER = -1 # EITER = 0x0 # Elt "name" HASH = 0xe6e17f14 # SV = PV(0x1801460) at 0x1800ea4 # REFCNT = 1 # FLAGS = (POK,pPOK) # PV = 0x401730 "victor" # CUR = 6 # LEN = 8 # Elt "child" HASH = 0x33ec6b5 # SV = RV(0x1817870) at 0x1832ca4 # REFCNT = 1 # FLAGS = (ROK) # RV = 0x1800484 # SV = PVHV(0x18309b0) at 0x1800484 # REFCNT = 2 # FLAGS = (SHAREKEYS) # IV = 2 # NV = 0 # ARRAY = 0x404db0 (0:6, 1:2) # hash quality = 125.0% # KEYS = 2 # FILL = 2 # MAX = 7 # RITER = -1 # EITER = 0x0 # Elt "parent" HASH = 0xa99c4651 # SV = RV(0x18178a0) at 0x1832c44 # REFCNT = 1 # FLAGS = (ROK) # RV = 0x18006dc
So how do we fix this? Quite simple, all we do is weaken the reference count using weaken(). Here is a proper way of patching up the memory leak we introduced in our program.
use Devel::Peek;
use Scalar::Util qw/weaken/;
foreach (1) {
my $parent = {name => ‘victor’ };
my $child = {name => ‘victor jr’ };
weaken($parent->{child} = $child);
# we weaken the reference at the parent and all is well.
$child->{parent} = $parent;
find_cycle($parent);
# find_cycle belongs to Devel::Cycle which prints out the
# circular reference to STDOUT
Dump($parent);
# Dump belongs to Devel::Peek , its extra verbose
# which prints out the reference count to STDOUT
}
We weaken the reference at the parent level to set it back to a reference count of 1, so when it reaches the end of scope it will be collected and the memory leak will be no more.
Hopefully this is a good primer for other Perl coders out there who are facing memory leaks in their running long running perl scripts.
Perlbal Sticky Sessions
3 Comments | Filed under programming (1 votes, average: 5.00 out of 5) Loading ...I *really* needed session affinity for our wicket application. HAproxy does session affinity but can’t be reconfigured at runtime without a restart. Perlbal is much more configurable, it lets you add and remove nodes in a pool at runtime. This makes deploying a new version of our web application a lot easier. I have the ability to test a new version of our application before putting it back into the pool of active nodes.
This is my first attempt at writing a sticky sessions plugin for Perlbal.
Update 06/26/09 Now on github perlbal-plugin-stickysessions
Update 04/30/09 Added Perlbal::XS::HTTPHeaders Support. Faster header parsing performance.
Update Fixed the Set-Cookies merge bug with the way Perlbal handles headers.
use Perlbal;
use strict;
use warnings;
use Data::Dumper;
use HTTP::Date;
use CGI qw/:standard/;
use CGI::Cookie;
use Scalar::Util qw(blessed reftype);
# LOAD StickySessions
# SET plugins = stickysessions
#
# Add $self->{service}->run_hook(‘modify_response_headers’, $self);
# To sub handle_response in BackendHTTP after Content-Length is set.
#
sub load {
my $class = shift;
return 1;
}
sub unload {
my $class = shift;
return 1;
}
sub get_backend_id {
my $be = shift;
for ( my $i = 0 ; $i <= $#{ $be->{ service }->{ pool }->{ nodes } } ; $i++ )
{
my ( $nip, $nport ) = @{ $be->{ service }->{ pool }->{ nodes }[$i] };
my $nipport = $nip . ‘:’ . $nport;
if ( $nipport eq $be->{ ipport } ) {
return $i + 1;
}
}
# default to the first backend in the node list.
return 1;
}
sub decode_server_id {
my $id = shift;
return ( $id - 1 );
}
sub get_ipport {
my ( $svc, $req ) = @_;
my $cookie = $req->header(‘Cookie’);
my %cookies = ();
my $ipport = undef;
%cookies = parse CGI::Cookie($cookie) if defined $cookie;
if ( defined $cookie && defined $cookies{ ‘X-SERVERID’ } ) {
my $val =
$svc->{ pool }
->{ nodes }[ decode_server_id( $cookies{ ‘X-SERVERID’ }->value ) ];
my ( $ip, $port ) = @{ $val } if defined $val;
$ipport = $ip . ‘:’ . $port;
}
return $ipport;
}
sub find_or_get_new_backend {
my ( $svc, $req, $client ) = @_;
my Perlbal::BackendHTTP $be;
my $ipport = get_ipport( $svc, $req );
my $now = time;
while ( $be = shift @{ $svc->{ bored_backends } } ) {
next if $be->{ closed };
# now make sure that it’s still in our pool, and if not, close it
next unless $svc->verify_generation($be);
# don’t use connect-ahead connections when we haven’t
# verified we have their attention
if ( !$be->{ has_attention } && $be->{ create_time } < $now - 5 ) {
$be->close("too_old_bored");
next;
}
# don’t use keep-alive connections if we know the server’s
# just about to kill the connection for being idle
if ( $be->{ disconnect_at } && $now + 2 > $be->{ disconnect_at } ) {
$be->close("too_close_disconnect");
next;
}
# give the backend this client
if ( defined $ipport ) {
if ( $be->{ ipport } eq $ipport ) {
if ( $be->assign_client($client) ) {
$svc->spawn_backends;
return 1;
}
}
} else {
if ( $be->assign_client($client) ) {
$svc->spawn_backends;
return 1;
}
}
# assign client can end up closing the connection, so check for that
return 1 if $client->{ closed };
}
return 0;
}
# called when we’re being added to a service
sub register {
my ( $class, $gsvc ) = @_;
my $check_cookie_hook = sub {
my Perlbal::ClientProxy $client = shift;
my Perlbal::HTTPHeaders $req = $client->{ req_headers };
return 0 unless defined $req;
my $svc = $client->{ service };
# we define were to send the client request
$client->{ backend_requested } = 1;
$client->state(‘wait_backend’);
return unless $client && !$client->{ closed };
if ( find_or_get_new_backend( $svc, $req, $client ) != 1 ) {
push @{ $svc->{ waiting_clients } }, $client;
$svc->{ waiting_client_count }++;
$svc->{ waiting_client_map }{ $client->{ fd } } = 1;
my $ipport = get_ipport( $svc, $req );
if ( defined($ipport) ) {
my ( $ip, $port ) = split( /\:/, $ipport );
$svc->{ spawn_lock } = 1;
my $be =
Perlbal::BackendHTTP->new( $svc, $ip, $port,
{ pool => $svc->{ pool } } );
$svc->{ spawn_lock } = 0;
} else {
$svc->spawn_backends;
}
$client->tcp_cork(1);
}
return 0;
};
my $set_cookie_hook = sub {
my Perlbal::BackendHTTP $be = shift;
my Perlbal::HTTPHeaders $hds = $be->{ res_headers };
my Perlbal::HTTPHeaders $req = $be->{ req_headers };
return 0 unless defined $be && defined $hds;
my $svc = $be->{ service };
my $cookie = $req->header(‘Cookie’);
my %cookies = ();
%cookies = parse CGI::Cookie($cookie) if defined $cookie;
my $backend_id = get_backend_id($be);
if ( !defined( $cookies{ ‘X-SERVERID’ } )
|| $cookies{ ‘X-SERVERID’ }->value != $backend_id )
{
my $backend_cookie =
new CGI::Cookie( -name => ‘X-SERVERID’, -value => $backend_id );
if ( defined $hds->header(‘set-cookie’) ) {
my $val = $hds->header(‘set-cookie’);
$hds->header( ‘Set-Cookie’,
$val .= "\r\nSet-Cookie: " . $backend_cookie->as_string );
} else {
$hds->header( ‘Set-Cookie’, $backend_cookie );
}
}
return 0;
};
$gsvc->register_hook( ‘StickySessions’, ‘start_proxy_request’,
$check_cookie_hook );
$gsvc->register_hook( ‘StickySessions’, ‘modify_response_headers’,
$set_cookie_hook );
return 1;
}
# called when we’re no longer active on a service
sub unregister {
my ( $class, $svc ) = @_;
$svc->unregister_hooks(‘StickySessions’);
$svc->unregister_setters(‘StickySessions’);
return 1;
}
1;
