Br0kenB1ts - Yan's blog

The future of glu - glu in the cloud

2011-09-30T00:00:00-07:00

I have started thinking of where I would like to see glu go in the future (3 to 6 months timeframe). It seems to me that what makes the most sense for the evolution of glu is to embrace the cloud.

glu today…

Today glu works really well (at least I assume :) when the glu environment is setup: machines are up and running with a glu agent on it, ZooKeeper is installed, the console is up and running. The tutorial that comes with glu has streamlined this effort for a quick and easy demonstration when everything runs locally on the same machine. The production setup is a bit more involved and could also be made more streamlined (there are several tickets tracking this like #58 and #84).

In the end, glu is a deployment automation platform which allows you to automate the deployment of your applications and services on a set of nodes. The missing piece is that glu does not know how to provision itself. One of the challenge of course is that in order to provision glu, first you need a node. In many cases (and this is the case at LinkedIn), this step involves an actual human being who is going to rack a physical piece of hardware, then install the OS on it, etc…

One of the key concept of glu is that it turned provisioning into an API. Since it is an API, you can programmatically call it (which means you can programmatically provision a node) and of course you can automate it.

In my mind, one of the biggest gain that the cloud is bringing, is the fact that you can now treat hardware as software: you now have an API that allows you to bring machines up and down. This is a big mind shift. The beauty of it is since it is an API you can automate it!

glu in the cloud

At the heart of glu relies the model which defines what applications (services) need to run where. The where part is already pretty abstract but glu assumes that it is a node that is already glu aware.

Can we remove this assumption? After all, provisioning your applications is simply a matter of calling some API, and in the cloud, provisioning glu is also simply a matter of calling some API (albeit a very different one).

In this scenario we could envision that the model defines nodes in a more abstract way, for example: ec2:standard:XL/agent-1, which essentially defines agent-1 to be running on an extra large standard EC2 instance.

The same way that glu computes a delta to provision your applications, it could also determine that the node agent-1 does not exist and as a result, needs to be provisioned with glu, in which case it would use the EC2 apis to create a brand new VM and install what is necessary on the node for glu to run.

Mind map

In order to bootstrap the process I have created a mind map to capture my first thoughts (you can click the image to view the full version).

My dream

Currently I deploy my personal website kiwidoc on rackspace. The process is manual (this was done prior to glu and I did not port it). My dream is that I can use glu to deploy kiwidoc. And this is what I really want to be able to do: rackspace is having technical difficulties today… not a problem let’s switch to ec2 while it gets fixed. Dreaming is good :)

Joke aside, kiwidoc would be a fairly interesting use case. Although very small there are already a bunch of ‘issues’ that glu in the cloud would need to address. For example, kiwidoc is split amongst 2 machines (a nginx frontend and a tomcat backend). The frontend needs to talk to the backend and for that it needs to use the ‘internal’ IP so that it not counted as external traffic. Also kiwidoc uses data (which is not code), like the search (Lucene) index. What happens if the node hosting the backend (and hence the index) needs to be recreated?

Feedback

In the end, the community is what makes glu both successful and interesting. I am soliciting feedback. Please leave some. Even if it is just a simple idea or comment, it doesn’t matter. I would really like this process to be collaborative. I would love to be able to derive a roadmap and start implementing it. Feel free to leave feedback on the forum.

java.io.File.deleteOnExit() is evil

2011-05-17T00:00:00-07:00

If you take a look at the javadoc for File.deleteOnExit() it clearly states:

Once deletion has been requested, it is not possible to cancel the request. This method should therefore be used with care.

It is relatively easy to overlook this kind of statement. In my case it led to a slow memory leak that was reported in glu. In the end, finding out the memory leak turned out to be relatively simple thanks to jmap and YourKit (although it seems to always be simple after the fact :)).

The piece of (groovy) code that was causing the issue was the following:

  static String cat(location) throws IOException
  {
    File tempFile = File.createTempFile("GroovyIOUtils.cat", ".txt")

    tempFile.deleteOnExit()

    try
    {
      fetchContent(location, tempFile)
      return tempFile.text
    }
    finally
    {
      tempFile.delete()
    }
  }

What the code does: it creates a temporary file, then downloads the location (URL) in this temporary file and simply returns the content as a String. In order to be safe, I called tempFile.deleteOnExit(), but as you can see the file is deleted in the finally.

Non intuitively (but to be fair as stated in the javadoc!), once deletion has been requested, there is no way to cancel it, even if you delete the file!

The implementation of java.io.File.deleteOnExit() simply keeps a list of String representing all the files that need to be deleted when the VM exits. And this list grows and grows as the cat method is called (which in this specific scenario was every 15 seconds, so the leak took many weeks to manifest itself!).

Looking back at the issue and this specific java call, I actually do not see a real use for it and I have decided to ban it from my code for several reasons as I actually think it is evil:

it gives you a false sense of security: “oh the VM will take care of it…” which is not true: if the VM does not terminate properly, for any particular reason (power outage, kill -9,…) then the files will simply not be deleted. If your code relies on the fact that those files should not be present on VM restart then one day you will have a nasty surprise.
calling this method will eventually lead to a memory leak (granted of course it is called repeatedly, even if the frequency is very slow!)

In the end, dealing with files is hard. If you generate a lot of temporary files you need to delete them at some point or another or you will fill up the disk. It is not always easy to know when a temporary file is safe to be deleted, especially if a method creates it and returns it. I think this is where java.io.File.deleteOnExit() totally fails: it gives you the impression that it implements GC for files but as I mentionned before there are a lot of shortcommings.

How to treat documentation as code: gradle + sphinx

2011-04-25T00:00:00-07:00

Introduction

About a month ago, I embarked on a complete rewrite of the glu documentation (here is the result). The first iteration of the documentation was created using the wiki hosted on github, alongside the source code.

I was not very happy with this solution for several reasons:

I knew that writing the documentation was going to be a pretty big effort and I did not want to have the wiki broken as I was writing it
You can argue that the wiki on github is managed, under the cover, as a git repository and that you can check it out locally, even create branches, and push it all at once. But this is where it falls apart:
- the wiki is not wysiwyg (far from it), and since github uses its own processing pipeline (which to my knowledge is not available as a local command line), there is no way to actually see what you get until you push it live (how would you see a branch for example ?)
- github seems to be using different pipelines depending on where you are on the website (for example the rendering of a gist is different from the exact same input in the wiki given a file with the same extension!)

it is hard to control the rendering of a wiki (wiki is usually rendered inside a shell and you cannot control too much the layout or add your own javascript)

I needed to inject some information in the wiki (like the latest changes, or the version number)

you cannot bundle the documentation with the product

Choosing a different solution

I settled for using a markup based solution like what I have done for this blog. The main reasons are:

it is easy to write
it can be checked in with the rest of the source code
it can have its own build process (treat the documentation as code!)

Choosing a solution was not easy as there seems to be a lot of choices. jekyll was actually not a practical solution because it is really well suited for blogs, but not really for heavy documentation with cross references.

I ended but choosing sphinx. sphinx was originally created to write the new python documentation. The sphinx documentation is written with sphinx and it is beautiful (as they mention it themselves) and very comprehensive. It has a lot of bells and whistles like:

changing the theme
generating different outputs (html, pdf, man pages, etc…)
setting up the project for you (Makefile, configuration file, directory structure, etc…)
generating a self contained web site
…

In the end I am really happy with the choice as it has been fairly easy to concentrate on the documentation itself rather than the tool. I am also very happy with the final result: it is using the agogo theme with some customizations (which are fairly easy to do).

Integrating sphinx in the build process

glu uses gradle for its build framework and in order to integrate the documentation generation as part of the build, I simply treated the documentation as another project in the source tree. You can check it out on github.

sphinx generates a Makefile as well as a configuration file (conf.py) for you. I simply had to create a gradle task to do the following:

copy/inject the version (coming from the build)
copy the release notes in the sphinx directory structure
invoke the Makefile

This is what it looks like (you can view the full build file)

sphinxDir = new File(buildDir, "sphinx")
htmlDocDir = new File(sphinxDir, "build/html")

task doc << {
  // copy/replace @version@
  copy {
    from("src/main/sphinx") {
      filter(tokens: ["version": version], ReplaceTokens)
    }
    into sphinxDir
    exclude "**/*.png"
    exclude "**/*.gif"
  }
  // copy images (no replacement)
  copy {
    from("src/main/sphinx")
    into sphinxDir
    include "**/*.png"
    include "**/*.gif"
  }
  // copy the release notes at the root of the tree
  copy {
    from new File(rootDir, "RELEASE.rst")
    into new File(sphinxDir, "source")
  }
  // execute the makefile
  ant.exec(executable: "make", dir: sphinxDir, failonerror: true) {
    arg(line: "html")
  }
}

Finally, the documentation gets copied in the final product: in the end it is a static web site with a self contained set of html pages, css and images.

To generate the tables, I used the table-capture mode from emacs :)

Conclusion

Treating documentation as code has a lot of advantages:

it can be checked in with the rest of the source code thus sharing the same lifecycle, branching strategy, etc…
using a markup language (text based) allows for easy source control management of the documentation (you can do diffs, and whatever else you can on regular source code)
it can have its own build process allowing you to inject information from the build itself (in this case I am injecting only the build version, but it opens up the door to a lot more (like test results, etc…))
you can build any kind of automation around it

Note about github

I may seem to be ranting about github, but really I love what they have done. It is an amazing idea and if you are a software developer, you should have an account there! From my point of view, github is for developers what a portfolio is for artists!

Building a monitoring solution with glu

2011-03-18T00:00:00-07:00

Introduction

glu is a deployment automation platform which allows you to easily and efficiently deploy applications on many hosts. In the glu model, there is an active agent (aka, the glu agent) running on each host. Since this agent is always up and running on the host, it can be used to monitor the host itself and/or the applications running on it. By using the timers feature of the agent and ZooKeeper, it is possible to build a very solid monitoring solution by reusing the infrastructure already in place for glu.

glu agent timers

The timers feature of the glu agent allows you to schedule a timer, defined by a piece of code (a closure), which is executed at regular intervals. The agent manages timers appropriately, taking care of running them for you and restarting them in the event that the agent restarts. The full api of the timers feature can be found on github. Here is how you schedule and cancel a timer in a glu script:

class MonitorScript
{
  def monitor = {
    // code that will run every 5s
 }

 def start = {
   timers.schedule(timer: monitor, repeatFrequency: "5s")
 }

 def stop = {
   timers.cancel(timer: monitor)
  }
}

The code defined in the monitor closure will be executed with the frequency defined at scheduling time until cancelled.

Adding monitoring code to the script

Now that we have timer, we can add the monitoring code. For the sake of this blog post, we will simply execute the uptime shell command.

class MonitorScript
{
  def monitor = {
    log.info shell.exec("uptime")
 }

 def start = { timers.schedule(timer: monitor, repeatFrequency: "5s") }
 def stop = { timers.cancel(timer: monitor) }
}

Here is the result of running this script: every 5 seconds, a message containing the output of the uptime command gets displayed in the agent log file.

Detecting and reporting errors from within the script

We can add logic to the script to change the state based on the result of the command (using the stateManager). The state will automatically be reported in ZooKeeper and propagated to the console.

class MonitorScript
{
  static String CMD = 
    "uptime | grep -o '[0-9]\\+\\.[0-9]\\+*' | xargs"

  def monitor = {
    // capturing current state
    def currentError = stateManager.state.error
    def newError = null
    def uptime = shell.exec(CMD)
    def load = uptime.split()
    // check for load (provided optionally as an init parameter)
    if((load[0] as float) >= ((params.maxLoad ?: 4.0) as float))
        newError = "High load detected..."
    // change the state when currentError != newError
    if(currentError != newError)
      stateManager.forceChangeState(null, newError)
 }
 
 def start = { timers.schedule(timer: monitor, repeatFrequency: "5s") }
 def stop = { timers.cancel(timer: monitor) }
}

In the console, you simply add this entry for each agent (in the model):

{
  "agent": "agent-1",
  "initParameters": {"maxLoad": 1.0},
  "mountPoint": "/monitor",
  "script": "http://localhost:8080/glu/repository/scripts/MonitorScript.groovy",
  "tags": ["monitor"]
}

Note how I set the maxLoad parameter to 1.0 for being able to test more easily.

When the load is normal, the console dashboard looks like this:

As soon as the load is too high, the console looks like this:

Making monitoring information available to ZooKeeper

We have seen how we can change the state so that it gets reported right away in the console. This is fine, and it also means that the detection logic lives in the script. If we want to be able to do more sophisticated monitoring (trending, etc…) the logic needs to live outside the glu script. We still want the script to collect the monitoring data and simply make it available so that an external process can consume it. This is actually trivial to do:

class MonitorScript
{
  static String CMD = 
    "uptime | grep -o '[0-9]\\+\\.[0-9]\\+*' | xargs"

  // load is automatically available in ZooKeeper
  def load    

  def monitor = {
    def currentError = stateManager.state.error
    def newError = null
    def uptime = shell.exec(CMD)
    
    // load is simply assigned to the field rather than a 
    // local variable
    load = uptime.split()
    
    if((load[0] as float) >= ((params.maxLoad ?: 4.0) as float))
        newError = "High load detected..."
    if(currentError != newError)
      stateManager.forceChangeState(null, newError)
 }
 
 def start = { timers.schedule(timer: monitor, repeatFrequency: "5s") }
 def stop = { timers.cancel(timer: monitor) }
}

The code is almost identical: the only difference is that we simply use a field rather than a local variable!

In the glu world, all fields are automatically made available to ZooKeeper.

We can verify that it makes it in the console:

In this screenshot, everything below ‘View Details’ is coming directly from ZooKeeper. We can see the error message set by the script because it detected a high load, as well as the load itself. Note how the load is actually an array (under the cover the representation is json). This entry will change every time the timer executes (unless the load does not change of course).

Collecting and processing the monitoring data

As we have seen the data makes it in ZooKeeper. The fact that you can view it in the console is nice, but if you want to process the data, you can write an external program that will connect directly to ZooKeeper. You can reuse the convenient class that the console uses: AgentsTracker and register a set of listeners to be notified when things change.

 1 public class MonitorMain
 2 {
 3   def zk = "localhost:2181"
 4   def fabric = "glu-dev-1"
 5 
 6   void run()
 7   {
 8     // establishes connection to ZooKeeper
 9     def zkClient = new ZKClient(zk, Timespan.parse("30s"), null)
10     zkClient.start()
11     zkClient.waitForStart(Timespan.parse("5s"))
12 
13     // creates a tracker to track events in ZooKeeper
14     String zkAgentRoot = "/org/glu/agents/fabrics/${fabric}".toString()
15     def tracker = new AgentsTrackerImpl(zkClient, zkAgentRoot)
16     
17     // registers the listeners
18     tracker.registerAgentListener(agentEvents as TrackerEventsListener)
19     tracker.registerMountPointListener(mountPointEvents as TrackerEventsListener)
20     tracker.registerErrorListener(errorListener as ErrorListener)
21 
22     tracker.start()
23     tracker.waitForStart()
24     awaitTermination()
25   }
26 
27   // track agent events (agent up or down)
28   def agentEvents = { Collection<NodeEvent<AgentInfo>> events ->
29     events.each { NodeEvent<AgentInfo> event ->
30       switch(event.eventType)
31       {
32         case NodeEventType.ADDED:
33         case NodeEventType.UPDATED:
34           // ignoring add/update event
35           break
36         case NodeEventType.DELETED:
37           log.warn "Detected agent down: ${event.nodeInfo.agentName}"
38         break
39       }
40     }
41   }
42 
43   // track mountPoint events
44   def mountPointEvents = { Collection<NodeEvent<MountPointInfo>> events ->
45     events.each { NodeEvent<MountPointInfo> event ->
46       // only interested in the monitor events (/monitor was defined in 
47       // the console entry)
48       if(event?.nodeInfo?.mountPoint == MountPoint.fromPath("/monitor"))
49       {
50         switch(event.eventType)
51         {
52           case NodeEventType.ADDED:
53           case NodeEventType.UPDATED:
54             log.info "${event.nodeInfo.agentName} |" +
55                "${event.nodeInfo.data?.scriptState?.script?.load?.join(',')}"
56             break
57           case NodeEventType.DELETED:
58             // ignoring delete event
59           break
60         }
61       }
62     }
63   }
64   
65   // when an error happens
66   def errorListener = { WatchedEvent event, Throwable throwable ->
67     log.warn("Error detected in agent with ${event}", throwable)
68   }

8-11: uses ZKClient to establish a connection with ZooKeeper
13-23: creates a tracker, registers the listeners and starts the tracker
28-41: this listener gets called back whenever an agent appears/disappears from ZooKeeper. Agents use the ephemeral node capability of ZooKeeper which means that if the agent fails for any reason (ex: machine reboots), the ephemeral node will go away and as a result this listener will be called.
This is a very cheap and effective way to monitor the fact that a machine is still alive.
44-63: this listener gets called back whenever there are changes in the mountPoints installed on any agents. For the sake of monitoring we do not care about any other but /monitor which is why we filter it out. The expression: event.nodeInfo?.data?.scriptState?.script gives access to any of the fields of the glu script and in this case we are only interested in the load field.
66-68: this listener gets called back whenever there are any errors (ex: loss of communication with ZooKeeper)

Here is the result of running this program:

Note that I intentionally killed the agent to demonstrate that I was receiving the event, which can be seen on line 5.

Conclusion

This post shows you how to:

create a glu script which installs a timer on an agent/host
run a monitoring command on the host and react based on the result (change the state)
propagate monitoring information to ZooKeeper
capture and handle this information in an external program

In a real monitoring solution, you probably will want to do a lot more than capturing the load and displaying it on the standard output. But the concepts will remain the same. I think the beauty of this solution lies in the fact that:

if you use glu for deployment then the infrastructure is already in place! And if you don’t use glu (yet), then maybe you should consider using it as you can kill 2 birds with one stone!
it is event based so you get notified only when things change or in other words you don’t need to ask every host what their status is: every host tell ZooKeeper what their status is and the external program is simply listening to ZooKeeper events.
it is all open source and free.

References

For more information about glu, check the wiki.
You can download glu including the tutorial from the download page.
MonitorScript source code from github.
MonitorMain source code from github.

Migrating my blog from JRoller 4.0.1 to Jekyll

2011-03-02T00:00:00-08:00

Let’s face it, JRoller is dead. Like many others I have decided to move my blog away from it. I have decided to use Jekyll because it basically generates a static website so I don’t need a java web server or a database to run my blog. The main issue of course is comments but there is a solution for this as well: Disqus. This post relates my experience.

Extracting content from JRoller 4.0.1

The biggest problem to solve is how to extract the blog posts from JRoller including the comments. This is how I did it:

Add a new template
In the roller interface, under the Design tab, Templates subtab, I added a template. The template looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<posts>
#set($entries = $model.weblog.getRecentWeblogEntries('nil', 100))
#foreach( $entry in $entries )
  <post id="$entry.id" permalink="$utils.escapeXML($entry.permalink)" title="$utils.escapeXML($entry.title)" date="$utils.formatIso8601Date($entry.pubTime)" tags="$utils.escapeXML($entry.tagsAsString)" category="$utils.escapeXML($entry.category.name)">
    <content>$utils.escapeXML($entry.displayContent)</content>
    #set($comments = $entry.getComments(true, true))
    <comments>
    #foreach( $comment in $comments )
      <comment name="$utils.escapeXML($comment.name)" email="$utils.escapeXML($comment.email)" url="$utils.escapeXML($comment.url)" remoteHost="$utils.escapeXML($comment.remoteHost)" date="$utils.formatIso8601Date($comment.postTime)">
        <content>$utils.escapeXML($comment.content)</content>
      </comment>
    #end
    </comments>
  </post>
#end
</posts>

Export the content
Note that on the page there is a text field called Link and a link below. Simply use this link to export the content:
```
curl <link> > blog.xml
```

Now I have a file called blog.xml which contains all my data.

Generating content for Jekyll

Jekyll requires you to create a specific directory structure with all your posts under a folder called _posts with a specific naming convention. To generate the content I wrote a simple groovy script available on github (you may need to tweak the script for your own needs). The script uses the blog.xml file previously generated and creates one file per blog post. It uses the permalink tag so that the final link is the same as it was with JRoller!

Click to view: jroller401_to_jekyll.groovy

Importing comments in Disqus

Disqus is a third party website which can host your comments. Thankfully the service has the capability to import comments formatted in WPR xml format (which is a wordpress xml representation). I also wrote a simple groovy script to do the job. It is available on github. The script uses the blog.xml file previously generated and ouputs the xml document.

Click to view: jroller401_to_disqus.groovy

Final touches

Once the data was out of JRoller and in Jekyll and Disqus, I spent a lot of time (way too much :)), redesigning the blog, making it more up to date and adding a new social twist to it, integrating with Twitter and LinkedIn. In a way it is more dynamic than it ever was. Here are several interesting things I had to do with Jekyll:

1. Listing Categories

Although Jekyll supports the notion of categories, there is no native support to generate category pages. For this I used the generate_categories.rb plugin. I had to enhance the plugin to list the categories (why isn’t it something that Jekyll offers by default is beyond me…): you can view my enhancement here

# Yan addition: accessing the list of categories
alias orig_site_payload site_payload

def site_payload
  h = orig_site_payload
  payload = h["site"]
  payload["categories_list"] = payload["categories"].keys.sort
  h
end

Then I can simply use site.categories_list in my code.

2. Dynamic css

Any file in Jekyll can be processed as a template file. This is what I did with my css: I defined various values in my config file:

# css related entries
mainColor1: '#005a87'
mainColor2: '#0075b0'
contentBGColor: '#eeeeee'
contentFGColor: 'black'
titleBGColor: '#600000'
titleFGColor: 'white'
contentFontFamily: 'Helvetica,Arial,sans-serif'
monospaceFontFamily: 'Courier New,monospace'
dateBackgroundColor: '#888888'

and simply use them in my css file:

---
---
a:visited {
  text-decoration: none;
  color: {{ site.mainColor1 }};
}

Of course my css editor is unhappy about the invalid syntax but it totally works. The first 2 lines are the marker to tell Jekyll to process this file as a template and are very important. You can take a look at the css file.

3. Reusing the post layout

I wanted that the latest blog post, displayed on the home page, look exactly the same as it looks on each individual blog page without having to copy/paste the code. For this I used an include but it is pretty tricky (I literally spent a full day on this :(). Here is how I did it:

_layouts/post.html:
# the post.html layout simply includes another template
---
layout: default
---
{% include post_include.html %}


index.html:
# the index.html main home page includes the same template
{% for post in site.posts limit:1 %}
{% include post_include.html %}
{% endfor %}


_includes/post_include.html:
# now for the tricky part: the template needs to have some conditional!
{% unless post %}
  {% assign post = page %}
{% else %}
  {% assign content = post.content %}
{% endunless %}
<div class="post-title">
  {{ post.title }}
</div>
<div class="post-content">
  {{ content }}
</div>

Source code

The entire source code of this blog is available on github and is available under a Commons Creative license.

The nightmares of setting up a wireless N network

2011-02-12T00:00:00-08:00

After converting the guest room into a second office, I was faced with a dilemma: the internet is in one office but of course both offices need to access the internet. My home is fairly recent and as a result, extremely well wired. The only catch is that it is wired for coax and phone jacks but not for cat5 (we don't even use land line phones anymore!).

We opted for upgrading our network to wireless N (it was G) so that my MacBook Air could benefit from it and my desktop could also get good throughput.

First nightmare: I now own an 'old' Mac Pro. It is a dual Xeon Quad-Core 2.8Ghz with 14GB of memory purchased in early 2008. The catch is that all 'new' Apple computers now come with wireless built-in including the Desktops. And of course Apple is not selling a wireless N card (or B or G for that matter): this is too cheap and really Apple would prefer you to buy a new Mac. A trip to my local Apple Store revealed totally clueless employees who clearly did not give a damn about the 'past' and the only thing they cared about was for you to buy a brand new machine because it comes with wireless built-in (why would you buy a $100 card when you can buy a $3000 brand new Mac ? Really ?).

In the end it is the traditional pattern: there are a ton of choices if you own a PC. But if you own a Mac and it is not Apple supported, it is really hard/impossible to find anything decent. You can pretty much count on your fingers the number of available options for the Mac. And when you start adding more constraints, like 5Ghz then the number of choices drops even further.

So the solution that we opted for was to install a dual-band wireless N router that would be connected to the internet in one office and a wireless N bridge (5Ghz) in the other office. That way the Mac is connected to the wireless network through ethernet, thus being transparent to the Mac.

Second nightmare: have you shopped lately for anything wireless N ? This is a freaking nightmare wrapped into layers of marketing bullshit. N600, extreme, gigabit, ultimate, 12x, 300Mbps... you name it. It is claim after claim of false advertising. I am not entirely sure how this is even legal. I would just like to remind DLink, Netgear, Cisco/Lynksis and others that the fastest you can go will always be limited by the slowest link in the chain. So when you claim that your devide goes up 300Mbps and the ethernet jack is regular ethernet (100Mbps), then really you can't go faster than 100! If you have tried to buy a digital camera it is the same kind of marketing bullshit with a meaningless race on the megapixel count.

From my experience, with all the devices I tried, the fastest I ever saw with the devices as close as possible to each other (touching each other antennas!) I could reach ~140Mbps...

In the process, we ended up trying 5 different devices!

First the 3 we did not keep:

DLink DIR-855 which is a wireless N router with 4 gigabit ports. The major issue with this one was that the best reception I could get with my laptop touching the antenna was ~70%. Also the maximum transfer rate I ever saw was ~72Mbps in the best case scenario possible.
DLink DAP-1522 which is a wireless N access point / bridge. This one has a serious issue: there is 1 page in the configuration menu (web based) which requires Internet Explorer to work. This is a very well known bug that has numerous people complaining about and it has been reported more than a year ago. DLink does not bother to fix it nor even change the text on the box or the web site still claiming it works for Windows and Mac with IE or Firefox. It is just a lie. It does not. Also during my experiments it seemed very flaky as it required many reboots.
Cisco/Linksys WET610N which is a dual band wireless N bridge. It is branded as a wireless gaming and video adapter but really it is bridge. The one I had was literally a piece of garbage (but to be fair I do not know if it was the unit I had that was defective or if it is really the way it works). The web based interface is slower than christmas with (I am not kidding!) a wait of over 10s to go from page to page! Not to change anything... just going from page to page! We had a really hard time to make it connect to the router and we finally changed the 5Ghz channel on the router and the bridge could finally see it (this is totally undocumented nor configurable in this device). Also the wizard would not accept our passphrase because it contains spaces (hello! a passphrase!). We managed to make it connect to the router but for no more than 5s: it would constantly connect/disconnect over and over. Note that during this experiment, the device was literally touching the router so it could hardly be closer!

Then the 2 we ended up keeping:

Asus RT-N56U which is a dual band gigabit wireless N router. I know that it is supposed to look nice, but to be fair I prefer when it is more badass looking (I really like the blue Netgear switches with the connectors in the front :). This one is very impressive. The UI is actually really nice and it is a breath of fresh air. The same test done with the laptop sitting right next to it gave very different results: the reception was 98% and the transfer rate was ~140Mpbs. It also has 2 USB ports and we ended up buying a cheap 16GB USB key at Micro Center and plugging it directly in the router as our common storage. Very sweet and works like a charm. It has been working great since we had it. This one is highly recommended.
DLink DAP-2553 which is a dual band access point but can also act as a bridge. This one is definitely more expensive and is in the business line so it is not as customer friendly. It definitely looks badass especially the antennas. Although not a customer oriented product, it was a million time easier to make it work than the stupid wireless gaming adapter! We simply selected to work as a bridge, entered the WPA2 key and voila... it just worked. And sitting in the kitchen which is really far from the router was getting 60% reception! Amazing! Also the Ethernet connection is Gigabit (it is not even mentioned on the box!). It has a lot more feature than I need, as it can run as a full access point, acting as a DHCP server, etc... It has been rock solid since we installed it.

Clearly it has been quite an adventure and I am still unclear why I can never see higher than ~140Mbps transfer rate. The reality is my Internet connection is less than 20Mbps so it will never be an issue with the Internet. If I want to transfer big files I can simply add a gigabit switch (which I already own) to my access point so that the transfer go through the switch.

In conclusion I wanted to emphasize 2 points:

In this case more expensive really meant better quality. I ended up paying ~$160 for the Access Point but it is worth it! (I know it is cheaper on Amazon but I could buy it right away and return it quickly in the event it did not work)

Make sure you buy at a store where you can easily return! Clearly Amazon is awesome in that regards but the turn around is way too slow. I ended up buying at Fry's. It was more expensive but I could buy and return several items the same day!

LinkedEnd...

2010-12-07T00:00:00-08:00

When I look at LinkedIn's company page as of December 2010, I can see that the company has now 1000+ employees spread all around the world.

When I started at LinkedIn, 8 years ago, there wasn't such a page to give me information about LinkedIn. LinkedIn simply did not exist. Period.

Looking back, this is the true American Dream/Silicon Valley-garage-startup deal! Of course we did not really work in a garage. We started meeting in Reid's apartment in downtown Mountain View (CA) with a handful of people. We had no office. We did not even have a name for the company (I am so glad we did not pick 'getin.com' which was on the shortlist!)

We just started from there and worked really hard. Official launch date was Cinco de Mayo 2003. The rest is history.

Fast forward 8 years later. I have decided to leave. It is never an easy decision after all these years but I leave happy and extremely proud of what I have been able to accomplish at LinkedIn. I am not going anywhere, just taking some time off, for a little while. I have various projects that I will continue working on like my own website kiwidoc or the open source projects I just released. But more importantly, I also want to relax and enjoy life.

My most favorite projects

Throughout my career at LinkedIn, I have had the chance to work on a wide variety of projects. Here is a compilation of the ones that matter the most to me, either because they were particularly challenging or simply because they were solid building blocks on which a lot of the infrastructure was built.

Resource framework: an easily extensible framework which hides the location of resources (files) no matter where they are located (local filesystem, in a local jar file, in a remote jar file, within the servlet context, on the classpath, in multiple locations...). This framework is part of the linkedin-utils open source effort (javadoc, source). #java
jsp20 / EL compiler: a highly efficient implementation of jsp20 and expression language from the ground up (uses Resource framework for easy development and centralized deployment) used at LinkedIn for generating the html. #java, #javacc, #compiler
UI Framework: the framework used by the frontend applications to deal with forms and requests. #java
LinkedIn Spring: built on top of Spring Framework 2.5 to support the notion of components (concept of nested beans). The core of the wiring system used at LinkedIn. #java, #spring
GoBack framework: a framework that manages the concept of go back in a web application (after following a flow of several screens, you complete an action which brings you back to where you started, no matter where the starting point was) used on the website. #java
Databus: a highly scalable database replication system which also offers an event based interface used for event propagation and replication in LinkedIn's infrastructure. #java, #jdbc
Cloud Server: the initial scaleable implementation of the graph engine powering LinkedIn. #java, #graph
Search Engine: the initial scaleable implementation of the search engine powering LinkedIn. #java, #lucene
Comm system: initial design and prototype of the comm/inbox system. #ruby
i18n profile: a massive effort to handle extensible profiles and internationalization. #java, #i18n
Authentication server: design and implementation of the authentication server used for security at LinkedIn. #java, #security
Payment System: initial design and implementation of the payment system which processes the credit card transactions. #java, #jms, #castor
OSGi investigations: investigations and prototyping of an OSGi based infrastructure. #osgi
Media server: design of the media server, the server which processes and delivers the pictures (and other media) on the website. #java
glu: the deployment automation platform used at LinkedIn to provision (efficiently and in parallel) the services on multiple data centers that comprise the LinkedIn system. This project is now open source. #java, #groovy, #rest, #zookeeper, #grails, #devops
and many many more...

Conclusion

A friend of mine was pointing out that I am no longer a LinkedIn employee, but I will always remain a LinkedIn co-founder, associated for the rest of my life to this amazing experience. I am extremely grateful to all the people that were (and still are) involved in this dream, because without them, it would not have been possible.

Automating publishing to maven central via Sonatype

2010-11-27T00:00:00-08:00

After releasing several open-source projects which are not using maven for the build framework (they use gradle), I was faced with the harsh reality that if you don't use maven, it is pretty hard to automate the process. The current version of gradle allows you to upload to a maven compatible repo, but to upload to maven central there are some constraints (like pgp signing) that gradle does not (currently) support. In order to publish your artifacts to maven central via Sonatype (unclear if there is another way to be honest), you need to follow the instructions on how to set it up (you need to create a Jira Ticket). You also need to have a pgp key and if you don't, you can follow these instructions. Using gradle, you can simply create a maven compatible layout of your artifacts:

// example for one of my project
uploadArchives {
  repositories {
   mavenDeployer {
    repository(url: "file://localhost/my/local/repo")
    pom.whenConfigured { pomToConfigure ->
      pomToConfigure.project {
        name project.name
        description "A useful set of gradle plugins"
        url "http://github.com/linkedin/gradle-plugins"
        licenses {
          license {
            name "The Apache Software License, Version 2.0"
            url "http://www.apache.org/licenses/LICENSE-2.0"
          }
        }
        developers {
          developer {
            id "ypujante"
            name "Yan Pujante"
            email "yan@pongasoft.com"
          }
        }
        scm {
          connection "scm:git:${spec.scmUrl}"
          developerConnection "scm:git:${spec.scmUrl}"
          url spec.scmUrl
        }
      }
    }
  }
}

Note that a big part of the previous code is setting up the right information in the pom file to comply with the set of requirements. Also note that you do not need to use gradle at all as long as you generate a maven compatible layout (with pom files). You can then use the script I have 'gisted' on github: maven_staging_upload.groovy. The script simply takes the pom files you want to upload as an input and will prompt you for your pgp passphrase only once! Make sure you follow the initial setup:

/**
 * This script uploads all your artifacts to maven.
 * 
 * To run this script there are several asumptions:
 * - gpg is installed and available in the path
 * - you have followed the instructions to set up your pgp key
 *   (http://www.sonatype.com/people/2010/01/how-to-generate-pgp-signatures-with-maven/)
 * - maven is installed and available in the path
 * - you have defined the proper server section in your ~/.m2/settings.xml file:
 *   <settings>
 *    ...
 *    <servers>
 *       <server>
 *        <id>nexus-releases</id>
 *        <username>ypujante</username>
 *        <password>********</password>
 *       </server>
 *     </servers>
 *    ...
 *   </settings>
 *
 * You run the script this way: groovy maven_staging_upload.groovy pathToPomFile1 pathToPomFile2...
 *
 * groovy
 */

You then need to login to Nexus to verify that the upload worked properly. You can then close the staging repository and release it. It will make its way to maven central shortly. The very first time, you need to activate central sync! I guess it is not fully automated as you still need to manually close/release the staging repository but the painful part (the part about uploading the artifacts to staging) is fully automated thanks to gradle and the script. You can use the gradle plugins that I open sourced to help with the release: the org.linkedin.repository and org.linkedin.release plugin can simplify your build code (for example the plugin takes care of the javadoc and sources requirements). Note that this script can be used totally independently of gradle and you do not need to use gradle to use it! Thanks to Jakub Holy for this blog post which helped me in understanding the steps I needed to follow in order to automate them. I am also hopeful that the 1.0 release of gradle will render my script obsolete!

Using gradle with grails

2010-08-28T00:00:00-07:00

I just recently migrated my entire kiwidoc project to use gradle for its build framework (replacing maven). The biggest challenge I faced in the exercise was to have grails work with gradle. I spent way too much time on this part due to my lack of deep knowledge of each framework but in the end it is extremely simple. The reason why it was challenging in the first place is that my project is a multi-project build as gradle calls it (see Chapter 36). What I wanted was that my other projects be recompiled before starting grails to get the latest version. There are 2 ways you can go about it:

1. Use the grails gradle plugin

I actually opened a Jira Ticket for this and within a couple of days there was already a solution out there. The details are on the jira ticket and on the github project. Here is how you use it:

// external plugin
buildscript {
  repositories {
    mavenCentral()
    mavenRepo urls: "http://repository.jboss.org/maven2/"
  }

  dependencies {
    classpath "org.grails:grails-gradle-plugin:1.0",
              "org.grails:grails-bootstrap:${grailsVersion}"
  }
}

apply plugin: "grails"

repositories {
  mavenRepo urls: "http://repository.jboss.org/maven2/"
}

dependencies {
  compile "org.grails:grails-crud:${grailsVersion}"
  compile "org.grails:grails-gorm:${grailsVersion}"

  compile project(":kiwidoc:com.pongasoft.kiwidoc.index.api")
  compile project(":kiwidoc:com.pongasoft.kiwidoc.index.impl")
  compile project(":kiwidoc:com.pongasoft.kiwidoc.builder")
  compile project(":utils:util.html")

  runtime "org.slf4j:slf4j-log4j12:1.5.8"
  runtime "org.aspectj:aspectjrt:1.6.8"
}

grailsVersion is a variable defined in my 'root' build script (grailsVersion="1.3.4"). As can be seen on this short example, I have dependencies to my other projects built by gradle as well. Please note the extra dependency on aspectj due to an improper pom file :(.

2. Use gradle to build the classpath

Solution #1 works great but everything you do needs to go through gradle and it also downloads a new version of grails. So if you already have grails installed there is another solution:

apply plugin: "groovy"

dependencies {
  compile project(":kiwidoc:com.pongasoft.kiwidoc.index.api")
  compile project(":kiwidoc:com.pongasoft.kiwidoc.index.impl")
  compile project(":kiwidoc:com.pongasoft.kiwidoc.builder")
  compile project(":utils:util.html")

  groovy: "org.codehaus.groovy:groovy:1.7.4"
}

task lib(dependsOn: jar) << {
  fileTree(dir: "lib").each { it.delete() }
  copy {
    from configurations.runtime
    into "lib"
  }
}

["run-app", "war"].each { taskName ->
  task "${taskName}"(dependsOn: lib) << {
    ant.exec(dir: ".", executable: "grails") {
      arg(line: "${taskName}")
    }
  }
}

By declaring the lib task to depend on the jar task, the groovy plugin will automatically build my classpath which I copy in the lib folder that grails know about. I can then simply invoke the grails command line (which is assumed to be in your path). The beauty with this solution is you can simply run gradle lib and once you are done, you can go on the command line and simply run grails directly as now all the dependencies you need are where grails expects them. If you want to run the app with gradle, you need to use gradle -i run-app otherwise you won't see any of the messages that grails log.

ZooKeeper loss of events problem... fixed

2010-07-24T00:00:00-07:00

In my latest project at LinkedIn, I have been using ZooKeeper to track the state of all deployed services on every machine in production. The state is then used to drive the deployment as well as monitor the system. I ran into a really tricky bug which took me several months to address (the hardest part was finding what the actual problem was). The (simplified) code was the following:

ZooKeeper zk = ...
Map<String, byte[]> state = [:]

synchronized(lock) { state = trackChildren(state) }

// handle add/delete of children
private Map trackChildren(Map oldState) {
  def newState = new HashMap()
  def children = zk.getChildren("/state", childrenWatcher as Watcher)
  children.each { child -> 
    if(oldState[child] == null) // ADD
      trackChild(newState, "/state/${child}")
    } else { 
      newState[child] = oldState[child] // no change
    }
  }
  // DELETE: newState does not contain the children that have disappeared
  return newState
}

// handle child update
private void trackChild(Map newState, String path)
{
  if(path == null) return
  def child = new File(path).name
  try { newState[child] = zk.getData(path, childWatcher as Watcher, null) }
  catch(NoNodeException e) {  /* ok node is gone */ newState.remove(child) }
}

def childrenWatcher = { WatchedEvent event -> 
  if(event.type == EventType.NodeChildrenChanged) {
    synchronized(lock) { state = trackChildren(state) }
  }
}

def childWatcher = { WatchedEvent event -> 
  def child = event.path ? new File(event.path).name : null
  synchronized(lock) {
    if(event.type == EventType.NodeDataChanged) { 
      trackChild(state, event.path)
    }
    if(event.type == NodeDeleted) { state.remove(child) }
  }
}

What this code essentially does is the following:

Sets a children watcher on /state to be notified of children addition/deletion
For each child, sets a node watcher to be notified of child modification and deletion. Note that you need this watcher because ZooKeeper will not call the parent watcher when a child gets modified!
As a result, the map called state always contain a 'copy' of the data of all children.

From the ZooKeeper documentation, there are some key concepts to remember about watches:

Watches are one time triggers; if you get a watch event and you want to get notified of future changes, you must set another watch.

Because watches are one time triggers and there is latency between getting the event and sending a new request to get a watch you cannot reliably see every change that happens to a node in ZooKeeper. Be prepared to handle the case where the znode changes multiple times between getting the event and setting the watch again. (You may not care, but at least realize it may happen.)

A watch object, or function/context pair, will only be triggered once for a given notification. For example, if the same watch object is registered for an exists and a getData call for the same file and that file is then deleted, the watch object would only be invoked once with the deletion notification for the file.

The code I wrote seems to handle #1 perfectly. On the other end, it does not handle #2 properly and this is where the bug gets triggered. I ran into a situation where a server would delete and rewrite the data very fast:

byte[] child1State = ...
zk.delete("/state/child1", -1)
zk.setData("/state/child1", child1State, -1)

In this scenario, I would receive 2 events:

/state childrenWatcher would be fired with a NodeChildrenChanged event. And the bug gets triggered here. According to key concept #2, essentially events can be collapsed: in this case I am getting only 1 NodeChildrenChanged event and when I list my children (getChildren), I actually see no difference: child1 has been removed and added, but I don't see it. I am effectively loosing the 'add' child event with the code written this way.
/state/child1 childWatcher would we fired with a NodeDeleted event resulting in state.remove('child1')... at this point I have indeed lost the state of child1.

With the following code on the server side:

byte[] child1State = ...
zk.delete("/state/child1", -1)
Thread.sleep(5000)
zk.setData("/state/child1", child1State, -1)

I actually get 2 events and I have enough time to process them so it works. But that is clearly not the right fix. In order to fix the problem, I did 2 things:

The children watcher is only dealing with 'ADDs' and does not handle 'DELETE' anymore
The child watcher handles 'DELETE' but also tries to set a watcher no matter what

ZooKeeper zk = ...
Map<String, byte[]> state = [:]

synchronized(lock) { state = trackChildren(state) }
// handle ADD only... let the children watcher handle DELETE
private Map trackChildren(Map oldState) {
  def newState = new HashMap(oldState)
  def children = zk.getChildren("/state", childrenWatcher as Watcher)
  children.each { child -> 
    if(newState[child] == null) 
      trackChild(newState, "/state/${child}")
  }
  return newState
}

// handle child update
private void trackChild(Map newState, String path)
{
  if(path == null) return
  def child = new File(path).name
  try { newState[child] = zk.getData(path, childWatcher as Watcher, null) }
  catch(NoNodeException e) {  /* ok node is gone */ newState.remove(child) }
}

def childrenWatcher = { WatchedEvent event -> 
  if(event.type == EventType.NodeChildrenChanged) {
    synchronized(lock) { state = trackChildren(state) }
  }
}

def childWatcher = { WatchedEvent event -> 
  def child = event.path ? new File(event.path).name : null
  synchronized(lock) {
    if(event.type == EventType.NodeDataChanged) {
      trackChild(state, event.path)
    }
    if(event.type == NodeDeleted) { 
      state.remove(child)
      // WARNING!!! UNINTUITIVE BUT CRITICAL!!!
      trackChild(state, event.path)
      // WARNING!!! UNINTUITIVE BUT CRITICAL!!!
    }
  }
}

The code (in the childWatcher) most likely looks very unintuitive, but what is really happening in this case, is that the child on which the watcher is set will always receive the NodeDeleted event and according to key concept #1, if you want to be able to receive further events you need to set another watcher. It seems unintuitive to set a watcher on a node that just got deleted (and most of the time it will trigger a NoNodeException which is handled properly), but between the time the node was deleted and the time you check, it may very well have been recreated in which case you guarantee that you will have a watcher set on it and you won't loose it. If the node was deleted and not recreated right away, then the childrenWatcher will be the one setting it up again in the future if it gets recreated (pay close attention to the synchronization in the code as it is pretty critical for the system to work properly). Here is an example:

/state/child1 simply deleted

1. childrenWatcher receives NodeChildrenChanged => does nothing
2. childWatcher receives NodeDeleted => state.remove('child1')
1. childWatcher receives NodeDeleted => state.remove('child1')
2. childrenWatcher receives NodeChildrenChanged => does nothing

/state/child1 deleted / recreated (events get collapsed)

1. childrenWatcher receives NodeChildrenChanged => does nothing
2. childWatcher receives NodeDeleted => state.remove('child1') then state['child1']=new state + watcher
1. childWatcher receives NodeDeleted => state.remove('child1') then state['child1']=new state + watcher
2. childrenWatcher receives NodeChildrenChanged => does nothing

/state/child1 deleted / recreated (events are not collapsed)

1. childrenWatcher receives NodeChildrenChanged => does nothing
2. childWatcher receives NodeDeleted => state.remove('child1')
3. childrenWatcher receives NodeChildrenChanged => state['child1']=new state + watcher
1. childWatcher receives NodeDeleted => state.remove('child1')
2. childrenWatcher receives NodeChildrenChanged => state['child1']=new state + watcher
1. childrenWatcher receives NodeChildrenChanged => does nothing
2. childWatcher receives NodeDeleted => state.remove('child1') then state['child1']=new state + watcher
3. childrenWatcher receives NodeChildrenChanged => does nothing

As always with this kind of problem, the fix is really not that complicated. It is understanding the problem in the first place which is hard... You can check the ZooKeeper javadoc on kiwidoc :)