Version Management and OSGi

On this previous post on the LinkedIn blog, I talked about bundle repositories. In this one I am going to cover version management and the particularities of OSGi.

How dows OSGi handle version ?

In OSGi you define your dependencies using headers in the Manifest. There are several ways to define dependencies (Import-Package, Require-Bundle, etc...). One way to constrain a dependency is to use version. A version in OSGi is defined as Major.Minor.Micro.qualifer.

Example of versions:

1
1.0
2.4.5.ABC_DEF

• Major, Minor and Micro must be numbers. The qualifier is a string (with some constraints) and is not interpreted as a number (check the javadoc for the details).OSGi does not attach any more meaning to the numbers and it is up to the user to manage the numbers the way they want.
• When defining a dependency, you can use a version or a version range.

Example of ranges:

version=1         => means          v >= 1.0.0
version=[1.1.0,2) => means 1.1.0 <= v < 2.0.0
version=(1,2]     => means 1.0.0 <  v <= 2.0.0

Versionning convention

As mentionned previously, OSGi does not attach any meaning to the various components of a version. Here is the convention that seems to have been adopted by some open source projects.

• The Major number represents a major version: it is assumed that there is no backward compatibility between 2 different versions of the same bundle where the major number is different. Usually it means that APIs have changed in a non compatible way. (It would for example be the case if java serialized objects have been changed in a way that their serial version ID is different).
• The Minor number represents a version which contains changes that are backward compatible. A backward compatible change is for example, the addition of a method to an interface or new classes and objects not present in a previous version.
• The Micro number represents a version which is also backward compatible but does not contain any api enhancements. It is usually used for bug fixes and minor improvements.
• The qualifier is a string and is being used for various purposes depending on the project.

Upgrading version

Let's take the following example:

There is a service which is exposed as a java interface. This java interface resides in the bundle api-3.0.0. A service does not really have a version but since it uses this api it is fair to say that the version of the service is 3.0.0. The bundle impl-3.0.2 provides the implementation of the service and exports it to the OSGi registry. There are 2 clients of the service (client1 and client2). They both depend on the api. Also there is another external bundle (called lib-2.0.0) which happens to be used by both clients both directly (in their code) and indirectly because the api exposes some objects from this library in the api.

Service API (3.0.0)
-------------------
void f(FromLib200 param1);

Upgrade scenario

Lets now assume we enhance the service in a backward compatible way by offering a new api (new method on the java interface).

Service API (3.1.0)
-------------------
void f(FromLib200 param1);
void g(FromLib210 param1);

The new service API actually uses a new class which was not defined in the previous version of lib, thus requiring a new lib-2.1.0. We then assume that client1 uses this new enhanced api while client2 is left unchanged. Of course there needs to be a new implementation for this new api.

Upgrade results with minimal version lockdown

In this first case, we are assuming that we lockdown version only for protecting against incompatible changes. In other words we use ranges like this, locking down only on the major version number:

client1: api;version=[3.1.0,4),lib;version=[2.1.0,3)
client2: api;version=[3.0.0,4),lib;version=[2.0.0,3)
api-3.0.0: lib;version=[2.0.0,3)
api-3.1.0: lib;version=[2.1.0,3)

Here is the result:

Although client1 has not been updated, it is going to start using the new service. Since the new service is backward compatible it is not really an issue per se. What is an issue though is that it is also going to start using lib-2.1.0. Why is it an issue exactly ? In a very dynamic production environment (like LinkedIn's), this scenario is very frequent. The danger comes from the fact that by simply deploying a new version of a service, it ends up affecting a client in a way that has most likely not being tested.

Upgrade results with maximal version lockdown

In this second case, we are assuming that we lockdown the version entirely. In other words we use ranges like this, locking down major, minor and micro:

client1: api;version=[3.1.0,3.1.1),lib;version=[2.1.0,2.1.1)
client2: api;version=[3.0.0,3.0.1),lib;version=[2.0.0,2.0.1)
api-3.0.0: lib;version=[2.0.0,2.0.1)
api-3.1.0: lib;version=[2.1.0,2.1.1)

Here is the result:

Is there a solution then ?

As we mentionned previously, client2 should be able to talk to the new service because it is backward compatible. The only reason it cannot talk to it is due to class loading. If we were in separate containers we would not really have this problem (we would use spring rpc which does java serialization). So the idea is to replicate what happens when we are remote:

We can deploy a service which uses service 3.0.0 api and proxies all the call to the real service (we know that due to backward compatibility, the API of Service 3.0.0 is a subset of 3.1.0 so we should be able to proxy all calls). Due to class loading issues, the calls must go through java serialization (exactly like what would happen if it was remote...): in other words, we serialize all parameters with the class loader which loaded Service 3.0.0 and we deserialize with the one which loaded Service 3.1.0 (and vice versa for the return value/exceptions).

Conclusion

Solution 2 is not going to work. The choice is then between Solution 1 and 3. Solution 3 is not supported out of the box by OSGi and requires writing the proxy and the mechanisms to do the serialization / class loader transfer which is not necessarily an easy piece of code to write. Solution 1 is most likely the one that is going to be used in the end, and it is fine, as long as we are careful and aware of the 'dangers' of deploying more than one service in the same container. Distributed OSGi (RFC 119) is coming up and I think they will have to address some of the issues cross containers (the ability to upgrade a remote service to a newer backward compatible version without having to change the clients). So the point I was making is still valid: if it is going to work cross containers, it should also work in the same container (which is essentially Solution 3)...

Grails - Invoking a tag lib from another tag lib

Grails comes with a predefined set of tags that you can use in your gsp pages. If you want to add your own tags, it is pretty simple and you can simply check the Dynamic Tag Libraries reference documentation. I created my own version of the <g:each> tag which allows you to provide a begin, end and separator attributes:

class MyTagLib {
  static namespace = 'my'
  // Equivalent to g:each but allow for begin/end and separator attributes
  def each = { attrs, body ->
    def var = attrs.var ?: "var"

    def begin = attrs.begin ?: ""
    def end = attrs.end ?: ""

    def writer = out

    if(attrs.in)
    {
      // not null and not empty (definition of truth in groovy)
      attrs.in.eachWithIndex { elt, i ->
        if(i == 0)
        {
          writer << begin
        }
        else
        {
          writer << attrs.separator
        }
        writer << body((var):elt)
      }

      writer << end
    }
    else
    {
      if(attrs.alwaysBeginEnd?.toString() == "true")
      {
        writer << begin << end
      }
    }
  }
}

Here are some examples of rendering in gsp:

<my:each in="${[1,2,3]}" var="i">${i}</my:each>
produces: 123
<my:each in="${[1,2,3]}" var="i" begin="{" end="}" separator=",">${i}</my:each>
produces: {1,2,3}
<my:each in="${[1]}" var="i" begin="{" end="}" separator=",">${i}</my:each>
produces: {1}
<my:each in="${[]}" var="i" begin="{" end="}" separator=",">${i}</my:each>
produces: 
<my:each in="${[]}" var="i" begin="{" end="}" separator="," alwaysBeginEnd="true">${i}</my:each>
produces: {}

This tag is pretty convenient as it automatically takes care of an empty list or one that has only one element to properly display the separator and the begin and end attributes. The last example shows how you can 'force' to display the begin and end attributes when the list is empty.

Now, let's say I want to create another tag which will reuse the code I already wrote. In other words, I need to call a tag from within a tag. Here is how I would do it:

def csv = { attrs, body ->
  def var = attrs.var ?: "var"
  out << my.each(in: attrs.in, var: 'v', separator: ',') { map ->
  def elt = map.v
  out << "{"
  out << body((var):elt)
  out << "}"
  }
}

And here is the rendering in gsp:

<my:csv in="${[1,2,3]}" var="i">[${i}]</my:csv>
produces: {[1]},{[2]},{[3]}

It is actually not that trivial to call a tag from within a tag (and to my knowledge it is not documented)... let's cover each details:

• referencing another tag is used with the notation: namespace.tagName (ex: my.each)
• simply calling the other tag is not enough and the result must be sent to the writer (ex: out << my.each(...))
• each attribute is passed in as a map, so you simply use the groovy map notation (ex: (in: attrs.in, var: 'v', separator: ','))
• now the really tricky part is the closure which corresponds to the children tags in gsp... the argument that you get is a map (because in the my.each code, the body closure is called with a map!). Although it makes sense, it is not that trivial because in gsp you don't see it. This is why I need to use map.v to have access to the element that is being iterated over (the variable v is because it is the one that I used in the call (my.each(..., var: 'v', ...)))

Although a little tricky to write, it is very powerful to be able to create tags that build upon other tags. There is one little caveat in how null is being handled and I opened a Jira ticket for it (GRAILS-4449) as it does not seem to be consistent.

Starting from scratch... domain name and web hosting

When I started having the idea of a project that I could build, two things popped up right away: I needed a domain name and I needed some sort of web hosting solution. To be honest, both topics were totally brand new to me and I spent a significant amount of time investigating what is available.

Getting a domain name sounds like a trivial thing to do, but it turned out to be not such an easy task for two reasons:

• Finding a 'good' domain name is important and of course most of them are already taken. A big portion of them are simply unavailable because people owns them in the hope of reselling them for profit. Once you have decided on a domain name, then figuring out if it is available or not is thankfully an easy task as every single name registrar allows you to check if it is available.
• There is a lot of websites that offer domain name registration, all trying to compete on which one is the most flashy. They all seem to have tons of different options or programs which adds to the confusion.

I ended up settling down for Network Solutions for the domain name of the company (pongasoft.com) because it had been recommended as being serious, they are offering a privacy feature (to not expose your private information in the whois database (albeit for an extra $10)) and I could see the potential of using their web hosting solution and 'Servlet' support. Clearly they are not the cheapest one, but I needed to start somewhere.

Once I had a domain name, I started looking for web hosting. I think it is probably even worse than domain name hunting as the variety of what is offered is way more diverse. On top of that, most of them adding to the confusion by bundling domain name for free (under some conditions). In the end, for me it was pretty difficult to choose also because I was not too familiar with what the terminology meant and what you could and most importantly could not do unless you actually tried it. It would definitely have helped to be really crisp on what I really wanted, but I wasn't sure at the time.

I decided to try with the web hosting solution provided by my name registrar (Network Solutions) because it offered blogging and 'Servlet' support. After giving it a shot for a while, I realized that 'Servlet' support is really a joke. First of all, finding documentation for it was a big challenge. But once I was able to experiment with it, it became clear that it was going nowhere: the only thing you could do is literally create a Servlet (= a class) and drop it in their web container... You just cannot deploy a war, so as a result you cannot have any dependency on anything that is not already in the container. Very very limited in what you can actually do.

The second one I tried was Oxxus.net. They offer your own private container which you can start and stop at will. They have different kinds of web hosting, some of them including blog, but the one that offers tomcat does not offer blog. It was not too much of an issue since you can install a blog application in tomcat (the blog you are currently reading is running on Apache Roller within tomcat :) ). What really ticked me off is how lax they were with security. First of all, the very first email they sent me after registration contained passwords in clear in the body (and we all know how secure email is...). Although not a very good practice, it is not too much of an issue if you can just change them yourself, which I did. To be able to use scp or sftp, they told me to login on their website with elevated privileges (as a non chroot jailed user) and to my surprise to do that I needed to use the same password that they had sent me earlier on, even after I had changed it on the website. In other words, they have a 'backdoor' which is totally accessible with passwords sent in clear that you cannot change... that was enough for me.

Finally the one I settled with and the one currently running this blog is RimuHosting (I found a lot of good comments on the web about them). I was very impressed right away by how much they care about security and how good their documentation is. Just check out their HowTo section and you will understand what I am talking about. What I have now is my own private VPS (Virtual Private Server). So I can do whatever I want on it. Install and run whatever kind of program. No restrictions. I must admit that I was a little bit afraid at first to have to manage a full blown OS since I am not a system administrator. Nonetheless, with the help of their very down to earth documentation and the Webmin interface, it turned out to be not a problem at all. Their support is very good too and they can help you set up whatever you want. I am currently very satisfied with this choice.

I investigated AWS (Amazon Web Services) but it is definitely a more expensive solution if you want to have a server up and running all the time (it costs 10c an hour... hence about $70 a month if you run it all the time). I think it is more suited if you can bring it up and down whenever you want to use it, but not if you want to run a blog or an email server which have to be up 100% of the time.

I recently bought another domain name for my product and this time I went with a company called Gandi. They were recommended by RimuHosting and they offer the same privacy protection offered by Network Solutions, except for free...

As a conclusion, to be fair to the companies that I have tried for web hosting, they all offered a 30 day money back guarantee. And it really works! I had no problem getting my money back. So I think it is ok to experiment if you don't know too much what will work for you and what will not. Usually it does not take that long to figure out whether you are happy or not.

Grails - Proper shutdown in dev mode

For my main project I am using Grails for the front-end (I will relate in an upcoming post why I chose this technology in the first place). Grails has this very interesting development mode which allows you to continue working on your application and see the changes right away. To start the application you usually issue the command:

grails run-app

or if you use maven

mvn grails:run-app

To shutdown (for restarting for example), you do a CTRL-C which terminates the process. Grails uses the Spring framework to bootstrap your application. It also allows you to define your own beans. However, I noticed that when terminating the application, the beans that I had registered with a destroy-method were not being properly shutdown (the destroy method is simply not called). I tried to find a way to change this behavior by default but did not find anything. I then implemented my own shutdown solution in this manner:

I created a simple class in grails-app/utils/com/mypackage/ShutdownHook.groovy which registers a VM-wide shutdown hook when it gets called by Spring (ApplicationContextAware)

package com.mypackage

import org.springframework.context.ApplicationContextAware
import org.springframework.context.ApplicationContext
import org.apache.commons.logging.Log
import org.apache.commons.logging.LogFactory

public class ShutdownHook implements ApplicationContextAware
{
  public static final Log log = LogFactory.getLog(ShutdownHook.class)

  public void setApplicationContext(ApplicationContext applicationContext)
  {
    Runtime.runtime.addShutdownHook {
      log.info("Application context shutting down...")
      applicationContext.close()
      log.info("Application context shutdown.")
    }
    log.info("Shutdown hook setup...")
  }
}

Then I added the following block in grails-app/conf/spring/resources.groovy which conditionally creates the bean only in development mode (thanks to groovy Spring DSL!).

if(grails.util.GrailsUtil.isDevelopmentEnv())
{
  myShutdownHook(com.mypackage.ShutdownHook)
}

It works really well as my beans get properly destroyed when the application terminates. Nonetheless it would be better if it was part of the Grails framework by default. I opened a Jira (GRAILS-4404) ticket for it.

pongasoft.... a new adventure

One thing that I have been wanting to talk about is the 'story' behind pongasoft. It all began during the holidays 2008 when I realized that I had been working for LinkedIn for a rather long period of time (you could say that over 6 years is an eternity in startup lingo ;) ). LinkedIn is a cool company, don't get me wrong, and the technologies behind are very cool and also bleeding edge (OSGi for example). Nonetheless, you are still pretty constrained in what you can and cannot do for practical reasons. Also in 6 years, things have changed quite drastically in some areas: for example, web hosting is widely available and affordable, cloud computing is changing the face of how you can bootstrap an idea and be able to scale it with little investment upfront, new (dynamic) languages like Groovy and Scala are changing the landscape and offering new ways to implement ideas.

While brainstorming with my partner I had an idea for a tool that I wanted to build and decided to give it a shot. The purpose would be to essentially start from scratch with no preconceived ideas and try to use as much open source software and new technologies as possible. I have been very lucky that LinkedIn allowed me to work part-time since it allows me to concentrate on my own project while still having a portion of my income. I have been working on the tool for about 3 months now and it has been an incredible learning experience: from domain name purchase to cloud computing (barely started yet), the gamut of technologies involved is pretty large. I think the main drawback is that the progress is slower than I was anticipating and can sometimes be frustrating, but the journey in itself is awesome.

I am planning to share a lot of this experience in subsequent blog posts and of course release the [beta version] of the tool at some point in the near future for feedback. I am excited to see if people would be interested. If not then I won't be regretting a thing as what matters to me the most is the adventure.

Welcome to the software cookbook!

Hello kind reader. I guess it is tradition to introduce oneself as the first post on the blog. So here I go. My name is Yan Pujante. I am originally from France and currently living in the Silicon Valley close to San Francisco. I am a software engineer at heart and have been since I was 11 years old... a long time ago :). If you are interested to learn a lot more about me, you can always check my fairly complete profile on LinkedIn.

In this blog I am planning to write mainly about software, with an emphasis on sharing my experiences with the new technologies I am working with both at LinkedIn and pongasoft. I hope you find it interesting. Don't hesitate to leave feedback!