Indexing android 'froyo' javadoc in kiwidoc

I just indexed android 'froyo' javadoc in kiwidoc and it was quite a challenge.

1. Downloading the source code

You have to go to http://source.android.com/ to find instructions on how to get it. This part is actually quite challenging because there are lots of steps to simply 'get it'. I understand why it is being done this way: you end up with a ready to develop/contribute environment. On the other end, if you simply want to look at the source code it is quite complex and potentially time consuming.

2. Downloading the 'correct' version of the source code

I know it is written in the manual but it is not very intuitive to download the correct version. Under the cover, it uses git but it is wrapped by a high level command called repo. When you use plain git in a project, you simply clone the project and then you can switch to a branch. When you use the repo command, you end up with an error message if you try to follow the git paradigm.

> repo branches
   (no branches)
> repo checkout froyo
error: no project has branch froyo

It is very important to 'clone' with the correct version originally:

repo init -u git://android.git.kernel.org/platform/manifest.git -b froyo 

I am sure there are very good reasons to do it this way, but it would certainly be easier if the command were to behave like git.

3. Compiling android

I use a Mac OS X machine (Snow Leopard) and bumped into a lot of issues:

• Snow Leopard is not supported
• It requires a case-sensitive file system (which is not the default on Mac OS X)
• It requires Java 1.5 (which is eol now...)

Thankfully I had an old drive with Leopard on it and was able to eventually compile it using this version of the OS and creating a case-sensitive partition.

4. Make

For my own purpose I actually only needed the javadoc and the java source code. After running a full build (I just issued the make command) which took several hours to run (no kidding), there was no javadoc. Not being very familiar with make, it took me a while to figure out that you can run the command:

make -p > android-make.db

which "Print make's internal database". In this case it generated a 156MB file! Sifting through this file, I was able to identify the command I needed:

make offline-sdk-docs

which generated the javadoc.

5. kiwidoc generation

By slightly tweaking the javadoc command in the make file, I was able to generate the information I needed for kiwidoc: mainly all the source files to take into account as well as the classpath. Once I had this information, actually indexing it tooks 20s.

6. Android versions

I think that the android version naming, although I understand why it is being done this way, is quite confusing:

• Platform version (2.2 is the 'current' latest one)
• API Level (8 is the 'current' latest one)
• Code Name (froyo is the 'current' latest one)

Code names are cool but it make it very hard to remember to which actual version they tie to: the Android API Levels page does not even mention codenames (so how do you know which version is 'eclair' for example ?). In kiwidoc, I chose to use the api level as the version of the library because it is what reflects what you program against: there could be a new platform version (ex: 2.2.1) without a change in api level. What matters from an application developer point of view is the api level.

7. Conclusion

It took me a really long time to index froyo (api level 8) but I succeeded which is very satisfying. I also 'promoted' the android platform all the way to the home page.

pongasoft presents... kiwidoc

Back in April, I introduced the new adventure I was embarking on with pongasoft. It has been quite a ride but I am very proud to finally be able to release the service: it is called kiwidoc. You can find more information about the service itself on the About page including a description of all the main features that I was able to implement for the beta release. Here is the introduction:

In a few words, kiwidoc can be described as javadoc on steroids. The main goal is to help software developpers quickly find the information about java libraries in a single location:

• proximity search and typeahead allow you to quickly locate what you are looking for.
• IDE-style display shows you the relevant information in a familiar format.
• immediate access to additional information such as library dependencies, manifest, OSGi headers, etc.
• the private view can provide even more details if you need to dig deeper (in order to, for example extend a library or better understand its internals).

Now that the kiwi has been unleashed (there was a little clue on this blog about the name/logo :)), I will be able to talk more freely about the technologies and some of the issues I faced during the development. There is already some information on the Credits page about all the open source libraries I used. Of course due to the amount of work and the fact that I already have a full time job, I was unable to implement all the features I wanted to include in the beta launch, and I have many more coming up to improve the service. But it will also depend on the feedback I am going to receive. So please, leave feedback (on this blog or at feedback@kiwidoc.com), even if you hate the product and you think it is a stupid idea. Finally I wanted to particularly thank my partner, Markus Tuberville, for the awesome logos and icons that he created for the website (including this blog).

CSS for the UI design

In a previous post, I was talking about using Grails/Groovy to build the front-end. The final rendering needs to be made pretty/attractive. The old days of using tables and spacer gif to have the page looks good have thankfully been replaced with CSS. Not being a web designer myself, I had some very rudimentary notions of what CSS is. I used it a little to build/tweak the blog you are reading, but I must admit I was kind of flying blind as I did not understand it fully nor did I know enough to be efficient. The result although looking nice (at least to my tastes :)), certainly left to be desired and there are many things I wanted to do that I could not achieve (just check how the right sidebar seems to be detached and disappears if the window gets too small...). I hunted for books about CSS and ended up buying 2:

CSS: The Definitive Guide: the title essentially sums it all. It is a definitive guide, pretty hard to read cover to cover (but to be fair, not designed to be read this way). It is kind of a shame that the book is not in color which makes it hard to follow the examples especially the one using colors... There is an entire chapter about selectors and I think it is pretty useful to grasp the concepts.

Bulletproof Web Design: Improving flexibility and protecting against worst-case scenarios with XHTML and CSS: I love this book! It is very well written and progressive (so it is perfect for beginners). Most importantly it gave me a starting point to work on the front-end of my application. There are a million different ways to achieve the same effects or results and when you are new to a technology it is really difficult to know which way is better than the other (or even find a way to start). I think this is where this book shines. The angle chosen is to design your html (or xhtml) and css in a way that will accommodate changes (like your text span 2 lines instead of one because it is longer than anticipated) and make it useable even for a user who does not have images or css at all! The reason why this book really grasped me is because it is very close in spirit to what I normally do when I write good object oriented code: there are techniques that I learned throughout my career to make my designs flexible and bulletproof (meaning it will accommodate changes without having to start from scratch) and this is exactly what this book teaches you for the front-end. A must read!

git for source control management

When I started working on my project, I needed to setup an scm (source control management). One of the goal of the project was to look at new technologies and trends and see how well they can solve some issues I have encountered throughout my career. I am familiar with 2 popular scms: cvs and svn.

At LinkedIn, we started with cvs. Although being widely popular, it has several shortcomings, mainly non transactional commits and very heavy branching / tagging operations. Transactional commit is quite important as it insures that whenever you checkout code, you will always get a consistent view and not a partial commit that somebody else is currently checking in (or even worse if two people are checking in at the same time). "Consistent" is obviously from the point of view of the scm, as a developer may still check in an inconsistent set of files by forgetting to check in one of them for example. cvs fails in that regard as there is no consistency guaranteed. Branching and tagging in cvs is painful because it happens at the file level, so the bigger the project is, the longer it takes.

With a rapidly growing project and team, it was becoming unmanageable with cvs. We then moved to svn which was solving the two main issues I was talking about: with svn, commits are transactional: every time you commit you actually have an ever increasing commit number which refers to the entire commit. Either the entire commit will go through or not and another developer will never see an inconsistent view. Branching and tagging are very lightweight operations in svn as it does not copy the entire tree.

After using svn for a while, although being an improvement over cvs I am still pretty unhappy with it as there are several big issues like loosing the history of commits (mild problem) to loosing changes when files are moved around on a different branch (really bad). Also, although branching is much easier/faster than cvs, the syntax to merge is quite complicated and error prone by having to constantly figure out the correct commit numbers to use. To be fair, some of those issues are being addressed in more recent versions (although it is still not there yet).

In my quest for something better, I started hunting around and clearly the new trend is Distributed SCM. Open source projects are the perfect example of highly distributed development environments and having a centralized scm (like cvs or svn) is showing its limitations. Hence the need for an scm that would handle this kind of projects. I believe that the most prominent distributed scms today are git and mercurial. Both projects were started around the same time as a free/open source response to BitKeeper which was not going to be free anymore. git was created by Linus Torvald for handling the Linux kernel development. mercurial was created by Matt Mackall.

On the paper, despite some minor differences the concepts are essentially the same. Not having any preconceived ideas with either of them, I decided to use git for my own project mainly because of the IDE support: it has very good support in Intellij IDEA. I am by no mean an expert in git but I can share my experience after using it for several months now.

The very first thing that I noticed right away and I am still blown away every time, is how fast it is. There is just no comparison possible with the others I was mentioning... Of course I am comparing apples and oranges because my project is by no mean the size of LinkedIn's source code, but even on smaller projects there is a clear difference mainly because there is no network access involved (and even when there is, it is still blowing me away).

Command line syntax is a lot easier to use than svn: creating branches, switching between them, merging,... is all a breeze. Nonetheless, there is something that takes time to get used to: the 'staging' area which is definitely not very intuitive at first especially coming from a different model. Let's take an example: you start modifying a file then you want to commit the changes. You must first move the file in the staging area by issuing 'git add <file>'. If before committing, you modify the file again, you must add it to the staging area again. In the IDE, you don't have to worry about any of this which makes it very transparent. On the command line, you just need to be a little bit more careful, but it has become a habit to always issue a 'git status' command which tells you the state of your changes and which ones need to be moved in the staging area.

I guess one might wonder if it is not overkill to use a distributed scm when I am essentially the only developer on the project! Actually the more I use it and the more I discover how beneficial it is even in this situation. I am actually writing this post from a meeting in south of France. I wanted to be able to continue working on the project even while being remote. My project is currently on my desktop at home so if I wanted to use svn for example, I would have to be able to somehow connect to my desktop to be able to continue working... But I don't have any of this issue with git at all: I simply cloned my repository on my laptop before leaving which you can do over ssh very easily as it is built-in: 'git clone user@machine:/path/to/repo'. I can then work on my laptop during the entire time I am away, creating branches, committing as many time as I want. When I come back I will simply issue a 'git push' command (yes that is it... I don't even have to tell it where to push to!!) to move back all the changes I have made onto my desktop with full history! Really neat! By the way since it is all local, it is also perfect for plane work!

On the mac, there is a very neat application called GitX which can help you visualize the repository and also move files in the staging area.

There is also a service called github that has been built around git to facilitate the creation of open source projects. By lowering the barriers to be able to contribute to open source projects, I think it has great potential to become quite a nice platform.

I have not tried mercurial at all so I do not have anything on my own I can share. I have heard people complaining about the fact that mercurial does not allow you to rewrite the history where git does and depending on which camp you are, it may be good or bad. I don't mind that git allows you to do that and if you don't like it then you just don't use the feature.

As I was mentioning previously, distributed scm is the new trend. After trying it for myself for a while I have become very excited about it and I really don't believe it is just a fad. My personal opinion is that it is the future. Even for non distributed projects (like mine at the moment), the benefits are obvious. I do not regret the decision I have made as it allowed me to see the potential of this emerging technology! Distributed scm is a brilliant idea and I invite you to try it out: once you make the switch you will not want to go back.

Starting from scratch... domain name and web hosting

When I started having the idea of a project that I could build, two things popped up right away: I needed a domain name and I needed some sort of web hosting solution. To be honest, both topics were totally brand new to me and I spent a significant amount of time investigating what is available.

Getting a domain name sounds like a trivial thing to do, but it turned out to be not such an easy task for two reasons:

• Finding a 'good' domain name is important and of course most of them are already taken. A big portion of them are simply unavailable because people owns them in the hope of reselling them for profit. Once you have decided on a domain name, then figuring out if it is available or not is thankfully an easy task as every single name registrar allows you to check if it is available.
• There is a lot of websites that offer domain name registration, all trying to compete on which one is the most flashy. They all seem to have tons of different options or programs which adds to the confusion.

I ended up settling down for Network Solutions for the domain name of the company (pongasoft.com) because it had been recommended as being serious, they are offering a privacy feature (to not expose your private information in the whois database (albeit for an extra $10)) and I could see the potential of using their web hosting solution and 'Servlet' support. Clearly they are not the cheapest one, but I needed to start somewhere.

Once I had a domain name, I started looking for web hosting. I think it is probably even worse than domain name hunting as the variety of what is offered is way more diverse. On top of that, most of them adding to the confusion by bundling domain name for free (under some conditions). In the end, for me it was pretty difficult to choose also because I was not too familiar with what the terminology meant and what you could and most importantly could not do unless you actually tried it. It would definitely have helped to be really crisp on what I really wanted, but I wasn't sure at the time.

I decided to try with the web hosting solution provided by my name registrar (Network Solutions) because it offered blogging and 'Servlet' support. After giving it a shot for a while, I realized that 'Servlet' support is really a joke. First of all, finding documentation for it was a big challenge. But once I was able to experiment with it, it became clear that it was going nowhere: the only thing you could do is literally create a Servlet (= a class) and drop it in their web container... You just cannot deploy a war, so as a result you cannot have any dependency on anything that is not already in the container. Very very limited in what you can actually do.

The second one I tried was Oxxus.net. They offer your own private container which you can start and stop at will. They have different kinds of web hosting, some of them including blog, but the one that offers tomcat does not offer blog. It was not too much of an issue since you can install a blog application in tomcat (the blog you are currently reading is running on Apache Roller within tomcat :) ). What really ticked me off is how lax they were with security. First of all, the very first email they sent me after registration contained passwords in clear in the body (and we all know how secure email is...). Although not a very good practice, it is not too much of an issue if you can just change them yourself, which I did. To be able to use scp or sftp, they told me to login on their website with elevated privileges (as a non chroot jailed user) and to my surprise to do that I needed to use the same password that they had sent me earlier on, even after I had changed it on the website. In other words, they have a 'backdoor' which is totally accessible with passwords sent in clear that you cannot change... that was enough for me.

Finally the one I settled with and the one currently running this blog is RimuHosting (I found a lot of good comments on the web about them). I was very impressed right away by how much they care about security and how good their documentation is. Just check out their HowTo section and you will understand what I am talking about. What I have now is my own private VPS (Virtual Private Server). So I can do whatever I want on it. Install and run whatever kind of program. No restrictions. I must admit that I was a little bit afraid at first to have to manage a full blown OS since I am not a system administrator. Nonetheless, with the help of their very down to earth documentation and the Webmin interface, it turned out to be not a problem at all. Their support is very good too and they can help you set up whatever you want. I am currently very satisfied with this choice.

I investigated AWS (Amazon Web Services) but it is definitely a more expensive solution if you want to have a server up and running all the time (it costs 10c an hour... hence about $70 a month if you run it all the time). I think it is more suited if you can bring it up and down whenever you want to use it, but not if you want to run a blog or an email server which have to be up 100% of the time.

I recently bought another domain name for my product and this time I went with a company called Gandi. They were recommended by RimuHosting and they offer the same privacy protection offered by Network Solutions, except for free...

As a conclusion, to be fair to the companies that I have tried for web hosting, they all offered a 30 day money back guarantee. And it really works! I had no problem getting my money back. So I think it is ok to experiment if you don't know too much what will work for you and what will not. Usually it does not take that long to figure out whether you are happy or not.

pongasoft.... a new adventure

One thing that I have been wanting to talk about is the 'story' behind pongasoft. It all began during the holidays 2008 when I realized that I had been working for LinkedIn for a rather long period of time (you could say that over 6 years is an eternity in startup lingo ;) ). LinkedIn is a cool company, don't get me wrong, and the technologies behind are very cool and also bleeding edge (OSGi for example). Nonetheless, you are still pretty constrained in what you can and cannot do for practical reasons. Also in 6 years, things have changed quite drastically in some areas: for example, web hosting is widely available and affordable, cloud computing is changing the face of how you can bootstrap an idea and be able to scale it with little investment upfront, new (dynamic) languages like Groovy and Scala are changing the landscape and offering new ways to implement ideas.

While brainstorming with my partner I had an idea for a tool that I wanted to build and decided to give it a shot. The purpose would be to essentially start from scratch with no preconceived ideas and try to use as much open source software and new technologies as possible. I have been very lucky that LinkedIn allowed me to work part-time since it allows me to concentrate on my own project while still having a portion of my income. I have been working on the tool for about 3 months now and it has been an incredible learning experience: from domain name purchase to cloud computing (barely started yet), the gamut of technologies involved is pretty large. I think the main drawback is that the progress is slower than I was anticipating and can sometimes be frustrating, but the journey in itself is awesome.

I am planning to share a lot of this experience in subsequent blog posts and of course release the [beta version] of the tool at some point in the near future for feedback. I am excited to see if people would be interested. If not then I won't be regretting a thing as what matters to me the most is the adventure.