TDD FTW! - Android Cineworld App fixed in seconds

That is, Test Driven Development, for the win! =)

I just received an email from a confused user of my Cineworld Android application (UK only) telling me that it wasn't working. I was especially grateful though, as most people just change their rating to one star and leave a comment, but the Android market place doesn't push those comments to the developers.

Anyway, when the application loads for the first time it queries the Cineworld website for a list of cinemas, but in this case an error was occurring so the application was popping up a message saying "Unable to connect to server. Check your settings and try again later.".

At first I didn't think I had time to fix the problem but then I remember I had a bunch of unit tests for the application. So I loaded up the workspace and ran the tests and sure as hell, one of them failed.

But this was fantastic!

I could instantly see that the Cineworld's home page format had changed slightly and that might my app was unable to find the indicator it was looking for in order to extract the list of cinemas. Thankfully, it was only a change on the home page (they have installed some fancy-shmancy Javascript thing) and the 'cinemas' page still contained the list required by the application, so I only had to update the URL I was using, re-run my tests to verify and voila, a fix!

No complicated debugging session, or log viewing or anything like that. Just run the unit tests, make the fix, verify with the unit tests and job done - just the way it should be.

Agile OSGi with Eclipse and Nimble

There's a view that doing things 'the OSGi way' adds overhead. In fact, I have even been told in the past "don't worry about the modularity thing just now, just get the demo out of the door".

However, for me, working with OSGi is a no brainer, and adds virtually no overhead even for the smallest projects. Additionally, I find that not only is it sensible to start with OSGi, but doing so speeds up my ability to deliver quality modular software.

There are a couple of tools that help make life easy for me (as it should be for a developer) so that I am not wasting time with my environment. These tools are Eclipse and Nimble.

Firstly, my IDE, Eclipse. I give it a hard time sometimes but it does let me get on in what I feel is a very productive manner. Eclipse 3.5 is generally great and has even more support for OSGi and my personal favourite is the support for declarative services - the editor making it very easy to spit out the required XML making it really quick to build a component based bundle in hardly any time. For those 'small' projects that people consider an overhead, I still recommend this approach, as it is noddy to add a main method (either to that component or another class which uses that component) and you still end up with a jar file you can execute from the command line, especially if you add the Main-Class attribute to the manifest. At the end of the day we're still talking about simple Java objects, but with a little bit of extra metadata.

Finally, the key to clarity when developing OSGi apps in Eclipse is 'targets'. So for each workspace I would create a meta-project that contains an Eclipse target and any top level external dependency bundles. The target references these bundles and when I apply the target to my workspace I now only have visibility of the exports from the bundles in the manifest editor, and those plugins in Eclipse's plugin perspective. All the noise from the hundreds of bundles that Eclipse uses to run is gone.

(For me, the only thing that lets Eclipse down is that I feel it could be more defensive about how it handles plugins which don't always play nicely)

Until recently I made the framework I was deploying to a big part of the development architecture. In fact I would write an ant script to build my bundles, the framework, it's dependencies and then deploy my bundles in to that framework and then run it to do my integration testing.

While doing a development cycle of code/compile/deploy I would tend to have Felix running with fileinstall configured to look at a folder called plugins and then use Eclipse's export tool to drop new versions of my bundles in to that folder. Fileinstall detects the new bundles and adds/updates as appropriate. But what about dependencies? They would have to already be installed in the framework. For for each external dependency I am using I would have to resolve that bundle's dependencies manually... if those bundles have dependencies and so on, this soon becomes long winded and tedious!

These days I am much more agile about this and use 'Nimble', which is OSGi resolver technology from Paremus. To use Nimble, you simply install it and you get 'posh' - the Paremus OSGi Shell, another clever piece of technology that can save a lot of time.

So if I wanted to test my application on Felix I would start posh as follows:

posh -C -F

The -C parameter clears the cache and downloads dependencies from scratch. -F specifies that I want to use Felix as my framework. In a few seconds I have a running Felix OSGi framework and a sophisticated OSGI shell but not much else. (At this point the docs on posh are really useful to see what you can do.)

Once I've indexed my bundles I create a nimble rule which defines a dependency on the activated state of my bundles and osgi services and save this in an xml file of it's own to the plugins folder as well:



Using the command 'nim add app/myapp' my bundles are installed, resolved (including dependancies) and started.

The last thing I have to do is tell the HttpService what port to start port:

setpid org.apache.felix.http org.osgi.service.http.port==8080

but I only have to do that once per Posh session.

The key thing here is that all my dependencies are resolved for me. I don't have to do anything except worry about my production bundles.

Then when I want to distribute my application to colleagues or the customer or deploy to another environment, I just distribute the bundles and my repo xml files, and they fire it up with minimal effort. Glorious!

Please note, that the above does not necessarily constitute best practice. I have been chatting with the Paremus guys about best practices and will follow up with information about that at a later date.

2010 and me

Firstly, welcome to 2010 everybody!

I don't know about you, but I'm kind of disappointed we haven't had any manned missions to planets in our solar system. By this stage in Arthur C Clarke's books we were on our second trip to Jupiter. Oh well. :)

So what does 2010 have in store for me?

My focus is going to stay on OSGi, of course. In January I am attending the OSGi UK Users Forum, again as secretary. The UK forum has been going for nearly a year now and has some impressive growth and all the attendees are great people with lots of interesting stories to share, so I'm looking forward to that one.

I am also attending Jax London (and OSGi Devcon) - last March I attended The Server Side conference in Las Vegas which was excellent, but frankly a little expensive in terms of cost and time (and environment?). Better all round to support local events, I feel.

I'm still loving Flex and db4o, so plan to keep using those technologies this year, but expand my view in to other OODBs (e.g. Versant and InterSystem's Caché) and rich internet technologies (e.g. Apache Pivot - Java RIA platform, Vaadin - HTML/CSS/Ajax RIA platform. Flash is great, but there is a lot of leverage to be had from 'pure browser' RIAs).

Actually, I've been very quiet on the Apache Pivot front this year, but graduation is afoot and Pivot is maturing nicely, being used in a couple of production environments I believe. While I'd love to donate more time to this, I would be well just donate a little as I have been far too inactive of recent times. I had starting building a an Eclipse plugin to preview Pivot views, so I will discuss with the community how I can enhance this further.

Taban, my REST and OSGi based JSON storage application is still floating around in my mind, but like everything else, it is just a matter of finding the time to finish it off and then find an appropriate place to host it.

I'm still enjoying Android as a platform, but feel it is a little too much in Flux. While 53% of all handsets are running Android 1.6, about 25% are running Android 2.0 or greater and I feel like I shouldn't invest too much time in an SDK which will soon become obsolete.

On the Google front, in May my wife and I are attending Google IO in San Francisco (at our personal expense), though I am sure she is less excited about it than I am. ;) Last year attendees were presented with an Android phone. I can only hope we are presented with something as cool as that, my wildest dream being handed a Google Tablet. We shall see!

With regards to Arum, before Christmas I worked on a very interesting market data platform based on OSGi, Flex and db4o before Christmas and am hoping to have further input in to that. The potential for extending the application is great and has been helped in no small part by the flexibility of the technologies involved and by instigating a modular architecture from the beginning. I used to think that RAD projects should be considered 'disposable' as the speed at which they were built usually meant you were doing something wrong, but by using OSGi, Flex and db4o we reduce time to market for a functional (though not feature complete), rich application, to just over 3 weeks worth of work between two people (myself and Vince). The current application is being demonstrated to the community that will adopt it and I am sure that more functionality will be requested in the new year.

I am also hoping to start on a new version of Arum DataEye, taking lessons learned from the last two years and building an even better management information dashboard, considering software as a service as well as customer site options for deployment. With 2 major sales under our belt and nearly 300 active users, we have excellent feedback with which to make the new version of DataEye even better.

At Arum, we have also been considering a brand new product, so I am looking forward to working with the team to flesh that out, get it built and make it available to the masses. More information on this as it happens, but no doubt will be built on and driven by the excellent technology we've had much success with so far.

I also have plans to build an Eclipse based IDE for rapid development of modular applications based on Flex, OSGi and db4o. This tool will reduce time to market by a massive amount, allowing Arum to produce the majority of a functional application in a fraction of the amount of time, drawing on our library of OSGi bundles that we've developed, and upon those distributed in remote repositories such as Spring's OBR and Maven. Again, watch this space ... but don't hold your breath. :)

With Arum's focus on building low-cost, high quality, open source based, rich internet applications and products, it is a pretty exciting place to be.

Towards the end of 2009, I started doing a little freelance work with Paremus, the makers of Service Fabric (fantastic distributed OSGi cloud platform), Nimble (excellent OSGi resolution tool), Posh (Paremus OSGi Shell) and Sigil (OSGi development tooling, now part of Apache Felix). These are a great bunch of really clue up guys, and I hope to continue working with them in to 2010.

A more general comment about consumer technology; my prediction for this year is that 2010 will be 'the year of the tablet' with Apple (OSX or iPhone OS?), Google (Chrome OS probably) and various Android based tablet products on the horizon, we could be entering a new era of always on and always connected lifestyle. While tablets have been around for sometime I believe that the 7" or 10" touch-screen internet tablet still has room for impacting in popular culture and that 2010 is the year for that. Expect to see people carrying these around commonly, much like the e-book readers that I regularly see fellow commuters using, but accessing electronic magazines and other web applications.

All in all, I am sure that 2010 is going to be an exciting year, in my work, personal projects and with regards to technology in general.

Introducing Taban - The JSON REST Database for OSGi

I've been toying with the concept for a while that some web based applications (call them RIAs, Web 2.0, whatever) don't really need much in the way of 'server' infrastructure, and that a lot of the time they just need somewhere to persist their data. It is easy to make this complex but since the UI often knows the model it wants to use for persistence, it could be a lot easier. Even if they do need something complex rapid development of UI prototypes often exclude the persistence layer.

Here is where I introduce 'Taban - The JSON REST Database for OSGi'. However, at this stage I don't know if I'll actually use it, or if anyone else will find it useful, but I am also using this to 'practice' my OSGi coding (not that I don't do enough) and to play around with some web client technologies, namely Apache Pivot (I am a PPMC member and minor committer) and GWT (for no other reason than just because, really).

The idea behind Taban is that an web application will simply use HTTP methods (GET, PUT and DELETE) to perform the usual CRUD operations required for persistence, but using JSON as the data exchange format. In fact, the default implementation of Taban will pretty much store the JSON as is in to a db4o database, interrogating the JSON only to allow 'indexing' of the objects for the purpose of querying/filtering.

Alternative implementations could convert the JSON to a real object model and store that, again either straight in to db4o (which is the absolute easiest option), or if they really want the pain, in a RDBMS via some ORM technique. Taban will facilitate all these options, but will be fully useable 'out of the box'.

Since REST is more of a concept than a specification I have designed the following approach to URI handling, but please consider that this is still a work in progress:

GET - If the URI ends in a forward slash, the default action is to return a list of children as a JSON array. However, if the URI does not end in a forward slash the JSON content at that location (if it exists) will be returned directly.

PUT - If the URI ends in a forward slash the JSON content will be inserted and an ID will be automatically generated. ID generation is plug-able, but the default will be to use an integer based auto-increment for that particular URI. If the URI does not end in a forward slash the JSON content will overwrite any content at that address, if it exists, inserting it otherwise.

DELETE - Only URIs without a forward slash are supported by the DELETE method.

POST, TRACE, OPTIONS - Not likely to be used, though I am toying with the idea that a POST might support 'updating' the JSON at that location, rather than replacing it, but how and what to update at this point doesn't seem trivial to determine, though I have done similar work on this on Apache Pivot's resource handling which is JSON based.

In addition to the HTTP methods listed there will be a number of additional HTTP headers used in order to allow more fine grained control over the response as well as facilitating filtering and limiting of results.

Taban is (will be) implemented as OSGi bundles and will call out to/depend on a number of OSGi services, namely:

• Configuration Admin (optional)
• Event Admin (optional, but will allow other bundles to do stuff with the JSON that is being stored)
• HTTP Service (required)

It will also depend on:

• db4o to provide the default implementation of the database, though it will be designed such that any persistence layer could be implemented and replace this dependency
• Arum Glue, an interface/convention based Inversion of Control and Dependency Injection manager
• Jackson JSON, which is a fantastic and super-fast JSON library

Finally, securing the database is also pluggable but Taban will not be shipped with an authentication module, at least initially. I envisage it would be very easy to add BASIC or Digest authentication support, but for the purpose of getting Taban off the ground this is a low priority for me.

So, applications served from the web which have a rich client (by rich I mean most or all of the functionality is loaded in to the client/browser as opposed to a click-request-response approach) can use Taban to persist their application model data with very little effort in terms of setting up 'backend' services, and yet Taban will allow developers to make the server as complex as their business requirements dictate. We make no distinction between these front end technologies, so long as they can support a full range of HTTP interactions (sorry Flex people).

At this stage, it is likely to be a programming exercise, so I am very interested to hear from people if they think this is something they might find useful. If there is enough demand I will probably push my work to github.com.

Oh, I should also point out that Taban is a celtic/gaelic name that means 'genius' - not blowing my own trumpet, but more of an expression of the simplicity and flexibility of the product.

Weak References in Java

There seems to be confusion about when weak references should be used. I'm going to present two cases - the first one is when NOT to use them, then second case demonstrates what they should be used for.

Don't use weak references for listener management

So firstly - don't use weak references for handling 'listeners'. Say you have an object that allows listeners to be added to it, it may occur to you that those listeners might never get removed. The reason they don't get removed is :

a) because they aren't supposed to be

b) the programmer is lazy and/or doesn't understand 'lifecycle management'

It would be easy to think that storing the listener as a weak reference would avoid these potential memory leaks but what actually happens is that you introduce seemingly random behaviour to your application, since the listener could be garbage collected *at any time*.

The solution is - learn how to clean up after yourself.

Do use weak references for objects you don't care about

For instance, let's say your application exposes a front end to a database. The user is traversing your application instanciating objects probably based on data loaded from the database. In order to speed things up a little you might keep a handle to these objects for future reference, but actually you don't care if the object is available or not because you can reload it from the database as when it is needed again in the future.

What does that sound like ... that's right - a cache!

Weak references are ideal for this scenario.

Google Wave Robot - Twitter Wave Bot

Just a quick one.

I got my Google Wave invite the other day. I'm enjoying it, but it is a little bit too quiet for me to use productively at the moment. When I get some invites myself, I'll send them out to the people who've asked for them.

What I really wanted to post about was the robot I wrote. I was quite impressed by the easy of the API, and the integration with Google App Engine, which makes writing a robot really easy to do. My robot, Twitter Wave Bot, takes #hashtags and @usernames and simply links them up to twitter. It took me about 8 hours to implement and you can see the code here:

http://code.google.com/p/twitterwavebot/

To get this robot working, add 'twitterwavebot@appspot.com' to your google contacts, and then add it to any wave and it'll start working for you straight away.

Arum DataEye™ as a Service

I've been fairly busy recently with a couple of exciting activities.

Firstly, in my spare time I have continued to increase my experience with the Android platform, but more Android stuff can be found on my Android projects home page. I plan on submitting at least one, possibly two applications, to the Android Developer Challenge.

For my paid job, at Arum I have been exploring the possibility of using DataEye in a hosted environment. Symetriq.com were kind enough to allow us to Beta their new Cloud product and after a minor false start, which with the help of Sytemtriq I got over quickly enough, it didn't take long to get DataEye up and running on their cloud infrastructure.

You can see DataEye in action right now by clicking here:
Arum DataEye Demo

Once the application loads you can login with the user "E:Benson" and password "p" but without the quotes.

DataEye as a hosted service presents both an opportunity and a challenge.

For organisations who are light on infrastructure or internal IT support the hosted solution means they don't have to worry about looking after DataEye, backing up the data and those other IT tasks, reducing the cost of ownership considerably. However, the challenge is that we still have get their management information data in to DataEye.

To address this challenge I was able to get even more leverage out of the fact that DataEye is based on OSGi. DataEye is constructed using what is known as "the whiteboard model". Simply put this means that components register services in preference to looking them up. This level of de-coupling then makes it easy to replace services with alternative implementations.

So to cut a long story short, I created a bundle that can accept DataPoint information over the web (via HTTP) as JSON. The bundle decodes the JSON and then inserts the objects in to our embedded db4o via the regsitered DataPointRegistry interface.

On the other end of the wire, which would be in the customer's infrastructure, I created a minimal OSGi environment using Equinox and then installed a bundle which registers an implementation of the DataPointRegistry interface which is responsible for pushing the data to the remote DataEye server.

By using the same interface on the server and client, we can then build integration bundles on customer site as if DataEye was hosted locally. If we later decided to move DataEye in to the customer's environment the same integration bundles will work without *any* change, because they are talking to the same interface.

At this point it feels appropriate to talk about Distributed OSGi. Distributed OSGi is a change to the current specification which lays out my approach above in a standard and container supported way, but at the same time without enforcing any particular transport protocol.

The main advantage of the new specification is that in my DataPointRegistry changes above I have hard coded the ability to receive JSON data over HTTP. With the new specification I would simply define the interface and drop in a provider which can do the 'remoting' for me. While the above change only took a couple of days to implement, the new specification would reduce that to almost nothing.

Once the new specification is widely available and in use, we will move DataEye to use an OSGi server based on the latest OSGi specification, which then also gives us access to a raft of new, enterprise focus, features in the OSGi container.