Saturday, 27 September 2008

music brain - part 1

I sometimes feel as though I am so focussed on one particular thing (at the moment Arum DataEye) that my various other skills that I am not using so much at the time might become rusty.

As a result I have embarked on a little side project... Music Brain. It has probably been done before, but it is really just a programming exercise for myself more than a serious project.

Problem: The new Metallica album, Death Magentic, was released recently and I want to know how "Metallica" it really it is. My gut feeling has a human is that it's pretty Metallica. :) But I got to wondering if a computer could make the same judgement?

Solution: Train a neural net with my library of Metallica tracks, except for the current album. Then run the new album through the neural net and see if the program agrees that this is a Metallica album. I will also run a few other band's tracks through the neural net after training has been completed and see what it "thinks" about them. Hopefully the program should be able to differentiate between existing Metallica tracks and other bands, while being able to identify new Metallica music.

Approach: Somehow I will have to extract the music from my music library and feed it in to the neural net. How many input nodes will I need? How can I extract the raw music?

Today's first steps were to read up on the WAV file format. It is a pretty straight forward format and using the information on this page I was able to create a "WAVIterator" class which takes an input stream and using the Iterator pattern allows me to pull "MusicSamples" out of a WAV file. In my API a "MusicSample" contains a one or more channels of data for a very small slice of time.

I have placed some restrictions on the files I am going to be able to parse in order to make life easier. Basically the WAV file must have been recored with 16 bits using PCM (Uncompressed) at 44100hz. I figured this is close to CD quality and good enough for the human ear so it would be good enough for this program.

The resulting test program was an interesting and humbling reminder about how slow using an unbuffered FileInputStream really is. My test code was taking nearly 100 seconds to parse the WAV file which was a little disappointing. So I decided to load the whole file in to memory first and use a ByteArrayInputStream instead. Wow! This reduced the parse time to just over 4 seconds, much more like it.

However there are a number of problems with this approach. Firstly, what if the file size is greater that Integer.MAX_VALUE? i.e. larger than the maximum size of a primitive array in Java. This is unlikely to happen since Integer.MAX_VALUE is 2^31 - 1, or 2147483647 (i.e. about 2gb), but that leads on to the next problem, you're soon going to run out of memory if you create 2gb byte arrays. It just isn't good practice to make the assumption that your files will be small enough and/or risk trying to allocate an array that big.

Of course, after a minute's pondering I remembered that there's a class just for this type of occasion; BufferedInputStream. So I replaced 9 or 10 lines of code and wrapped my FileInputStream in a BufferedInputStream and the performance was just as good. It feels good to get back to basics sometimes.

My next tasks will be to work out how to create a neural net in Java with appropriate inputs and outputs for my extracted music data. I don't really want to code a whole neural net, so I will probably use an existing framework... more next time.

Thursday, 11 September 2008

Arum DataEye

In work related news, I have been continuing work on Arum DataEye, our near-real time management information dashboard built on Solstice.

Our first customer is getting ready to go live soon and now Vince is back from his medical leave the team is back at full strength and eager to get DataEye delivered. I'm sure there will be a case study when that happens, so watch this space.

Below is an extremely simplistic Flex application that I put together which helps to demonstrate what Arum DataEye is essentially about. (Sorry about the size)

Arum DataEye has a customer and technology agnostic data repository which is easily extended through our pluggable administration interface allowing us to integrate DataEye with any host system you care to throw at us. Our consultants are experts at analysing host systems so that Arum DataEye can make periodic requests for live data without impacting operational performance. The data is retrieved, stored in our data repository and then displayed on a user's dashboard on demand using Flex, providing a rich and interactive dashboard for monitoring the performance of everything from user and team based call centre statistics to host system uptime.

While Arum DataEye is primarily a tool for getting your management information now, future versions of DataEye will also support forecasting, allowing you to make predictions and pro-actively change your strategies in order for your business to satisfy its key performance indicators.

If you'd like more information about how Arum DataEye can unlock your data now, instead of next week or next month, feel free to contact Arum by clicking here.


Google Chrome was released earlier in the month.  At first I greeted the news with trepedation as all the world doesn't need is yet more browser chaos.  

However, I was really impressed by the comic style of the documentation.  I was suckered in and decided to give it a try.

First impressions were good - it starts up fast.  I'd say about 3 times faster than Firefox 2 just based on casual observation, but about the same as Internet Explorer 7.  It feels fast when I use it and it's responsive, more so than Firefox, but again about the same as IE7.  It feels clean cut and modern, compared to Firefox 2 which now feels clunky and I never liked IE's interface anyway.

However, after using it almost exclusively for a while there are a few things about it which annoy me.  

Firstly, I admit it, I'm a Facebook user (don't bother adding me, I only add people I know in real life).  Chrome is hit and miss with Facebook and seems to only work about 50% of the time.  I've been told this is because Facebook delivers Ajax/Javascript appropriate to the browser being viewed and doesn't know what to do about Chrome.  I'm not convinced since it seems to work some times and not others.

Next up - Google's own homepage doesn't even show up right.  The radio buttons to allow me to choose to search my location (UK) just aren't visible, though they do seem to be there.  In fact, radio buttons ahve gone missing all over the Internet, from what I can see.   Then the biggest shock was that right mouse button support in Google Spreadsheets doesn't work at all when trying to get up a context menu for the rows and is fiddly at best on the cells.  It is a little shocking that Google's own applications don't work with their very own browser, it would seem logical to me that is where they would start.

I wouldn't have minded so much except that the marketting propoganda documentation states that they have a massive server farm testing all the most popular sites and have this almost miraculous approach to working how a web page should "conceptually" thus reducing the number of human testers required.   Smells of BS to me.

Next is how they say they've "redesigned web browsing" from the ground up.  Well not really, all the usual things are there... well nearly.  Most things you take for granted in Firefox (or even IE) are generally there - if you can find them.  I've not really noticed anything special that Chrome does the other browsers don't, apart from ...

Process per tab.  This is actually really good and makes Chrome feel really stable.  In fact, Chrome probably uses more memory than Firefox for the same kind of browsing, but the fact that when I close a tab I know I'll get that memory back (and it really does seem to work that way) makes me feel all warm and fuzzy inside.

Finally, Webkit.  A few people I know go on about how cool this is - I don't see it's much different from Firefox 2's rendering or even IE's.  I'm still trying to work out what makes it "cool".

So in conclusion, I'm still using Chrome primarily, but sometimes I really have to switch back to Firefox 2 to do some stuff (editing spreadsheets and adding friends in Facebook mainly).  I can only imagine that it will improve over time, though I am slightly worried that I am still "up to date" even after using it since it came out.  Perhaps the rumours about automatically downloading self-updates are true ... ?  

But anyway, if you haven't tried Chrome yet I'd recommend it so long as you remember that's still very new compared to Firefox 2 and especially Internet Explorer.