At some point in recent weeks we had a server essentially just give up. After a little research we found it to actually have been a result of a series of java heap space errors. Fortunately we were able to rescue all of the important data. However, we have not been able to get the server up and stable. It’s been up a couple of times for a number of hours but each time the heap space errors would eventually come back.

I’m fairly convinced the heap space errors were the result of a scheduled backup which was running like a cron job except it was scheduled through a java API. More specifically the error was caused by an attempt to write too much data to an xml file from memory. The interesting bit about this is the server ran basically untended for nearly a year. No one checked on it regularly (besides the hosting service) to see the application was still behaving normally, for all I know no one even used the deployed application until a couple weeks ago when someone uploaded a fairly large zip file which the backup attempted to save as binary data killing the server.

Somehow it was determined only the user data actually got corrupted so we saved what we’re referring to as the program data and attempted to rebuild Magnolia’s jackrabbit repository without having an xml file to import. The only backup we had was the actual repositories directory structure. About here is where things start to get messy.

Because the only backup available was a complete jackrabbit repository directory structure someone had to go mucking about in it. The first important discovery I made was that I could remove individual workspaces and when restarted Magnolia would re-initialize only the removed workspaces. This was a great find because it meant it was possible to remove just the repositories related to user information and then when the server was started again Magnolia would use the bootstrap files to recreate them. However, a large amount of user data had been added since the last successful automatic user repository backup (remember how I said no one was checking on the server?).

I’m not totally sure where I’m going with this. I just needed to walk through it in my head and writing it down helped to do that. I may add more later if I need to straighten my head out again.

At work I had been working on a project for a rather difficult client whose name I will decline to mention. But with that work wrapped up in the past week I’ve been a lot calmer and a lot happier. It has also meant I didn’t have a lot of billable work at the office. As a result I decided it was time I got started relearning things I have forgotten due to disuse over the past fourteen months or so. I thought the idea was a good one because from the sounds of things we may have a number of projects coming up soon which will require interaction with a database, something I haven’t had to deal with since changing jobs.

I did remember how much easier Hibernate made things when I did have to interact with a database so I thought that would be a great place to start my foray into relearning database interaction. Also the last version of Hibernate I used was 2.x and they’re now into the 3.x and I wanted to take Annotations for a spin. Put succinctly, Annotations are awesome. It took me a little bit of digging in documentation to remember what all the interactions meant and apply that to what I needed but once I got the hang of it the Hibernate Annotations made adding a new relationship magically easy.

Through the course of writing unit tests for my new database interaction I also discovered HSQLDB which can be used to create a database entirely in memory so your unit tests don’t have to rely on a database being up and running on your development machine. I thought that was kind of nice. Given that I found a three year old article at The Server Side this has been around for a while, none-the-less it was a pleasant discover for me.