Home Monitoring Upgrade (Part 2) – HSQLDB to MySQL

As mentioned in my previous post (see http://blog.v-s-f.co.uk/2017/04/home-monitoring-upgrade/), the first task I have is to migrate from HSQLDB to MySQL.

Because the system logs data every minute while I have power in the house and I want to minimise downtime when I actually have a fully working upgrade, I’ve experimented with a copy of the live HSQL database.

Once I’d copied it from my server to my laptop I then attempted to view the data in notepad++ – yeah, not a particularly smart move! NP++ cannot handle files over about 100M. It also turns out (having more’d the .data file) that the data is not in a readable state. I had a performance issue with a select query a long time back and changed the table to cached.

So, luckily at work a colleague had introduced us all to a great database tool called SQL Workbench. It can work with most databases and unlike SQuirreL, it doesn’t crash when looking at the work DB2 database.

Using SQL Workbench, I’ve then created a script file which creates the new consolidated table and loads data from the old tables in to the new + drops the old tables. The end result is a 122,398KB HSQLDB script file which is human readable.

Next step was to get MySQL running on my server. Instead of installing it directly though, there’s a Docker image available.

My first few attempts at inserting the data from the HSQLDB file in to the MySQL database were less than impressive! One of the attempts had the server running flat out (100% cpu) for over an hour when I finally decided that it was probably not going to complete the import this year and nuked it!

So having learnt a few lessons about not using single row inserts(!), but batching them in to multi-row inserts of 100,000 and a few MySQL deafult parameter increases (although I’m not sure if these are necessary as the batch inserts seemed to make most difference), I was finally able to import the data. It still took a few minutes from running the MySQL container to it being available – but that’s significantly better than running for hours importing the data!!

Now that I have the necessary scripts and knowledge to migrate the data, the next part is re-writing the application that receives the Arduino data, uploads to PVOutput and serves the hot water display Arduino.

Home Monitoring Upgrade

I’ve been monitoring stats from my meter, weather and hotwater tank for over two years now (see http://blog.v-s-f.co.uk/2015/04/home-monitoring-home-made-reborn/) and the application now needs an upgrade.

I now want to log more data from the weather station (temperature and humidity). This should be as simple as adding two new columns to the HSQLDB, changing the application to write in to the two new fields and adding two new fields to the service definition, but it’s not quite that straight forward…

The old app uses an out dated version of Mule on Tomcat in Docker and it’s far too heavy weight for what it needs to be. Therefore it’s time to give it a revamp.

It’s also occured to me recently that instead of storing the data in five separate tables (one for generation, upload info, hotwater, meter and weather data), why not store it in one table. This saves a significant amount of space as there are four less records per minute and it makes adding new columns for additional data sources relatively quick. The HSQLDB that I’ve been using for a while now is over 400M!

So the first task, which is possibly the biggest, is to migrate the data from the five tables in HSQLDB to a single table and then stop using HSQLDB and migrate to MySQL. Why MySQL – it’s actually quite a performant database, it’s free and easy to get running.

Butterfly Friendly Garden

It’s just over a year ago now that I started transforming the front garden into a butterfly and bee friendly garden (http://blog.v-s-f.co.uk/2015/08/front-garden-transformation/). In the last post I finished the article with some photo’s of the garden from April showing the newly planted borders.

The garden was a huge success and I’ve been very p. The plants grew up more than I’d expected, filling in the gaps nicely (in some cases too much!). My neighbours all commented on the Red Dahlia’s which I thought wouldn’t grow after being frosted a number of times!

Here’s a few pictures of the garden at various stages.

front-garden-grown-up-1

IMG_7066

IMG_7063

IMG_7065

But the main reason as stated above for transforming the garden was to encourage more butterflies and bees and the garden certainly delivered! We had a wide variety of butterflies in the garden at times with over 20 butterflies at a time. Here are pictures of the different butterflies and caterpillars we saw during the year.

Mullein Moth Caterpillars
mullein-moth-catterpillar

Yellow Tail Moth Caterpillar
yellow-tail-moth-catterpillar

Brimstone
brimstone

Ringlet
ringlet

Red Admiral
red-admiral

Painted Lady
painted-lady

Peacock
peacock

Speckled Wood
speckled wood

Small White(?)
small-white

Small Tortoiseshell
small-tortoiseshell

There were a couple more that I couldn’t get my camera out quick enough to take a photo unfortunately, but one was bright blue and small about the size of a 10 pence piece. The other was brown with orange spots and might well have been a female Adonis Blue. Next year I’ll have the camera ready!!