Home Monitoring Re-Write Number Four and the HTTP Status Code 499 Errors

As previously mentioned in the home monitoring re-write number four post, I have a number of Arduinos around the house that collect data and send the data using an HTTP POST to the home monitoring application. During the upgrade of the home monitoring stack, I decided to introduce Nginx to proxy the requests that go to the home monitoring application for visibility and also to do a little bit of url re-writing.

Before Nginx was introduced I had Arduino’s making POST requests to a set of PHP files which received the request, added a timestamp and a header and re-posted the data to the home monitoring application directly. That process was only slightly changed to instead of POST directly to the home monitoring application, POST to Nginx which would proxy the home monitoring endpoint.

I noticed a few weeks after this introduction of the new stack that the grid import figures seemed rather low (factor of 10!) compared to the units that the import meter was reporting and found that the majority of POST requests from the Arduinos were getting stopped mid flight between Nginx and the home monitoring application with a 499 HTTP status code, e.g.

192.168.XXX.XXX - - [27/Aug/2018:07:04:11 +0000] "POST /meter HTTP/1.1" 499 0 "-" "-" "-"
192.168.XXX.XXX - - [27/Aug/2018:07:05:11 +0000] "POST /meter HTTP/1.1" 499 0 "-" "-" "-"
192.168.XXX.XXX - - [27/Aug/2018:07:06:11 +0000] "POST /meter HTTP/1.1" 499 0 "-" "-" "-"
192.168.XXX.XXX - - [27/Aug/2018:07:08:11 +0000] "POST /meter HTTP/1.1" 499 0 "-" "-" "-"
192.168.XXX.XXX - - [27/Aug/2018:07:10:11 +0000] "POST /meter HTTP/1.1" 499 0 "-" "-" "-"

After a small bit of googling and checking the code used on the Arduinos it was obvious that the previous method of sending data would need to change a little to work with Nginx acting as a proxy. As mentioned in this stackoverflow answer “HTTP 499 in Nginx means that the client closed the connection before the server answered the request” the Arduino isn’t waiting for the response from the server and is closing the connection too early – therefore the data didn’t reach the home monitoring application. In the previous setup the Arduino could fire the data and forget (close the connection) immediately after sending the request as it wasn’t interested in the response.

To fix this, all I needed to do was introduce a small wait after sending the data to reduce the number of 499’s. The code now reads, establish a connection, send data, wait 15ms, close connection – or for those that want to see the full code:

  String postData = getPostData();

  if (pompeiiClient.connect(pompeii, pompeiiPort)) {
    Serial.println("connected to pompeii");
    // Make a HTTP request:
    pompeiiClient.println("POST " + String(pompeiiService) + " HTTP/1.1");
    pompeiiClient.println("Host: " + String(pompeii) + ":" + pompeiiPort);
    pompeiiClient.println("Content-Type: application/json");
    pompeiiClient.println("Content-Length: " + String(postData.length()));
    pompeiiClient.println("Pragma: no-cache");
    pompeiiClient.println("Cache-Control: no-cache");
    pompeiiClient.println("Connection: close");
    pompeiiClient.println();

    pompeiiClient.println(postData);
    pompeiiClient.println();

    delay(httpRequestDelay);
    pompeiiClient.stop();
    pompeiiClient.flush();
    Serial.println("Called pompeii");
  }

Ubiquiti mFI mPower Project – Part 1

This year I upgraded the home monitoring suite in my house from a Wattson to a Splunk dashboard running on a tablet in the kitchen. The primary reason for doing this was that after getting the Tesla Powerwall, Wattson no longer was able to correctly state how much power was being imported/exported from the home until the battery was completely charged. The downside of getting rid of Wattson was that we relied on the Optiplugs to run the immersion and a radiator when there was excess power being sent to the grid. So I needed another solution to running the immersion on the rare days when the Powerwall will be completely charged and we’re exporting to the grid.

After a fair amount of time searching for a wireless plug with an API, I found the following plug https://wifiplug.co.uk/ that ticked all the boxes, but being the little bit paranoid I am, I didn’t really want a plug that could theorietically be controlled by anyone from the internet. Also, if that company went bust, how would I control the plug without their API gateway?

I then spent another handful of hours searching and came across the following blog https://blog.linitx.com/ubiquiti-mfi-mpower/ which happened to use a wifi plug that doesn’t rely on an internet connection and is from the same company that I use for my WiFi access points! The only problem is that unlike the wifiplug, this one doesn’t have an official API and there isn’t a UK plug version… But what the heck, at least it’d be an interesting project!

So I bought the European single socket version from wifi-stock.co.uk/. The next day I got a call from wifi-stock to check “I hand’t made a mistake and that I did realise it wasn’t a UK power socket” – that was nice and a good idea! 🙂

I also needed a UK to European adaptor and a European to UK adaptor to be able to use it in a UK socket and power a UK device. This makes the plug stick out quite far from the socket, but it is secure and doesn’t drop over time.

A couple of days later my plug arrived and I set about logging into the device as mentioned on the blog using ssh. That was suprisingly easy as it uses the default username and password that’s used on some of the other Ubiquiti devices – username:ubnt, password:ubnt. I spent a couple of hours poking around on the file system, making the lights blink on and off and tested out that it was possible to turn the plug on and off by echoing 1/0 to the relay

echo 1 > /proc/power/relay1
echo 0 > /proc/power/relay1

I also found the code for the web GUI on the plug /usr/www/mfi/power.cgi (the code behind this: http:///mfi/power.cgi). And that got me thinking…

I was originally going to control the plug from my server via the home monitoring application using ssh, but that means the Java app will have to invoke the ssh command and I’d prefer to have an API on the plug that I could call. I could write another cgi file like the power.cgi file, remove the session aspect and control the plug as Ubiquiti have done, but why bother! Ubiquiti has done all the leg work – I can just use their web app! 🙂

So I began playing with the power GUI and found that it was making POST and GET requests from the GUI to http:///sensors/1/ to fetch the current status and turn the plug on and off. Well that’s easy to control! After a bit of experimentation, these are the four curl commands I need.

Login

You will get an AIROS_SESSIONID back which needs to be put in as a cookie header in the following curl requests.

Get plug data

curl -v \
    http://<device ip>/sensors/1/ \
    -H "Cookie: AIROS_SESSIONID=015152b0143a0267f8f0ae30fcb42f6f"
{"sensors":[{"state":{"output":0,"power":0.0,"energy":0.0,"enabled":0,"current":0.0,"voltage":0.0,"powerfactor":0.0,"relay":0,"lock":0}}],"status":"success"}
{"sensors":[{"state":{"output":1,"power":0.0,"energy":0.0,"enabled":0,"current":0.0,"voltage":0.0,"powerfactor":0.0,"relay":1,"lock":0}}],"status":"success"}

Switch the plug off

curl -v \
    -X PUT \
    http://<device ip>/sensors/1/ \
    -H "Cookie: AIROS_SESSIONID=015152b0143a0267f8f0ae30fcb42f6f" \
    -H "Content-Type: application/x-www-form-urlencoded" \
    -d "output=0"
{"status":"success"}

Switch the plug on

curl -v \
    -X PUT \
    http://<device ip>/sensors/1/ \
    -H "Cookie: AIROS_SESSIONID=015152b0143a0267f8f0ae30fcb42f6f" \
    -H "Content-Type: application/x-www-form-urlencoded" \
    -d "output=1"
{"status":"success"}

Output and enabled on

The above is only controlling the output field but on the GUI you will notice there are two fields, output and enabled. Switching enabled on gives stats about the plugs energy usage but it’s not required to control the device.

To switch the plug on with output and enabled on, simply run

curl -v \
    -X PUT \
    http://<device ip>/sensors/1/ \
    -H "Cookie: AIROS_SESSIONID=015152b0143a0267f8f0ae30fcb42f6f" \
    -H "Content-Type: application/x-www-form-urlencoded" \
    -d "output=1&enabled=1"
{"status":"success"}

If you then request the plug data, you would get something similar to the following response:

curl -v \
    http://<device ip>/sensors/1/ \
    -H "Cookie: AIROS_SESSIONID=015152b0143a0267f8f0ae30fcb42f6f"
{"sensors":[{"state":{"output":1,"power":958.341251254,"energy":44.0625,"enabled":1,"current":4.023638963,"voltage":238.905914783,"powerfactor":0.99695206,"relay":1,"lock":0}}],"status":"success"}

 

In the next part I’ll go through how I controled the plug from Java using the APIs above.

Keepass Auto-Type Examples

I’ve been using Keepass to store my passwords for years but it was only this year that I realised (yes, how could I not know?!) that the Auto-Type functionality can be customised to handle obscure login forms. Below are a list of some of the Auto-Types that I’ve created to handle a selection of sites I visit. Hopefully they will come in useful for someone else 🙂

EE

{USERNAME}{TAB}{TAB}{PASSWORD}{ENTER}

Nationwide
This only works from when you’ve entered your memorable word. Unfortunately the fields aren’t visible until that memorable word is entered.

Under the Advanced tab, enter a new String field with the key “PASSNUMBER” and the value of your pass, e.g. “123456”

{TAB}{TAB}{PICKCHARS:PASSNUMBER:ID=1, C=1}{TAB}{PICKCHARS:PASSNUMBER:ID=2, C=1}{TAB}{PICKCHARS:PASSNUMBER:ID=3, C=1}{TAB}{TAB}{TAB}{TAB}{TAB}{SPACE}

NS&I
Under the Advanced tab, enter a new String field with the key “SURNAME” and the value of your pass, e.g. “Bloggs”

{USERNAME}{TAB}{TAB}{TAB}{S:SURNAME}{TAB}{TAB}{PASSWORD}{ENTER}

Natwest (NWOLB)
This only works from the Step 2 page (after you’ve entered your Customer Number and clicked Continue. If you’re not on a shared PC you can optionally decide to get it to remember your Customer Number and therefore the first page you’ll land on is the Step 2 page).

Under the Advanced tab, enter a new String field with the key “PIN” and the value of your pass, e.g. “1234”

{PICKCHARS:PIN:c=3}{PICKCHARS:Password:c=3}{ENTER}

Sainsbury’s Bank
This only works from the Step 2 page (after you’ve entered your Username and clicked Continue. If you’re not on a shared PC you can optionally decide to get it to remember your Username

Under the Advanced tab, enter a new String field with the key “PIN” and the value of your pass, e.g. “123456”

{PASSWORD}{TAB}{TAB}{PICKCHARS:PIN:ID=1, C=1}{TAB}{PICKCHARS:PIN:ID=2, C=1}{TAB}{PICKCHARS:PIN:ID=3, C=1}{TAB}{TAB}{ENTER}

Scottish Widows
This has only been tested on the Personal Pensions page – https://personal.secure.scottishwidows.co.uk/login/

{USERNAME}{TAB}{TAB}{PASSWORD}{ENTER}