Updated Splunk Dashboards and Powerwall 100% charged in February!

Over the last few months since I re-wrote my home monitoring application and started logging data to Splunk (in addition to the existing two locations) there’s been an update to Splunk that seemed to fix an issue I had with updating dashboard panels in near realtime. This is a huge benefit as the lack of realtime querying with the free version that I was using was one of the reasons I’d considered purchasing a Splunk Enterprise license! The update also included the dark theme which I first thought was a bit of a gimmick but has proved really useful as having a dark background for the quick glace dashboard means you can see the colours easier when reflected in the kitchen window.

So since the first draft dashboard I posted back in July, I’ve re-written the dashboards and started using a Splunk App so I can copy the source files out into bitbucket and make sure I have a copy and also logically group my dashboards away from the default search app content. My Splunk App is called Home Monitoring (as you’ll see below in the screenshots) after the server app and the logo is a small snippet from a picture of my solar array.

Below is a screenshot of the quick glance section of a dashboard titled realtime. Although it’s titled realtime (RT), the quick glance dashboard panels aren’t actually using RT searches as I found them to be less correct than using a search combined with “head 1” and limiting the search window to around 15 seconds.

On the quick glance section I have a line for the Powerwall figures, displaying current charge percentage (taking in to account the reserved 5%), load on the battery, the amount of power in kWh in the battery (an alternative way of displaying the percentage) and a rough restimate on how long until full or empty at current load. All of these figures come from the Powerwall APIs.

On the second line is the current household load, load on the grid (both from Powerwall API), import today (from the Arduino in the meter cupboard), generation from the solar array, total enery generated today by the solar array (both of the solar figures are from the inverter data) and mains voltage (from the Arduino in the meter cupboard).

The third line displays the current temperature of the hot water tank and whether it’s going up or down based on the figure before (from the Arduino in the hot water cupboard), the status of the immersion plug (from the server code which controlls the plug) and how much data we’ve used in the month on the 4G connection (from the EE status page).

And finally on the fourth line is the wind figures from the Arduino down the garden.

The colours on the page as mentioned earlier help to be able to see at a quick glance what the state of each section is – even reflected in the kitchen window when standing at the sink!

Below the quick glance section is a graph showing the Powerwall meters API data over 48 hours and charge percentage on the Z axis. The graph showing February 26 and 27th shows that we had incredibly sunny February days and didn’t use up all of the electricity stored in the Powerwalls so we hit fully charged (100%) two days in a row! This is quite a common scenario in the summer but completely unexpected in February.

Below the Powerwall meters graph is a graph displaying daily geneartion and import over the last 30 days. As can be seen in one of the graphs, Sunday 24th to Wednesday 27th February were lovely days where the solar panels achieved near perfect output for four days in a row!

The next screenshot shows the monthly generation and import graph which sits below the daily graph along with the wind speed graph over the last 48 hours. There is one further graph on the realtime dashboard page but it’s just out of shot. It displays the Hot water temperature over the last 48 hours.

All of these screenshots are from the same realtime dashboard but only the quick glance section displays on the screen at one time and you have to scroll down to see the rest of the graphs.

Tesla Powerwall low state of energy / emergency grid charging events

Back in July I posted about changing the application that sits on my server to collect data from around the house and send it to Splunk. That app has been sending data for over 8 months now and during the winter we noticed that the Powerwall was consuming data from the grid nearly every night, yet not charging the battery above the reserved 5% mark.

I contacted Tesla to find out what was happening as these events were quite frequent in the winder months. Their response was that they were “low state of energy” or also known as “emergency grid charging events”. In the below graphs each of the orange spikes below the zero line and with a corresponding spike in the blue line at the same time period represents one of these low state of energy events.

Tesla offered some options on how to reduce the number of these low state of energy events but for the time being we’ve decided to leave the settings as is because the summer months are on the way and the number of low state of energy events has already started to reduce.

Home Monitoring Re-Write Number Four and the HTTP Status Code 499 Errors

As previously mentioned in the home monitoring re-write number four post, I have a number of Arduinos around the house that collect data and send the data using an HTTP POST to the home monitoring application. During the upgrade of the home monitoring stack, I decided to introduce Nginx to proxy the requests that go to the home monitoring application for visibility and also to do a little bit of url re-writing.

Before Nginx was introduced I had Arduino’s making POST requests to a set of PHP files which received the request, added a timestamp and a header and re-posted the data to the home monitoring application directly. That process was only slightly changed to instead of POST directly to the home monitoring application, POST to Nginx which would proxy the home monitoring endpoint.

I noticed a few weeks after this introduction of the new stack that the grid import figures seemed rather low (factor of 10!) compared to the units that the import meter was reporting and found that the majority of POST requests from the Arduinos were getting stopped mid flight between Nginx and the home monitoring application with a 499 HTTP status code, e.g.

192.168.XXX.XXX - - [27/Aug/2018:07:04:11 +0000] "POST /meter HTTP/1.1" 499 0 "-" "-" "-"
192.168.XXX.XXX - - [27/Aug/2018:07:05:11 +0000] "POST /meter HTTP/1.1" 499 0 "-" "-" "-"
192.168.XXX.XXX - - [27/Aug/2018:07:06:11 +0000] "POST /meter HTTP/1.1" 499 0 "-" "-" "-"
192.168.XXX.XXX - - [27/Aug/2018:07:08:11 +0000] "POST /meter HTTP/1.1" 499 0 "-" "-" "-"
192.168.XXX.XXX - - [27/Aug/2018:07:10:11 +0000] "POST /meter HTTP/1.1" 499 0 "-" "-" "-"

After a small bit of googling and checking the code used on the Arduinos it was obvious that the previous method of sending data would need to change a little to work with Nginx acting as a proxy. As mentioned in this stackoverflow answer “HTTP 499 in Nginx means that the client closed the connection before the server answered the request” the Arduino isn’t waiting for the response from the server and is closing the connection too early – therefore the data didn’t reach the home monitoring application. In the previous setup the Arduino could fire the data and forget (close the connection) immediately after sending the request as it wasn’t interested in the response.

To fix this, all I needed to do was introduce a small wait after sending the data to reduce the number of 499’s. The code now reads, establish a connection, send data, wait 15ms, close connection – or for those that want to see the full code:

  String postData = getPostData();

  if (pompeiiClient.connect(pompeii, pompeiiPort)) {
    Serial.println("connected to pompeii");
    // Make a HTTP request:
    pompeiiClient.println("POST " + String(pompeiiService) + " HTTP/1.1");
    pompeiiClient.println("Host: " + String(pompeii) + ":" + pompeiiPort);
    pompeiiClient.println("Content-Type: application/json");
    pompeiiClient.println("Content-Length: " + String(postData.length()));
    pompeiiClient.println("Pragma: no-cache");
    pompeiiClient.println("Cache-Control: no-cache");
    pompeiiClient.println("Connection: close");
    pompeiiClient.println();

    pompeiiClient.println(postData);
    pompeiiClient.println();

    delay(httpRequestDelay);
    pompeiiClient.stop();
    pompeiiClient.flush();
    Serial.println("Called pompeii");
  }