Tesla Powerwall low state of energy / emergency grid charging events

Back in July I posted about changing the application that sits on my server to collect data from around the house and send it to Splunk. That app has been sending data for over 8 months now and during the winter we noticed that the Powerwall was consuming data from the grid nearly every night, yet not charging the battery above the reserved 5% mark.

I contacted Tesla to find out what was happening as these events were quite frequent in the winder months. Their response was that they were “low state of energy” or also known as “emergency grid charging events”. In the below graphs each of the orange spikes below the zero line and with a corresponding spike in the blue line at the same time period represents one of these low state of energy events.

Tesla offered some options on how to reduce the number of these low state of energy events but for the time being we’ve decided to leave the settings as is because the summer months are on the way and the number of low state of energy events has already started to reduce.

Home Monitoring Re-Write Number Four and the HTTP Status Code 499 Errors

As previously mentioned in the home monitoring re-write number four post, I have a number of Arduinos around the house that collect data and send the data using an HTTP POST to the home monitoring application. During the upgrade of the home monitoring stack, I decided to introduce Nginx to proxy the requests that go to the home monitoring application for visibility and also to do a little bit of url re-writing.

Before Nginx was introduced I had Arduino’s making POST requests to a set of PHP files which received the request, added a timestamp and a header and re-posted the data to the home monitoring application directly. That process was only slightly changed to instead of POST directly to the home monitoring application, POST to Nginx which would proxy the home monitoring endpoint.

I noticed a few weeks after this introduction of the new stack that the grid import figures seemed rather low (factor of 10!) compared to the units that the import meter was reporting and found that the majority of POST requests from the Arduinos were getting stopped mid flight between Nginx and the home monitoring application with a 499 HTTP status code, e.g.

192.168.XXX.XXX - - [27/Aug/2018:07:04:11 +0000] "POST /meter HTTP/1.1" 499 0 "-" "-" "-"
192.168.XXX.XXX - - [27/Aug/2018:07:05:11 +0000] "POST /meter HTTP/1.1" 499 0 "-" "-" "-"
192.168.XXX.XXX - - [27/Aug/2018:07:06:11 +0000] "POST /meter HTTP/1.1" 499 0 "-" "-" "-"
192.168.XXX.XXX - - [27/Aug/2018:07:08:11 +0000] "POST /meter HTTP/1.1" 499 0 "-" "-" "-"
192.168.XXX.XXX - - [27/Aug/2018:07:10:11 +0000] "POST /meter HTTP/1.1" 499 0 "-" "-" "-"

After a small bit of googling and checking the code used on the Arduinos it was obvious that the previous method of sending data would need to change a little to work with Nginx acting as a proxy. As mentioned in this stackoverflow answer “HTTP 499 in Nginx means that the client closed the connection before the server answered the request” the Arduino isn’t waiting for the response from the server and is closing the connection too early – therefore the data didn’t reach the home monitoring application. In the previous setup the Arduino could fire the data and forget (close the connection) immediately after sending the request as it wasn’t interested in the response.

To fix this, all I needed to do was introduce a small wait after sending the data to reduce the number of 499’s. The code now reads, establish a connection, send data, wait 15ms, close connection – or for those that want to see the full code:

  String postData = getPostData();

  if (pompeiiClient.connect(pompeii, pompeiiPort)) {
    Serial.println("connected to pompeii");
    // Make a HTTP request:
    pompeiiClient.println("POST " + String(pompeiiService) + " HTTP/1.1");
    pompeiiClient.println("Host: " + String(pompeii) + ":" + pompeiiPort);
    pompeiiClient.println("Content-Type: application/json");
    pompeiiClient.println("Content-Length: " + String(postData.length()));
    pompeiiClient.println("Pragma: no-cache");
    pompeiiClient.println("Cache-Control: no-cache");
    pompeiiClient.println("Connection: close");
    pompeiiClient.println();

    pompeiiClient.println(postData);
    pompeiiClient.println();

    delay(httpRequestDelay);
    pompeiiClient.stop();
    pompeiiClient.flush();
    Serial.println("Called pompeii");
  }

Ubiquiti mFI mPower Project – Part 1

This year I upgraded the home monitoring suite in my house from a Wattson to a Splunk dashboard running on a tablet in the kitchen. The primary reason for doing this was that after getting the Tesla Powerwall, Wattson no longer was able to correctly state how much power was being imported/exported from the home until the battery was completely charged. The downside of getting rid of Wattson was that we relied on the Optiplugs to run the immersion and a radiator when there was excess power being sent to the grid. So I needed another solution to running the immersion on the rare days when the Powerwall will be completely charged and we’re exporting to the grid.

After a fair amount of time searching for a wireless plug with an API, I found the following plug https://wifiplug.co.uk/ that ticked all the boxes, but being the little bit paranoid I am, I didn’t really want a plug that could theorietically be controlled by anyone from the internet. Also, if that company went bust, how would I control the plug without their API gateway?

I then spent another handful of hours searching and came across the following blog https://blog.linitx.com/ubiquiti-mfi-mpower/ which happened to use a wifi plug that doesn’t rely on an internet connection and is from the same company that I use for my WiFi access points! The only problem is that unlike the wifiplug, this one doesn’t have an official API and there isn’t a UK plug version… But what the heck, at least it’d be an interesting project!

So I bought the European single socket version from wifi-stock.co.uk/. The next day I got a call from wifi-stock to check “I hand’t made a mistake and that I did realise it wasn’t a UK power socket” – that was nice and a good idea! 🙂

I also needed a UK to European adaptor and a European to UK adaptor to be able to use it in a UK socket and power a UK device. This makes the plug stick out quite far from the socket, but it is secure and doesn’t drop over time.

A couple of days later my plug arrived and I set about logging into the device as mentioned on the blog using ssh. That was suprisingly easy as it uses the default username and password that’s used on some of the other Ubiquiti devices – username:ubnt, password:ubnt. I spent a couple of hours poking around on the file system, making the lights blink on and off and tested out that it was possible to turn the plug on and off by echoing 1/0 to the relay

echo 1 > /proc/power/relay1
echo 0 > /proc/power/relay1

I also found the code for the web GUI on the plug /usr/www/mfi/power.cgi (the code behind this: http:///mfi/power.cgi). And that got me thinking…

I was originally going to control the plug from my server via the home monitoring application using ssh, but that means the Java app will have to invoke the ssh command and I’d prefer to have an API on the plug that I could call. I could write another cgi file like the power.cgi file, remove the session aspect and control the plug as Ubiquiti have done, but why bother! Ubiquiti has done all the leg work – I can just use their web app! 🙂

So I began playing with the power GUI and found that it was making POST and GET requests from the GUI to http:///sensors/1/ to fetch the current status and turn the plug on and off. Well that’s easy to control! After a bit of experimentation, these are the four curl commands I need.

Login

You will get an AIROS_SESSIONID back which needs to be put in as a cookie header in the following curl requests.

Get plug data

curl -v \
    http://<device ip>/sensors/1/ \
    -H "Cookie: AIROS_SESSIONID=015152b0143a0267f8f0ae30fcb42f6f"
{"sensors":[{"state":{"output":0,"power":0.0,"energy":0.0,"enabled":0,"current":0.0,"voltage":0.0,"powerfactor":0.0,"relay":0,"lock":0}}],"status":"success"}
{"sensors":[{"state":{"output":1,"power":0.0,"energy":0.0,"enabled":0,"current":0.0,"voltage":0.0,"powerfactor":0.0,"relay":1,"lock":0}}],"status":"success"}

Switch the plug off

curl -v \
    -X PUT \
    http://<device ip>/sensors/1/ \
    -H "Cookie: AIROS_SESSIONID=015152b0143a0267f8f0ae30fcb42f6f" \
    -H "Content-Type: application/x-www-form-urlencoded" \
    -d "output=0"
{"status":"success"}

Switch the plug on

curl -v \
    -X PUT \
    http://<device ip>/sensors/1/ \
    -H "Cookie: AIROS_SESSIONID=015152b0143a0267f8f0ae30fcb42f6f" \
    -H "Content-Type: application/x-www-form-urlencoded" \
    -d "output=1"
{"status":"success"}

Output and enabled on

The above is only controlling the output field but on the GUI you will notice there are two fields, output and enabled. Switching enabled on gives stats about the plugs energy usage but it’s not required to control the device.

To switch the plug on with output and enabled on, simply run

curl -v \
    -X PUT \
    http://<device ip>/sensors/1/ \
    -H "Cookie: AIROS_SESSIONID=015152b0143a0267f8f0ae30fcb42f6f" \
    -H "Content-Type: application/x-www-form-urlencoded" \
    -d "output=1&enabled=1"
{"status":"success"}

If you then request the plug data, you would get something similar to the following response:

curl -v \
    http://<device ip>/sensors/1/ \
    -H "Cookie: AIROS_SESSIONID=015152b0143a0267f8f0ae30fcb42f6f"
{"sensors":[{"state":{"output":1,"power":958.341251254,"energy":44.0625,"enabled":1,"current":4.023638963,"voltage":238.905914783,"powerfactor":0.99695206,"relay":1,"lock":0}}],"status":"success"}

 

In the next part I’ll go through how I controled the plug from Java using the APIs above.