Home Monitoring Re-Write Number Four and the HTTP Status Code 499 Errors

As previously mentioned in the home monitoring re-write number four post, I have a number of Arduinos around the house that collect data and send the data using an HTTP POST to the home monitoring application. During the upgrade of the home monitoring stack, I decided to introduce Nginx to proxy the requests that go to the home monitoring application for visibility and also to do a little bit of url re-writing.

Before Nginx was introduced I had Arduino’s making POST requests to a set of PHP files which received the request, added a timestamp and a header and re-posted the data to the home monitoring application directly. That process was only slightly changed to instead of POST directly to the home monitoring application, POST to Nginx which would proxy the home monitoring endpoint.

I noticed a few weeks after this introduction of the new stack that the grid import figures seemed rather low (factor of 10!) compared to the units that the import meter was reporting and found that the majority of POST requests from the Arduinos were getting stopped mid flight between Nginx and the home monitoring application with a 499 HTTP status code, e.g.

192.168.XXX.XXX - - [27/Aug/2018:07:04:11 +0000] "POST /meter HTTP/1.1" 499 0 "-" "-" "-"
192.168.XXX.XXX - - [27/Aug/2018:07:05:11 +0000] "POST /meter HTTP/1.1" 499 0 "-" "-" "-"
192.168.XXX.XXX - - [27/Aug/2018:07:06:11 +0000] "POST /meter HTTP/1.1" 499 0 "-" "-" "-"
192.168.XXX.XXX - - [27/Aug/2018:07:08:11 +0000] "POST /meter HTTP/1.1" 499 0 "-" "-" "-"
192.168.XXX.XXX - - [27/Aug/2018:07:10:11 +0000] "POST /meter HTTP/1.1" 499 0 "-" "-" "-"

After a small bit of googling and checking the code used on the Arduinos it was obvious that the previous method of sending data would need to change a little to work with Nginx acting as a proxy. As mentioned in this stackoverflow answer “HTTP 499 in Nginx means that the client closed the connection before the server answered the request” the Arduino isn’t waiting for the response from the server and is closing the connection too early – therefore the data didn’t reach the home monitoring application. In the previous setup the Arduino could fire the data and forget (close the connection) immediately after sending the request as it wasn’t interested in the response.

To fix this, all I needed to do was introduce a small wait after sending the data to reduce the number of 499’s. The code now reads, establish a connection, send data, wait 15ms, close connection – or for those that want to see the full code:

  String postData = getPostData();

  if (pompeiiClient.connect(pompeii, pompeiiPort)) {
    Serial.println("connected to pompeii");
    // Make a HTTP request:
    pompeiiClient.println("POST " + String(pompeiiService) + " HTTP/1.1");
    pompeiiClient.println("Host: " + String(pompeii) + ":" + pompeiiPort);
    pompeiiClient.println("Content-Type: application/json");
    pompeiiClient.println("Content-Length: " + String(postData.length()));
    pompeiiClient.println("Pragma: no-cache");
    pompeiiClient.println("Cache-Control: no-cache");
    pompeiiClient.println("Connection: close");
    pompeiiClient.println();

    pompeiiClient.println(postData);
    pompeiiClient.println();

    delay(httpRequestDelay);
    pompeiiClient.stop();
    pompeiiClient.flush();
    Serial.println("Called pompeii");
  }