Home Monitoring (home made) – Aggregation

Aggregation of the data before uploading to PVOutput

Why do we need to aggregate the data?

The data collected by the meter reading Arduino will be sent to PVOutput on the addstatus service using the standard parameters. The hot water Arduino will be sending the data on the extended parameters (only available to donors), but you can only use the extended parameters if a v1, 2, 3 or 4 is also transmitted in the same request. Both of the Arduino’s transmit their readings at different times so in order to be able to use the extended parameters on PVOutput, the data needs to be aggregated.

The scripts and programs for aggregation

Both Arduino’s transmit their data initially to my server (called Pompeii in case you hadn’t noticed from the code). Pompeii then aggregates the data before calling PVOutput with the meter readings and additional data.

The first scripts in the chain are php scripts, simply because they were there first – however the new aggregation service can easily replace the php scripts. Their job is to add the date and time to the data received and re-post to Pompeii.

pvoutput-post.php

<?php
$dte = date('Ymd');
$tme = date('H:i');

$url = 'http://localhost:22010/pvoutput-post-consumption';

$import = $_POST["iw"];
$export = $_POST["ew"];
$msBetweenCalls = $_POST["msBetweenCalls"];

$myvars = 'd=' . $dte . '&t=' . $tme . '&v4=' . $import . '&v9=' . $export . '&msBetweenCalls=' . $msBetweenCalls;

//echo "Post Data: " . $myvars;

$ch = curl_init( $url );
curl_setopt( $ch, CURLOPT_POST, 1);
curl_setopt( $ch, CURLOPT_POSTFIELDS, $myvars);
curl_setopt( $ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, 1);
curl_exec( $ch );
?>

pvoutput-post-temp.php

<?php
$dte = date('Ymd');
$tme = date('H:i');

$url = 'http://localhost:21010/pvoutput-post-hotwater';

$temperature = $_POST["t"];
$immersion = $_POST["i"];
$myvars = 'd=' . $dte . '&t=' . $tme . '&v7=' . $temperature . '&v8=' . $immersion;

//echo "Post Data: " . $myvars;

$ch = curl_init( $url );
curl_setopt( $ch, CURLOPT_POST, 1);
curl_setopt( $ch, CURLOPT_POSTFIELDS, $myvars);
curl_setopt( $ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, 1);
curl_exec( $ch );
?>

The next program is a Mule application which is listening on two ports. Each time it receives a message it converts it to an object and puts it on a vm queue with a correlation id of the date + time. If two messages are received with the same correlation id, the next phase is to combine the data and post the data to PVOutput and also write the data to file. The data is written to file in case I decide to write my own graphing app on top of the recorded data.

pvoutputaggregator

There’s also an exception strategy in case second message is never received. If the second message is not received and the first message is a Meter reading, we can still send the result to PVOutput

There are only 6 class files for the entire app – I’ll post them on the next part in order to not clutter this page

Next Part

Part 6: Java Code

Home Monitoring (home made) – Overview

We have been working towards monitoring more and more items within the house to see the benefits of the solar array. Last time I posted an article about the meter reader which uses an Arduino and a light voltage converter to read the impulses from the import meter (a huge thanks goes out to Lars for his post here: http://www.lindnilsson.dk/lars/en/powermeter.php) but we wanted to expand on that and add in:

(Because of the size of the code, this article has been split in to separate pages)

And finally the results of all work!

Arduino Meter Reader

I’ll post soon about my latest purchase – an Arduino and Wifi Shield. The reason I purchased this little toy was to read the impulses from our meter and upload the results to PVOutput. Yeah, I know… sad or what… but when you’re in software integration, it’s oddly good fun and allowed me to see what these Arduino things are like + it might save me money in the future.

It took a few iterations to get a working program which would send results to PVOutput and wasn’t originally as planned. Due to a lack of a clock on the Arduino, there’s a piece of software on my server which receives a post request from the Arduino, adds the date and time and then sends the result on to PVOutput.

The Server script is relatively straight forward. Receive a post request, get the date and time, create the curl request and then send it.

pvoutput-post.php

<?php

$dte = date('Ymd');
$tme = date('H:i');

//echo "Watts Consumed " . $_POST["w"] . " " . $dte . " " . $tme;

$url = 'http://pvoutput.org/service/r2/addstatus.jsp';

$import = $_POST["w"];
$myvars = 'd=' . $dte . '&t=' . $tme . '&v4=' . $import;

//echo "Post Data: " . $myvars;

$ch = curl_init($url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $myvars);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('X-Pvoutput-Apikey: REPLACE THIS',
                                        'X-Pvoutput-SystemId: REPLACE THIS'));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_exec($ch);

?>

The program is the one running on the Arduino. Every minute (or defined interval), the number of impulses from the meter will be used to calculate the watt hours over the time period. This result is then posted to the server for upload.

#include <SPI.h>
#include <WiFi.h>
#include <WiFiUDP.h>

///////// CHANGEABLE VALUES /////////

double multiplier = 1.25;

char ssid[] = "CHANGE THIS";
char pass[] = "CHANGE THIS";

char pompeii[] = "CHANGE THIS FOR IP ADDRESS OF SERVER";
int pompeiiPort = 80;

float minutesBetweenCalls = 1;

///////// CHANGEABLE VALUES ABOVE /////////

int status = WL_IDLE_STATUS;

WiFiClient pompeiiClient;
char pompeiiService[] = "/pvoutput-post.php";

unsigned long impulseCount = 0;

unsigned long lastTimeUploaded = millis();

boolean interruptAttached = false;

unsigned long millisecondsPerMinute = 60000;
unsigned long minutesInHour = 60;
unsigned long timeBetweenCalls = minutesBetweenCalls * millisecondsPerMinute;

void setup() {
  Serial.begin(9600);
  connectToWiFi();
}

void connectToWiFi()
{
  // attempt to connect to Wifi network:
  while ( status != WL_CONNECTED) {
    Serial.print("Attempting to connect to WPA SSID: ");
    Serial.println(ssid);
    // Connect to WPA/WPA2 network:    
    status = WiFi.begin(ssid, pass);

    delay(5000);
  }

  Serial.print("Connected to the network");
  printWifiData();
}

void loop() {
  addFlashInterrupt();

  //getTimeFromPompeii();
  if (isTimeToUploadData())
  {
    Serial.println("Uploading data");
    sendResultsToPompeii();
  }
}

void addFlashInterrupt()
{
  if (!interruptAttached)
  {
    interruptAttached = true;
    Serial.println("Attaching interrupt on pin 3");
    attachInterrupt(1, flash, FALLING);
  }
}

boolean isTimeToUploadData() {
  unsigned long time = millis();

  if( (time - lastTimeUploaded) >= timeBetweenCalls) {
    Serial.println("Time to upload");
    lastTimeUploaded = time;
    return true;
  }
  return false;
}


void printWifiData() {
  // print your WiFi shield's IP address:
  IPAddress ip = WiFi.localIP();
  Serial.print("IP Address: ");
  Serial.println(ip);
}

/* Handles the interrupt flash logic */
void flash() {
  impulseCount++;
}

void sendResultsToPompeii() {
  Serial.println("sendResultsToPompeii");

  int watts = calculateWattsAndResetFlashes();
  String postData = "w=" + String(watts);
  Serial.println(postData);

  if (pompeiiClient.connect(pompeii, pompeiiPort)) {
    Serial.println("connected to pompeii");
    // Make a HTTP request:
    pompeiiClient.print("POST ");
    pompeiiClient.print(pompeiiService);
    pompeiiClient.println(" HTTP/1.1");
    pompeiiClient.print("Host: ");
    pompeiiClient.print(pompeii);
    pompeiiClient.print(":");
    pompeiiClient.println(pompeiiPort);
    pompeiiClient.println("Accept: text/html");
    pompeiiClient.println("Content-Type: application/x-www-form-urlencoded; charset=UTF-8");
    pompeiiClient.print("Content-Length: ");
    pompeiiClient.println(postData.length());
    pompeiiClient.println("Pragma: no-cache");
    pompeiiClient.println("Cache-Control: no-cache");
    pompeiiClient.println("Connection: close");
    pompeiiClient.println();

    pompeiiClient.println(postData);
    pompeiiClient.println();

    pompeiiClient.stop();
    pompeiiClient.flush();
    Serial.println("Called pompeii");
  }
}

int calculateWattsAndResetFlashes()
{
  if (impulseCount == 0UL) {
    return 0;
  }

  double watts = (impulseCount * multiplier) / (minutesBetweenCalls / minutesInHour);
  Serial.print("Calc'd Watts: ");
  Serial.println(watts);
  impulseCount = 0UL;
  return int(watts);
}

Combined with the inverted data upload from my server, I now have the ability to see the electricity import and generation at home from work 🙂

pvoutput import

And for my colleague, that’s real coding – not a single unit test in sight 😉 (joke)