Home Monitoring (home made) – Aggregation

Aggregation of the data before uploading to PVOutput

Why do we need to aggregate the data?

The data collected by the meter reading Arduino will be sent to PVOutput on the addstatus service using the standard parameters. The hot water Arduino will be sending the data on the extended parameters (only available to donors), but you can only use the extended parameters if a v1, 2, 3 or 4 is also transmitted in the same request. Both of the Arduino’s transmit their readings at different times so in order to be able to use the extended parameters on PVOutput, the data needs to be aggregated.

The scripts and programs for aggregation

Both Arduino’s transmit their data initially to my server (called Pompeii in case you hadn’t noticed from the code). Pompeii then aggregates the data before calling PVOutput with the meter readings and additional data.

The first scripts in the chain are php scripts, simply because they were there first – however the new aggregation service can easily replace the php scripts. Their job is to add the date and time to the data received and re-post to Pompeii.

pvoutput-post.php

<?php
$dte = date('Ymd');
$tme = date('H:i');

$url = 'http://localhost:22010/pvoutput-post-consumption';

$import = $_POST["iw"];
$export = $_POST["ew"];
$msBetweenCalls = $_POST["msBetweenCalls"];

$myvars = 'd=' . $dte . '&t=' . $tme . '&v4=' . $import . '&v9=' . $export . '&msBetweenCalls=' . $msBetweenCalls;

//echo "Post Data: " . $myvars;

$ch = curl_init( $url );
curl_setopt( $ch, CURLOPT_POST, 1);
curl_setopt( $ch, CURLOPT_POSTFIELDS, $myvars);
curl_setopt( $ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, 1);
curl_exec( $ch );
?>

pvoutput-post-temp.php

<?php
$dte = date('Ymd');
$tme = date('H:i');

$url = 'http://localhost:21010/pvoutput-post-hotwater';

$temperature = $_POST["t"];
$immersion = $_POST["i"];
$myvars = 'd=' . $dte . '&t=' . $tme . '&v7=' . $temperature . '&v8=' . $immersion;

//echo "Post Data: " . $myvars;

$ch = curl_init( $url );
curl_setopt( $ch, CURLOPT_POST, 1);
curl_setopt( $ch, CURLOPT_POSTFIELDS, $myvars);
curl_setopt( $ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, 1);
curl_exec( $ch );
?>

The next program is a Mule application which is listening on two ports. Each time it receives a message it converts it to an object and puts it on a vm queue with a correlation id of the date + time. If two messages are received with the same correlation id, the next phase is to combine the data and post the data to PVOutput and also write the data to file. The data is written to file in case I decide to write my own graphing app on top of the recorded data.

pvoutputaggregator

There’s also an exception strategy in case second message is never received. If the second message is not received and the first message is a Meter reading, we can still send the result to PVOutput

There are only 6 class files for the entire app – I’ll post them on the next part in order to not clutter this page

Next Part

Part 6: Java Code

Home Monitoring (home made) – Wattson

Reading when the hot water is heated by solar from the Wattson plug

Since the Wattson plug and the hot water tank are in close proximity, we can reuse the Arduino that reads the temperate to determine when the Wattson plug is on and heating the water.

Another light voltage converter is used for this as the Wattson plug has a flash sequence that can be used to determine whether it’s off or on.

When it’s on, the plug flashes:

on-state

But when it’s off, the plug flashes:

off-state

So we can write a bit of code that counts the number of times it’s in the “on” state vs the “off” state and determine from the count whether it was on or off for the time period. [Note there are many other ways that could be written…]

The resulting code for the hot water tank temperature and the Wattson plug state looks like:

#include <SPI.h>
#include <Ethernet.h>

///////// CHANGEABLE VALUES /////////

char pompeii[] = "192.168.0.16";
int pompeiiPort = 80;

double minutesBetweenCalls = 1.0;

const double temperatureOffset = 0.0;

const double temperatureMultiplier = 1.252;
const double temperatureCalculationOffset = 1.188;


///////// CHANGEABLE VALUES ABOVE /////////

EthernetClient pompeiiClient;
byte mac[] = {
  0x90, 0xA0, 0xDA, 0x0E, 0x9B, 0xE6};
char pompeiiService[] = "/pvoutput-post-temp.php";

long counter = 1L;
long cReadings = 0L;

const int temperatureSensorPin = A0;
const double analogueRange = 1024.0;
const double voltage = 5.0;
const double offset = 0.5;
const double milliVolts = 100.0;

// timing stuff
unsigned long lastTimeUploaded = millis();

unsigned long millisecondsPerMinute = 60000;
unsigned long minutesInHour = 60;
unsigned long timeBetweenCalls = minutesBetweenCalls * millisecondsPerMinute;


// taos sensor
const int taosSensorPin = A5;
unsigned long onCount = 0L;
unsigned long offCount = 0L;
unsigned int threshold = 500L;

void setup() {
  Serial.begin(9600);
  connectToEthernet();
}

void connectToEthernet()
{
  // attempt to connect to Wifi network:
  // start the Ethernet connection:
  if (Ethernet.begin(mac) == 0) {
    Serial.println("Failed to configure Ethernet using DHCP waiting 1 minute");
    delay(millisecondsPerMinute);

    if (Ethernet.begin(mac) == 0)
    {
      Serial.println("Failed to configure Ethernet using DHCP waiting 1 more minute");
      delay(millisecondsPerMinute);

      if (Ethernet.begin(mac) == 0) {
        Serial.println("Failed to configure Ethernet using DHCP stopping - will need reset");
        while(true);
      }
    }

  }
  // give the Ethernet shield a second to initialize:
  delay(1000);
  Serial.println("connecting...");

  Serial.print("Connected to the network IP: ");
  Serial.println(Ethernet.localIP());
}

void loop() {
  readTemperatureSensorValue();
  readImmersionPlugSensorValue();

  //getTimeFromPompeii();
  if (isTimeToUploadData())
  {
    Serial.println("Uploading data");
    sendResultsToPompeii();
    resetReadingsAfterUpload();
  }

  delay(1);
}

void readImmersionPlugSensorValue()
{
  int sensorVal = analogRead(taosSensorPin);
  //Serial.println(sensorVal);

  calculateOnOffStatus(sensorVal);
}

void resetImmersionPlugSensorCounts()
{
  onCount = 0L;
  offCount = 0L;
}

void calculateOnOffStatus(int sensorVal)
{
  if( sensorVal < threshold)
  {
    offCount++;
  }
  else
  {
    onCount++;
  }
}

boolean isTimeToUploadData() {
  unsigned long time = millis();

  if( (time - lastTimeUploaded) >= timeBetweenCalls) {
    Serial.println("Time to upload");
    lastTimeUploaded = time;
    return true;
  }
  return false;
}

int getPeriodImmersionOnOffStatus()
{
  Serial.print("On Count: ");
  Serial.print(onCount);
  Serial.print(" Off Count: ");
  Serial.println(offCount);

  if( onCount > offCount)
  {
    return 10;
  }
  else
  {
    return 0;
  }
}

/* Reads the temperature sensor */
void readTemperatureSensorValue() {
  int sensorVal = analogRead(temperatureSensorPin);
  cReadings = cReadings + sensorVal;
  counter = counter + 1L;
}

void resetReadingsAfterUpload()
{
  resetTemperatureSensorCounts();
  resetImmersionPlugSensorCounts();
}

void resetTemperatureSensorCounts()
{
  counter = 1L;
  cReadings = 0L;
}

double calculateAverageTemperatureOverPeriod()
{
  Serial.print("Sensor Value: ");
  Serial.print(averageSensorVal());

  // convert the ADCreading to voltage
  double voltageAv = (averageSensorVal()/analogueRange) * voltage;

  Serial.print(", Av Volts: ");

  Serial.print(voltageAv);
  Serial.print(", degrees C: ");
  double temperatureAv = (voltageAv - offset) * milliVolts;
  temperatureAv = ((temperatureMultiplier * (temperatureAv + temperatureOffset)) + temperatureCalculationOffset);
  Serial.println(temperatureAv);

  return temperatureAv;
}

double averageSensorVal()
{
  return (double)cReadings/(double)counter;
}

void sendResultsToPompeii() {
  Serial.println("sendResultsToPompeii");

  String postData = getPostData();
  Serial.println("post data: " + postData);

  if (pompeiiClient.connect(pompeii, pompeiiPort)) {
    Serial.println("connected to pompeii");
    // Make a HTTP request:
    pompeiiClient.print("POST ");
    pompeiiClient.print(pompeiiService);
    pompeiiClient.println(" HTTP/1.1");
    pompeiiClient.print("Host: ");
    pompeiiClient.print(pompeii);
    pompeiiClient.print(":");
    pompeiiClient.println(pompeiiPort);
    pompeiiClient.println("Accept: text/html");
    pompeiiClient.println("Content-Type: application/x-www-form-urlencoded; charset=UTF-8");
    pompeiiClient.print("Content-Length: ");
    pompeiiClient.println(postData.length());
    pompeiiClient.println("Pragma: no-cache");
    pompeiiClient.println("Cache-Control: no-cache");
    pompeiiClient.println("Connection: close");
    pompeiiClient.println();

    pompeiiClient.println(postData);
    pompeiiClient.println();

    pompeiiClient.stop();
    pompeiiClient.flush();
    Serial.println("Called pompeii");
  }
}

String getPostData()
{
  double averagedTemperature = calculateAverageTemperatureOverPeriod();
  Serial.print("temp to post is: ");
  Serial.println(averagedTemperature);

  char tempChar[10];
  dtostrf(averagedTemperature,3,2,tempChar);
  return "t=" + String(tempChar) + "&i=" + getPeriodImmersionOnOffStatus();
}

Next Part

Part 5: Aggregation of the data before uploading to PVOutput

Home Monitoring (home made) – Temperature

Reading the temperature of the hot water tank

This one is not as easy as the export meter as the meter cupboard and the hot water tank are not in the same area as one another, so we had to purchase another Arduino. This time instead of a wifi shield I got the Ethernet shield – in reflection, not too wise a choice as the cat6 (yes overkill) cable that runs to the cupboard took a collective 5 hours to install.

For the temperature sensor we bought a linear thermistor (for those interested in replicating this, you can use one of these: http://cpc.farnell.com/microchip/mcp9700a-e-to/thermistor-linear-10mv-c-to-92/dp/SC10459)

The temperature sensor is then pushed in through the insulating hot water tank jacket with a dob of conducting gel to help the sensor get as good a reading as possible.

(The program loaded on the Arduino is in the next part)

Next Part

Part 4: Reading when the hot water is heated by solar from the Wattson