Open APIs

I’ve recently bought a device which lets me see how much electricity is being consumed, produced and the surplus electricity at any point in time. It’s called a Wattson and sits in the kitchen and lets us know when we can switch on the dishwasher for example and maximise the surplus electricity available (when it goes green). It comes with just the Wattson display and the unit to read the electricity consumption + production, but we’ve also invested in two home plugs to switch on appliances in the house when there’s surplus electricity which would have ordinarily gone to the grid.

You’re probably wondering what this has to do with APIs? Well, the Wattson display can be plugged in to a PC and the data can be read in to the proprietary piece of software to display usage over time. The software is ok, but it doesn’t have a lot of features I’d like and the display has to be plugged in to the PC to get to the data. I want to be able to plug an usb dongle in to my server, send requests to the main unit and receive back the import, export and consumption values. Then I want to combine those figures with my PV data and upload both figures to PVOutput.

I wrote to the people who make the device and asked them to share the API behind the device so as to be able to communicate with the device, but they declined to give the API away. FYI, all you need is the display plugged in to a PC running the software, Snoopy and a bit of time & patients. You’ll see all of the commands used to access data, but this still requires the display to be plugged in and my server is not located near where the display is usually kept.

In the end I bought an Arduino and got the import figure from the electricity meter directly, but it still doesn’t tell me how much consumption there is in the house.

Yesterday I was researching a weather station with a similar aim in mind. Upload all the data to the internet and combine the data from the weather station with my PV readings on PVOutput. Weather stations have been around for years and I assumed there would be plenty of options for well built (not budget) units that would upload to the internet without a PC being on (I don’t want to load on proprietary software and have a PC running 24/7 connected to the device to export data from their app to where ever I need the data. I want near real-time data that my server (not running Windows) can request).

Surprisingly not…

One unit I’ve found does upload to the internet directly, but to a website set up by the makers of the device. They do then offer the ability to push that data in to Weather Underground, but personally I’d prefer a device that uploads to WUnderground directly or one that I can make calls to using an API and retrieve just the data I’m after and then include in any app I choose to build. One particular make of weather station I was particularly interested in was La Crosse, but they also seem to have an antiquated view on publishing their API.

In this day and age it seems odd to me that manufacturers of products do not like the idea of publishing their APIs for their little black boxes. Despite what the manufacturers think about their own software that can be used to manage their device data, there are other people out there with better, more innovative ideas on how to integrate their devices with applications and websites. Maybe it’s just me in that my job is to integrate with third party services to help my company achieve business objectives, but I now see unrealised potential in the little devices lying around in my home. I think in the near future it will be the manufacturers that offer APIs for their devices that will find their products sell more as people find innovative ways of using the data created by those devices.

Arduino Meter Reader

I’ll post soon about my latest purchase – an Arduino and Wifi Shield. The reason I purchased this little toy was to read the impulses from our meter and upload the results to PVOutput. Yeah, I know… sad or what… but when you’re in software integration, it’s oddly good fun and allowed me to see what these Arduino things are like + it might save me money in the future.

It took a few iterations to get a working program which would send results to PVOutput and wasn’t originally as planned. Due to a lack of a clock on the Arduino, there’s a piece of software on my server which receives a post request from the Arduino, adds the date and time and then sends the result on to PVOutput.

The Server script is relatively straight forward. Receive a post request, get the date and time, create the curl request and then send it.

pvoutput-post.php

<?php

$dte = date('Ymd');
$tme = date('H:i');

//echo "Watts Consumed " . $_POST["w"] . " " . $dte . " " . $tme;

$url = 'http://pvoutput.org/service/r2/addstatus.jsp';

$import = $_POST["w"];
$myvars = 'd=' . $dte . '&t=' . $tme . '&v4=' . $import;

//echo "Post Data: " . $myvars;

$ch = curl_init($url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $myvars);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('X-Pvoutput-Apikey: REPLACE THIS',
                                        'X-Pvoutput-SystemId: REPLACE THIS'));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_exec($ch);

?>

The program is the one running on the Arduino. Every minute (or defined interval), the number of impulses from the meter will be used to calculate the watt hours over the time period. This result is then posted to the server for upload.

#include <SPI.h>
#include <WiFi.h>
#include <WiFiUDP.h>

///////// CHANGEABLE VALUES /////////

double multiplier = 1.25;

char ssid[] = "CHANGE THIS";
char pass[] = "CHANGE THIS";

char pompeii[] = "CHANGE THIS FOR IP ADDRESS OF SERVER";
int pompeiiPort = 80;

float minutesBetweenCalls = 1;

///////// CHANGEABLE VALUES ABOVE /////////

int status = WL_IDLE_STATUS;

WiFiClient pompeiiClient;
char pompeiiService[] = "/pvoutput-post.php";

unsigned long impulseCount = 0;

unsigned long lastTimeUploaded = millis();

boolean interruptAttached = false;

unsigned long millisecondsPerMinute = 60000;
unsigned long minutesInHour = 60;
unsigned long timeBetweenCalls = minutesBetweenCalls * millisecondsPerMinute;

void setup() {
  Serial.begin(9600);
  connectToWiFi();
}

void connectToWiFi()
{
  // attempt to connect to Wifi network:
  while ( status != WL_CONNECTED) {
    Serial.print("Attempting to connect to WPA SSID: ");
    Serial.println(ssid);
    // Connect to WPA/WPA2 network:    
    status = WiFi.begin(ssid, pass);

    delay(5000);
  }

  Serial.print("Connected to the network");
  printWifiData();
}

void loop() {
  addFlashInterrupt();

  //getTimeFromPompeii();
  if (isTimeToUploadData())
  {
    Serial.println("Uploading data");
    sendResultsToPompeii();
  }
}

void addFlashInterrupt()
{
  if (!interruptAttached)
  {
    interruptAttached = true;
    Serial.println("Attaching interrupt on pin 3");
    attachInterrupt(1, flash, FALLING);
  }
}

boolean isTimeToUploadData() {
  unsigned long time = millis();

  if( (time - lastTimeUploaded) >= timeBetweenCalls) {
    Serial.println("Time to upload");
    lastTimeUploaded = time;
    return true;
  }
  return false;
}


void printWifiData() {
  // print your WiFi shield's IP address:
  IPAddress ip = WiFi.localIP();
  Serial.print("IP Address: ");
  Serial.println(ip);
}

/* Handles the interrupt flash logic */
void flash() {
  impulseCount++;
}

void sendResultsToPompeii() {
  Serial.println("sendResultsToPompeii");

  int watts = calculateWattsAndResetFlashes();
  String postData = "w=" + String(watts);
  Serial.println(postData);

  if (pompeiiClient.connect(pompeii, pompeiiPort)) {
    Serial.println("connected to pompeii");
    // Make a HTTP request:
    pompeiiClient.print("POST ");
    pompeiiClient.print(pompeiiService);
    pompeiiClient.println(" HTTP/1.1");
    pompeiiClient.print("Host: ");
    pompeiiClient.print(pompeii);
    pompeiiClient.print(":");
    pompeiiClient.println(pompeiiPort);
    pompeiiClient.println("Accept: text/html");
    pompeiiClient.println("Content-Type: application/x-www-form-urlencoded; charset=UTF-8");
    pompeiiClient.print("Content-Length: ");
    pompeiiClient.println(postData.length());
    pompeiiClient.println("Pragma: no-cache");
    pompeiiClient.println("Cache-Control: no-cache");
    pompeiiClient.println("Connection: close");
    pompeiiClient.println();

    pompeiiClient.println(postData);
    pompeiiClient.println();

    pompeiiClient.stop();
    pompeiiClient.flush();
    Serial.println("Called pompeii");
  }
}

int calculateWattsAndResetFlashes()
{
  if (impulseCount == 0UL) {
    return 0;
  }

  double watts = (impulseCount * multiplier) / (minutesBetweenCalls / minutesInHour);
  Serial.print("Calc'd Watts: ");
  Serial.println(watts);
  impulseCount = 0UL;
  return int(watts);
}

Combined with the inverted data upload from my server, I now have the ability to see the electricity import and generation at home from work 🙂

pvoutput import

And for my colleague, that’s real coding – not a single unit test in sight 😉 (joke)

PVOutput integration part 2

After using my own code for over a year to upload CSV results to PVOutput, I wanted a solution that would give me real time stats instead of the previous days output. A bit naively, I thought that it would be as simple as writing a little bit of Java code to listen to a bluetooth connection and simply store the data. Ha! After spending 8 hours downloading all the relevant software across my slow broadband connection, I opened a socket to the inverter and put the code in debug, only to get an exception… 30 minutes later I had a bit of a “D’oh!” moment when I realised it’d taken so long to get everything running, the inverter had switched off for the day.

Next morning, I fired up the software to again get an exception, but not a connection problem this time. My inverter is an SMA Sunny Boy 3800 and after doing a bit of searching, SMA has a proprietary language for communicating with their inverters. I stumbled upon a piece of software called SMA-Bluetooth which did exactly what I wantedand saved me writing my own code. After struggling to install the software, eventually I had it up and running, but the software reported quite a few date errors and as such wasn’t usable for me. A member of the SMA-Bluetooth community suggested I try SMAspot which sorted the date issue and also had a patch from a community member for uploading results to PVOutput.

One week later, my server has been calling the inverter every 1 minute and uploading results to PVOutput allowing me to be really sad and watch the output stats of my panels from work!

Now all I have to do is find a small low energy display to put in the kitchen to view PVOutput.