blitzortung.org Daily Position Cron

As you probably know from my blog, I have a number of Arduino’s around the house for monitoring household and weather metrics and I’m always looking for ways to add more devices and data sets.

My dad happened to find a map one summer on blitzortung.org to view live lightning strikes and we decided to sign up for a lightning detector in December 2015 (I think!).  When you register you use your email address and can see where you are in the list of people who want a device by going back to en.blitzortung.org/cover_your_area.php and filling in your email + the other boxes.

After one year on the waiting list, our position in the queue had moved a bit, but it became tedious checking the list once a week, so I wrote a script that I’ve now made generic enough for anyone.  Simply add it to crontab and take the pain out of checking every week 🙂

GitHub gist: https://gist.github.com/vls29/aac9d3efaf265734dfc1b64c46482160#file-blitzortung-position-sh

#!/bin/bash

email=$1

epc=$(date +%s)
echo $epc

# Country does not seem to be important
res=$(curl --data "info_time=$epc&info_email=$email&info_country=United+Kingdom&info_text=TSqrb" http://en.blitzortung.org/cover_your_area.php)
#echo "-----------------------------"
#echo "HTML Response"
#echo $res
#echo "-----------------------------"

html=$(echo "$res" | grep $email)
#echo "-----------------------------"
#echo "Position Text"
#echo $html
#echo "-----------------------------"

textpositionstart=411+${#email}+1
echo "textpositionstart: $textpositionstart"

position=${html:$textpositionstart:6}
position=$(echo $position | sed 's@^[^0-9]*\([0-9]\+\).*@\1@')
echo "position: $position"

lastpositionfilename=blitzortung-last-position.txt
lastposition=$(cat $lastpositionfilename)
catresult=$?
echo "catresult $catresult"
if [ "$catresult" -eq "1" ]; then
    echo "didn't find last position file"
    lastposition=100000
else
    echo "found last position file"
fi
echo "lastposition: $lastposition"

if [ "$position" -lt "$lastposition" ]; then
    echo $position | mail -s "blitzortung.org position" $email
    echo $position > $lastpositionfilename
else
    echo "position not less than $lastposition: $position"
fi

exit

The only input to the script is the email address you’ve used on the waiting list (assuming you haven’t hardcoded it in the script like I have).  You don’t need country as that doesn’t appear to be used by the site to verify the email address.

Hot Water Tank Temperature on Arduino LCD

Back in March 2014, I posted about monitoring the hot water tank temperature and when the immersion was switched on by the solar surplus http://blog.v-s-f.co.uk/2014/03/home-monitoring-home-made-results/.

We have now decided to display the last temperature reading on an LCD next to the shower room and boiler control panel. The previous software I’d written to upload data to PVOutput had* to be re-written in order to accommodate this new Arduino which would request the last reading available and display it.

The Arduino code below is fairly simple. Every 1 minute it requests data from my server named Pompeii. If it receives data, it refreshes the screen with the response, changing the last character for an up or down arrow, to represent the temperature in/decreasing or a hyphen for level. It’s possibly not the most efficient code in the world, but it doesn’t need to be!

There is one notable item in the code and that’s the pin configuration of the LCD. It turns out that the pins referenced in the Arduino tutorial book I have are pins used for the Ethernet connection on the Ethernet board, therefore I’ve shifted pins 11 and 12 to 6 and 7 respectively.

So what does it look like?

IMG_5312

Code below

#include <SPI.h>
#include <Ethernet.h>
#include <LiquidCrystal.h>

///////// CHANGEABLE VALUES /////////

char pompeii[] = "192.168.0.16";
int pompeiiPort = 28080;

double minutesBetweenCalls = 1.0;

///////// CHANGEABLE VALUES ABOVE /////////

EthernetClient pompeiiClient;
byte mac[] = {
  0x90, 0xA2, 0xDA, 0x0F, 0xA1, 0xCF};
char pompeiiService[] = "/aggregator/services/hot-water-display";
double millisecondsInAMinute = 60000.0;

//LiquidCrystal lcd(12, 11, 5, 4, 3, 2);
LiquidCrystal lcd(7, 6, 5, 4, 3, 2);

const int LCD_WIDTH=16;

byte upArrow[8] = {
  B00100,
  B01110,
  B11111,
  B10101,
  B00100,
  B00100,
  B00000,
};

byte downArrow[8] = {
  B00000,
  B00100,
  B00100,
  B10101,
  B11111,
  B01110,
  B00100,
};

void setup() {
  lcd.createChar(1, upArrow);
  lcd.createChar(0, downArrow);
  lcd.begin(16, 2);
  lcd.setCursor(0,0);

  Serial.begin(9600);

  delay(1000);

  connectToEthernet();
}

void connectToEthernet() {
  // attempt to connect to Wifi network:
  // start the Ethernet connection:
  if (Ethernet.begin(mac) == 0) {
    Serial.println("Failed to configure Ethernet using DHCP waiting 1 minute");
    delay(millisecondsInAMinute);

    if (Ethernet.begin(mac) == 0)
    {
      Serial.println("Failed to configure Ethernet using DHCP waiting 1 more minute");
      delay(millisecondsInAMinute);

      if (Ethernet.begin(mac) == 0) {
        Serial.println("Failed to configure Ethernet using DHCP stopping - will need reset");
        while(true);
      }
    }

  }
  // give the Ethernet shield a second to initialize:
  delay(1000);
  Serial.println("connecting...");
  //lcd.print("connecting...");

  Serial.print("Connected to the network IP: ");
  Serial.println(Ethernet.localIP());
  //lcd.print("Network up...");
}

void loop() {
  displayData();

  Serial.println("fetched data");
  delay(millisecondsInAMinute * minutesBetweenCalls);
}

void displayData() {
  String data = receiveDataFromPompeii();

  if(data.length() > 0) {
    lcd.clear();
    lcd.home();

    String lastChar = "";
    if(data.endsWith(" U") || data.endsWith(" D") || data.endsWith(" L")){
      lastChar = data.substring(data.length() - 1);
      data = data.substring(0, data.length() - 1);
      Serial.println("last char " + lastChar);
    }
    Serial.println("display data " + data);

    if(data.length() > LCD_WIDTH) {
      lcd.home();
      lcd.print(data.substring(0, LCD_WIDTH));
      lcd.setCursor(0, 1);
      if(data.length() > (LCD_WIDTH * 2)) {
        lcd.print(data.substring(LCD_WIDTH, LCD_WIDTH * 2));
      }
      else {
        lcd.print(data.substring(LCD_WIDTH));
      }
    } 
    else {
      lcd.setCursor(0, 0);
      lcd.print(data);
    }

    if(lastChar == "U") {
      lcd.print((char) 1);
    } 
    else if (lastChar == "D") {
      lcd.print((char) 0);
    }
    else if (lastChar == "L") {
      lcd.print("-");
    }
  }
}

String receiveDataFromPompeii() {
  Serial.println("Request data from Pompeii");

  String response = "";

  if (pompeiiClient.connect(pompeii, pompeiiPort)) {
    Serial.println("connected to pompeii");
    // Make a HTTP request:
    pompeiiClient.print("GET ");
    pompeiiClient.print(pompeiiService);
    pompeiiClient.println(" HTTP/1.1");
    pompeiiClient.print("Host: ");
    pompeiiClient.print(pompeii);
    pompeiiClient.print(":");
    pompeiiClient.println(pompeiiPort);
    pompeiiClient.println("Accept: text/html, text/plain");
    pompeiiClient.println("Pragma: no-cache");
    pompeiiClient.println("Cache-Control: no-cache");
    pompeiiClient.println("Connection: close");
    pompeiiClient.println();

    Serial.println("Called pompeii");
    delay(3000);

    String dataRead = "";
    boolean reachedData = false;
    while (pompeiiClient.connected() || pompeiiClient.available()) {
      char c = pompeiiClient.read();
      //Serial.print(c);

      if( reachedData ) {
        response += c;
        //Serial.println(response);
      }

      dataRead += c;

      if(dataRead.endsWith("\r\n\r\n")) {
        Serial.println("Reached the data");
        reachedData = true;
      }
    }
    Serial.println("Finished reading data");

    pompeiiClient.stop();
    pompeiiClient.flush();
    Serial.println("Closed connection");
  }

  Serial.println("response data to return is " + response);
  return response;
}

* As any programmer knows, when you look at your old code, you think… “what on earth!” But also there was a technical reason for re-writing part of it (although I ended up re-writing it all) and that was to store the readings more locally rather than having to go out onto the internet to get the readings.

Arduino Meter Reader

I’ll post soon about my latest purchase – an Arduino and Wifi Shield. The reason I purchased this little toy was to read the impulses from our meter and upload the results to PVOutput. Yeah, I know… sad or what… but when you’re in software integration, it’s oddly good fun and allowed me to see what these Arduino things are like + it might save me money in the future.

It took a few iterations to get a working program which would send results to PVOutput and wasn’t originally as planned. Due to a lack of a clock on the Arduino, there’s a piece of software on my server which receives a post request from the Arduino, adds the date and time and then sends the result on to PVOutput.

The Server script is relatively straight forward. Receive a post request, get the date and time, create the curl request and then send it.

pvoutput-post.php

<?php

$dte = date('Ymd');
$tme = date('H:i');

//echo "Watts Consumed " . $_POST["w"] . " " . $dte . " " . $tme;

$url = 'http://pvoutput.org/service/r2/addstatus.jsp';

$import = $_POST["w"];
$myvars = 'd=' . $dte . '&t=' . $tme . '&v4=' . $import;

//echo "Post Data: " . $myvars;

$ch = curl_init($url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $myvars);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('X-Pvoutput-Apikey: REPLACE THIS',
                                        'X-Pvoutput-SystemId: REPLACE THIS'));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_exec($ch);

?>

The program is the one running on the Arduino. Every minute (or defined interval), the number of impulses from the meter will be used to calculate the watt hours over the time period. This result is then posted to the server for upload.

#include <SPI.h>
#include <WiFi.h>
#include <WiFiUDP.h>

///////// CHANGEABLE VALUES /////////

double multiplier = 1.25;

char ssid[] = "CHANGE THIS";
char pass[] = "CHANGE THIS";

char pompeii[] = "CHANGE THIS FOR IP ADDRESS OF SERVER";
int pompeiiPort = 80;

float minutesBetweenCalls = 1;

///////// CHANGEABLE VALUES ABOVE /////////

int status = WL_IDLE_STATUS;

WiFiClient pompeiiClient;
char pompeiiService[] = "/pvoutput-post.php";

unsigned long impulseCount = 0;

unsigned long lastTimeUploaded = millis();

boolean interruptAttached = false;

unsigned long millisecondsPerMinute = 60000;
unsigned long minutesInHour = 60;
unsigned long timeBetweenCalls = minutesBetweenCalls * millisecondsPerMinute;

void setup() {
  Serial.begin(9600);
  connectToWiFi();
}

void connectToWiFi()
{
  // attempt to connect to Wifi network:
  while ( status != WL_CONNECTED) {
    Serial.print("Attempting to connect to WPA SSID: ");
    Serial.println(ssid);
    // Connect to WPA/WPA2 network:    
    status = WiFi.begin(ssid, pass);

    delay(5000);
  }

  Serial.print("Connected to the network");
  printWifiData();
}

void loop() {
  addFlashInterrupt();

  //getTimeFromPompeii();
  if (isTimeToUploadData())
  {
    Serial.println("Uploading data");
    sendResultsToPompeii();
  }
}

void addFlashInterrupt()
{
  if (!interruptAttached)
  {
    interruptAttached = true;
    Serial.println("Attaching interrupt on pin 3");
    attachInterrupt(1, flash, FALLING);
  }
}

boolean isTimeToUploadData() {
  unsigned long time = millis();

  if( (time - lastTimeUploaded) >= timeBetweenCalls) {
    Serial.println("Time to upload");
    lastTimeUploaded = time;
    return true;
  }
  return false;
}


void printWifiData() {
  // print your WiFi shield's IP address:
  IPAddress ip = WiFi.localIP();
  Serial.print("IP Address: ");
  Serial.println(ip);
}

/* Handles the interrupt flash logic */
void flash() {
  impulseCount++;
}

void sendResultsToPompeii() {
  Serial.println("sendResultsToPompeii");

  int watts = calculateWattsAndResetFlashes();
  String postData = "w=" + String(watts);
  Serial.println(postData);

  if (pompeiiClient.connect(pompeii, pompeiiPort)) {
    Serial.println("connected to pompeii");
    // Make a HTTP request:
    pompeiiClient.print("POST ");
    pompeiiClient.print(pompeiiService);
    pompeiiClient.println(" HTTP/1.1");
    pompeiiClient.print("Host: ");
    pompeiiClient.print(pompeii);
    pompeiiClient.print(":");
    pompeiiClient.println(pompeiiPort);
    pompeiiClient.println("Accept: text/html");
    pompeiiClient.println("Content-Type: application/x-www-form-urlencoded; charset=UTF-8");
    pompeiiClient.print("Content-Length: ");
    pompeiiClient.println(postData.length());
    pompeiiClient.println("Pragma: no-cache");
    pompeiiClient.println("Cache-Control: no-cache");
    pompeiiClient.println("Connection: close");
    pompeiiClient.println();

    pompeiiClient.println(postData);
    pompeiiClient.println();

    pompeiiClient.stop();
    pompeiiClient.flush();
    Serial.println("Called pompeii");
  }
}

int calculateWattsAndResetFlashes()
{
  if (impulseCount == 0UL) {
    return 0;
  }

  double watts = (impulseCount * multiplier) / (minutesBetweenCalls / minutesInHour);
  Serial.print("Calc'd Watts: ");
  Serial.println(watts);
  impulseCount = 0UL;
  return int(watts);
}

Combined with the inverted data upload from my server, I now have the ability to see the electricity import and generation at home from work 🙂

pvoutput import

And for my colleague, that’s real coding – not a single unit test in sight 😉 (joke)