6th Power Cut In 2 Months Causes Networking Crisis In My House!!

Yesterday morning I woke up and checked my email from my phone (as I normally do)… I’d received no emails overnight which was a bit odd and then noticed that I’d switched off 4G the night before and didn’t have a Wifi connection.

Since I have a Wifi access point in my room, that was a little strange. The phone was stuck in “Obtaining IP Address” mode… Maybe the phone’s having problems with that access point.

I wandered off downstairs to feed the cats and noticed the oven times were blinking 00:00 – great, power cut… 6th in less than 2 months!! The dashboard in the kitchen was showing nothing with all the Splunk panels saying failed to create search… Refreshed the dashboard – still nothing, infact Splunk is down, not even an nginx failure! Uh-Oh…

It turns out that the pfsense box had got stuck during boot and required someone to press f1 to continue booting – not very helpful! Unfortunately I didn’t know this until about 3pm and had hastily ordered a new SSD before popping out to go shopping as I thought perhaps the boxes SSD had failed 🙁

It’s certainly made me wonder if a UPS might be a good idea!

Let’s Encrypt with NGINX and pfSense

I’ve been using self-signed certificates for a few of the applications and services I run at home on the local network for a year now and with Firefox it’s not an issue as you accept the risk the very first time and never get asked again. However in the kitchen, the Splunk dashboard runs using the Chrome browser because it’s a little more stable running 24/7 on the tablet. Chrome treats self-signed certificates in a different manner to Firefox – every week it pops up the same boring warning screen and you have to press Advanced then proceed to carry on to the site…

Chrome self signed certificate warning

So to fix the issue with the annoying warning screen, I need to get hold of a non-self-signed certificate.

To get a non-self signed certificate, I need to use a domain that I own or can prove I have control over so that rules out the local network fake domain set in pfSense which is “.home”. I have a few other domains but don’t want to mix those up with internal addresses, so I purchased a new domain from HOST-IT.co.uk.

Next up, trying out Let’s Encrypt – I’d read about Let’s Encrypt about two years ago and wanted to use it in a project at work for test environments but since we have a paid for wildcard certificate, we used that for all the test environments instead. This little project at home gives me a the perfect opportunity to try out Let’s Encrypt and get a non-self-signed certificate for free 🙂

I started with the docker image certbot/certbot but since I’ve never used certbot before, I decided after a few attempts to install certbot on my server instead and at least that way it prompts you to complete the certificate registration. For those interested, the command I used can be found below.

certbot certonly --config-dir config/ --work-dir work/ --logs-dir logs/ --manual --preferred-challenges dns --email email@address.com --agree-tos -d test.example.co.uk

To request a wildcard certificiate instead of a specific host certificate, use: -d *.example.co.uk

Once you run the above, you’ll get prompted to agree to your IP address being logged – entering No will cancel your certificate request! After you’ve agreed to your IP being logged, it will display a DNS TXT record that you need to place on your domain.

certbot request certificate dns challenge

Take the challenge code and put a DNS TXT file on your domain, e.g.

acme challenge text example

I’d highly recommend testing whether the TXT file can be found at this point before pressing enter on the certbot window. I found that the TTL on my hosting account was set to a day by default and had to change it to 1 minute. To test the DNS TXT can be seen, run something similar to:

dig -t TXT _acme-challenge.example.co.uk
dig text check

Once you’ve got your certificate, there’s a whole bunch more commands that may come in useful like:

certbot --config-dir config/ --work-dir work/ --logs-dir logs/ certificates

For listing all the certificates you own.

certbot list certificates
certbot --config-dir config/ --work-dir work/ --logs-dir logs/ renew

For renewing all the certificates you own.

certbot renew certificates
certbot --config-dir config/ --work-dir work/ --logs-dir logs/ --help

For a list of all the other commands available.

Next up, I’m keeping my home network domain as “.home” because I have too many services to change over now, so I need to override any requests in pfSense that go to specific hosts on the new domain name. To do this, go to the DNS Forwarder -> Host Override section and add a new entry, e.g. if a request to splunk.example.co.uk is received, use IP address 10.0.0.22 locally.

pfSense host override

And finally, the NGINX config needs to be updated to use the new certificates. Temporarily while I’ve been testing out these new certificates, I’ve left in the server_name configuration that I previously used “splunk.home” but in addition added the new domain so either could work.

nginx config

And the result is that the Chrome browser on the tablet no longer objects to the certificate. Cool 🙂

Splunk dashboard with new certificate

p.s. for those trying to use this guide, some of the images show an example whereas others are based off my real splunk setup – sorry! So the first set of certbot images I’ve asked for test.example.co.uk and therefore if you wanted to put a corresponding record into pfSense, you’d need to put host=test, domain=example.co.uk and in the nginx config it would say server_name test.example.co.uk.

Updated Splunk Dashboards and Powerwall 100% charged in February!

Over the last few months since I re-wrote my home monitoring application and started logging data to Splunk (in addition to the existing two locations) there’s been an update to Splunk that seemed to fix an issue I had with updating dashboard panels in near realtime. This is a huge benefit as the lack of realtime querying with the free version that I was using was one of the reasons I’d considered purchasing a Splunk Enterprise license! The update also included the dark theme which I first thought was a bit of a gimmick but has proved really useful as having a dark background for the quick glace dashboard means you can see the colours easier when reflected in the kitchen window.

So since the first draft dashboard I posted back in July, I’ve re-written the dashboards and started using a Splunk App so I can copy the source files out into bitbucket and make sure I have a copy and also logically group my dashboards away from the default search app content. My Splunk App is called Home Monitoring (as you’ll see below in the screenshots) after the server app and the logo is a small snippet from a picture of my solar array.

Below is a screenshot of the quick glance section of a dashboard titled realtime. Although it’s titled realtime (RT), the quick glance dashboard panels aren’t actually using RT searches as I found them to be less correct than using a search combined with “head 1” and limiting the search window to around 15 seconds.

On the quick glance section I have a line for the Powerwall figures, displaying current charge percentage (taking in to account the reserved 5%), load on the battery, the amount of power in kWh in the battery (an alternative way of displaying the percentage) and a rough restimate on how long until full or empty at current load. All of these figures come from the Powerwall APIs.

On the second line is the current household load, load on the grid (both from Powerwall API), import today (from the Arduino in the meter cupboard), generation from the solar array, total enery generated today by the solar array (both of the solar figures are from the inverter data) and mains voltage (from the Arduino in the meter cupboard).

The third line displays the current temperature of the hot water tank and whether it’s going up or down based on the figure before (from the Arduino in the hot water cupboard), the status of the immersion plug (from the server code which controlls the plug) and how much data we’ve used in the month on the 4G connection (from the EE status page).

And finally on the fourth line is the wind figures from the Arduino down the garden.

The colours on the page as mentioned earlier help to be able to see at a quick glance what the state of each section is – even reflected in the kitchen window when standing at the sink!

Below the quick glance section is a graph showing the Powerwall meters API data over 48 hours and charge percentage on the Z axis. The graph showing February 26 and 27th shows that we had incredibly sunny February days and didn’t use up all of the electricity stored in the Powerwalls so we hit fully charged (100%) two days in a row! This is quite a common scenario in the summer but completely unexpected in February.

Below the Powerwall meters graph is a graph displaying daily geneartion and import over the last 30 days. As can be seen in one of the graphs, Sunday 24th to Wednesday 27th February were lovely days where the solar panels achieved near perfect output for four days in a row!

The next screenshot shows the monthly generation and import graph which sits below the daily graph along with the wind speed graph over the last 48 hours. There is one further graph on the realtime dashboard page but it’s just out of shot. It displays the Hot water temperature over the last 48 hours.

All of these screenshots are from the same realtime dashboard but only the quick glance section displays on the screen at one time and you have to scroll down to see the rest of the graphs.