Load balancing the Tesla Powerwall Network Connections
Two weeks ago, we had a short powercut which knocked out my entire network (again!) and this time, something odd happened… My Home Monitoring application was unable to connect to the Tesla Powerwall despite a reboot of the router, home monitoring app and various other devices.
Upon investigating the issue, I found that it was calling the standard endpoint https://powerwall.mydomain.co.uk but wasn’t getting a response (Using my custom domain, I’ve setup a sub-domain powerwall.mydomain.co.uk in nginx and https on top with Let’s Encrypt). The nginx logs showed requests being received but them timing out on the downstream call to the powerwall ethernet connection. A bit more digging and the powerwall had connected via wifi but not ethernet after the powercut*.
I’d known for some time that this scenario could happen (ethernet down, wifi up on the Powerwall) but since it hadn’t happened in over two years since we’ve had the powerwall, I’d ignored it! Now I was forced to do something about it 🙁
My first attempt was to create a load balancer service in pfsense but that didn’t work, so I tried out HAProxy in a docker container on my server.
I setup two new dns forwarding entries in pfsense and routed them to my server:
- powerwall-ethernet.mydomain.co.uk
- and powerwall-wifi.mydomain.co.uk
Next, on my server, nginx would receive the https traffic for powerwall.mydomain.co.uk and proxy pass on to HAProxy. HAProxy would then have two backend servers powerwall-ethernet.mydomain.co.uk and powerwall-wifi.mydomain.co.uk. When traffic was sent from HAProxy to one of those backend servers, nginx would again receive the traffic and proxy pass on to the powerwall network connection of choice.
Setting up the above took 3 days 😂. I’d rather over-engineered the solution and I’d never used HAProxy before so it took time to get the health checks working correctly (I was using an incorrect ca-cert pem which was the main problem – see ** for more on how I identified that it was wrong).
nginx config
server {
listen 80;
server_name powerwall.mydomain.co.uk;
return 301 https://powerwall.mydomain.co.uk$request_uri;
}
server {
listen 443 ssl;
server_name powerwall.mydomain.co.uk;
ssl_certificate /certbot/mydomain.co.uk/fullchain.pem;
ssl_certificate_key /certbot/mydomain.co.uk/privkey.pem;
location / {
proxy_pass https://<server_ip_here>:12345;
}
}
server {
listen 8443 ssl;
server_name powerwall.mydomain.co.uk;
ssl_certificate /certbot/mydomain.co.uk/fullchain.pem;
ssl_certificate_key /certbot/mydomain.co.uk/privkey.pem;
location / {
proxy_pass https://<tesla_powerwall_ethernet_ip_here>;
}
}
server {
listen 9443 ssl;
server_name powerwall.mydomain.co.uk;
ssl_certificate /certbot/mydomain.co.uk/fullchain.pem;
ssl_certificate_key /certbot/mydomain.co.uk/privkey.pem;
location / {
proxy_pass https://<tesla_wifi_ethernet_ip_here>;
}
}
HAProxy config
global
ca-base /etc/ssl/mydomain.co.uk
crt-base /etc/ssl/mydomain.co.uk
ssl-default-bind-options ssl-min-ver TLSv1.2
frontend localnodes
bind *:443 ssl crt /etc/ssl/mydomain.co.uk/fullchain.pem
mode http
default_backend nodes
backend nodes
mode http
balance roundrobin
option httpchk
timeout check 10
server ethernet powerwall-ethernet.mydomain.co.uk:18443 check check-ssl ssl verify required ca-file /etc/ssl/ca-certificates/ca-certificates.crt
server wifi powerwall-wifi.mydomain.co.uk:19443 check check-ssl ssl verify required ca-file /etc/ssl/ca-certificates/ca-certificates.crt
listen stats
bind *:1936
mode http
stats enable
stats hide-version
stats realm Haproxy\ Statistics
stats uri /
stats auth Username:Password
After a few days of the above setup working, I decided to break it all and make it simpler. This time, rather than having nginx proxy pass on traffic to HAProxy and that in turn proxy nginx and the Tesla Powerwall network connections, I changed it to nginx proxy passing requests to HAProxy which in turn has the two Tesla Powerwall network connections directly specified in the backend config:
nginx config
server {
listen 80;
server_name powerwall.mydomain.co.uk;
return 301 https://powerwall.mydomain.co.uk$request_uri;
}
server {
listen 443 ssl;
server_name powerwall.mydomain.co.uk;
ssl_certificate /certbot/mydomain.co.uk/fullchain.pem;
ssl_certificate_key /certbot/mydomain.co.uk/privkey.pem;
location / {
proxy_pass https://<server_ip_address>:12345;
}
}
HAProxy config
global
ca-base /etc/ssl/mydomain.co.uk
crt-base /etc/ssl/mydomain.co.uk
ssl-default-bind-options ssl-min-ver TLSv1.2
frontend localnodes
bind *:443 ssl crt /etc/ssl/mydomain.co.uk/fullchain.pem
mode http
default_backend nodes
backend nodes
mode http
balance roundrobin
option httpchk
timeout check 10
server ethernet <tesla_powerwall_ethernet_ip>:443 check check-ssl ssl verify none
server wifi <tesla_wifi_ethernet_ip>:443 check check-ssl ssl verify none
listen stats
bind *:1936
mode http
stats enable
stats hide-version
stats realm Haproxy\ Statistics
stats uri /
stats auth Username:Password
Much simpler!
For the astute amongst you, you may notice that I did certificate validation on the first HAProxy config but not on the second. That’s because I was forwarding traffic onto nginx which had a valid Let’s Encrypt certificate so validating was easy. With the second example, because the traffic is proxied directly on to the Tesla Powerwall which has a self-signed certificate, I’ve switched off validation.
Now that the switch above the powerwall is up and running correctly, the Powerwall spends most of it’s time connected to the ethernet with wifi not connected but to the calling systems, there’s no change or system interruption if it decides to swap between network connections 🙂
* It turned out only a day later that I found out why it hadn’t connected on ethernet – the switch nearest the Tesla Powerwall had either got cooked in the 30(C) degree weather or not correctly re-established connection after the powercut. Either way, the switch needed a reboot.
** I found out it was an incorrect ca-cert pem because curl worked fine yet HAProxy was saying both endpoints were down, despite being configured with what I thought was a valid pem. When I gave the contianer the entire Ubuntu ca-certs directory and used the content of that directory in the ca-certs config setting, it magically started working correctly!
Please enable the Disqus feature in order to add comments