PVOutput integration v1

As I mentioned back in November, I’ve been working on integrating with the PVOutput website so I don’t have to manually read the Sunnybeam output files and add stats to the website.

In order to be able to look at creating a solution, I need a set of requirements. These requirements are ones I have thought up:

  • Read new files off of the sunny beam into specified directory

    I store the sunny beams output files on my computer as the device only keeps a rolling 90 days of output stats. If I ever wanted to look back at older stats, I would need them on my pc.

  • Separate the daily files from the monthly files

    The sunny beam creates a daily file and at the end of the month, a month file. Monthly files are stored in a different directory to the daily files on my pc.

  • Upload new file data to pvoutput

    I need to be able to upload the new files which have been transferred to my pc to pvoutput.

  • Upload weather data if available from csv file in root of file dir

    For each day, an average picture of the weather can be set on the record created on pvoutput. As such I shall be creating a simple csv on my pc in the root directory where the outputs are stored which will contain a date and a weather value for that day.

There are also requirements from PVOutput:

  • Make use of the API provided
  • Call services providing system Id and API key in the header
  • Limit the number of calls per hour and minute according to the guidelines

From those four requirements I can break the work needed down into the following:
Create a csv sunny beam reader which will gather the statistics from the file.
Create a csv reader for the weather file
Read the latest output files from the sunny beam and copy them to the directory chosen on pc, separating monthly files from daily files.
Ask pvoutput for the last recording.
The getMissing service on PVOutput takes two parameters df and dt which should be dynamic. So as I’m using Mule – the url contains the expressions to pick out the properties from the MuleMessage header. The response from this service is a list of dates. The dates need to be converted to a Calendar object once received.
Upload to pvoutput from the last recording.

The solution I have so far created is only partially written. I still need to add the weather section and decide how the application will be started / run. If the application is run as a service there are benefits, particularly in not breaking the guideline limits on the service calls to pvoutput. Also as new csv files are created on the device they could be uploaded almost straight away.

There is also no error checking on the service calls to pvoutput. This is relatively easy to add in and I’ll get round to it in version 2.

The testing also needs to be beefed up. There are tests in the project, but more are needed. Especially around reading files off the sunny beam and copying them. Also around error handling on service calls.

The next section is about the most interesting aspects of the code so far

I took the rather lazy approach to using Mule :-). At the same time, I decided to try out something I haven’t had a chance to do before in main code (only test code previously) – calling services using the MuleClient.
Creating the MuleContext and MuleClient is potentially expensive, so there is a factory storing the MuleContext and MuleClient for further use.

MuleFactoryImpl

package uk.co.vsf.solar.mule.impl;

import org.mule.DefaultMuleMessage;
import org.mule.api.MuleContext;
import org.mule.api.MuleException;
import org.mule.api.MuleMessage;
import org.mule.api.config.ConfigurationException;
import org.mule.api.lifecycle.InitialisationException;
import org.mule.context.DefaultMuleContextFactory;
import org.mule.module.client.MuleClient;

import uk.co.vsf.solar.mule.GeneralMuleException;
import uk.co.vsf.solar.mule.MuleFactory;

public class MuleFactoryImpl implements MuleFactory {

    private static final String MULE_CONFIGS = "mule/pvoutput.xml,mule/pvoutput-getmissing.xml,mule/pvoutput-addoutput.xml,spring/applicationContext.xml";

    private MuleClient muleClient;
    private MuleContext muleContext;

    /**
     * {@inheritDoc}
     */
    public MuleClient getClient() {
        if (muleClient == null) {
            try {
                DefaultMuleContextFactory mcfactory = new DefaultMuleContextFactory();
                muleContext = mcfactory.createMuleContext(MULE_CONFIGS);

                muleContext.start();
                muleClient = new MuleClient(muleContext);
            } catch (InitialisationException e) {
                throw new GeneralMuleException();
            } catch (ConfigurationException e) {
                throw new GeneralMuleException();
            } catch (MuleException e) {
                throw new GeneralMuleException();
            }
        }

        return muleClient;
    }

    /**
     * {@inheritDoc}
     */
    public MuleMessage createMuleMessage(Object payload) {
        return new DefaultMuleMessage(payload, muleContext);
    }
}

The add output Mule config file calls the addoutput service on pvoutput indirectly through TcpTrace (What is TcpTrace). The service is post and the mime type has been set to represent form data. Without that you may find unusual data sent…
In the post request, I’ve used message property transformers to add in the pvoutput apikey and systemId. These will appear in the header when sent.
pvoutput-addoutput.xml

<mule xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xmlns:spring="http://www.springframework.org/schema/beans" xmlns:http="http://www.mulesoft.org/schema/mule/http"
	xmlns:vm="http://www.mulesoft.org/schema/mule/vm"
	xsi:schemaLocation="
       http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
       http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/3.2/mule.xsd
       http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/3.2/mule-http.xsd
       http://www.mulesoft.org/schema/mule/vm http://www.mulesoft.org/schema/mule/vm/3.2/mule-vm.xsd">
       
	<flow name="addOutputFlow">
		<vm:inbound-endpoint path="addoutput" 
							 exchange-pattern="request-response"
							 connector-ref="pvoutputConnector" />
		<http:outbound-endpoint address="http://localhost:40000/service/r2/addoutput.jsp"
								method="POST" 
								connector-ref="HttpConnector" 
								exchange-pattern="request-response"
								followRedirects="false"
								mimeType="application/x-www-form-urlencoded">
			<message-properties-transformer>
				<add-message-property key="X-Pvoutput-Apikey" value="${pvoutput.api.key}" />
				<add-message-property key="X-Pvoutput-SystemId" value="${pvoutput.system.id}" />
			</message-properties-transformer>
		</http:outbound-endpoint>
	</flow>
	
</mule>

The same message transformation is applied to the get missing service request. Also for this service, the url has to change to get the dates of outputs not added from the system install date to today. For that I’ve used message property EL selection.
pvoutput-getmissing.xml

<mule xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xmlns:spring="http://www.springframework.org/schema/beans" xmlns:http="http://www.mulesoft.org/schema/mule/http"
	xmlns:vm="http://www.mulesoft.org/schema/mule/vm"
	xsi:schemaLocation="
       http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
       http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/3.2/mule.xsd
       http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/3.2/mule-http.xsd
       http://www.mulesoft.org/schema/mule/vm http://www.mulesoft.org/schema/mule/vm/3.2/mule-vm.xsd">

	<flow name="getMissingFlow">
		<vm:inbound-endpoint ref="GetMissingEndpoint" />
		<http:outbound-endpoint address="http://localhost:40000/service/r1/getmissing.jsp?df=#[header:INBOUND:df]&amp;dt=#[header:INBOUND:dt]"
								method="GET" 
								connector-ref="HttpConnector" 
								exchange-pattern="request-response"
								followRedirects="false">
			<message-properties-transformer>
				<add-message-property key="X-Pvoutput-Apikey" value="${pvoutput.api.key}" />
				<add-message-property key="X-Pvoutput-SystemId" value="${pvoutput.system.id}" />
			</message-properties-transformer>
		</http:outbound-endpoint>
	</flow>
	
</mule>

Complete Code

Can be found here: http://blog.v-s-f.co.uk/2012/01/pvoutput-integration-v1-code/

Solar Panel Output

When I was deciding which solar company to go with, I found out that some of the inverters had inbuilt internet access and others could have data logging added. The system I have (Sunny Boy 3800 and bluetooth piggyback with Sunny Beam) allows me to collect csv files of the day before output. The csv contains limited data, max panel power in a ten minute bracket, total energy generated and all time energy generated, but this is where PVOutput comes into the equation. PVOutput allows one to manually (or automatically if you have the correct setup) upload that data to the website, where it’s stored for free, provides statistics on the data and allows comparisons against other systems.

Here is our output graph:

I’m currently working on an app which when the Sunny Beam is plugged in, will read the new files and send outputs to the website automatically, which will save time and prevent typing errors.


Solar Panels!

I had my solar panels switched on today! I’ve wanted solar panels or a wind turbine for over 3 years now and finally have my own.

The process started in July this year when I decided to find out the true cost of installation. I contacted a local building firm who came the next month to give a quote. After that a neighbour recommended getting some more quotes and three more later I’d chosen the company I wanted to use.

I chose to go with Ecohouse (although two other quotes were real contenders*) as my neighbour used them and he highly recommended their installation team. I signed up and sent my deposit mid September and installation has taken place this week. There have been a number of small issues during the installation (E.g. asbestos), so instead of taking a day the installation was spaced over 5 days, but the quality of installation is fantastic, the installers very friendly and I’d highly recommend them myself.

Details about the system:
250W Suntech solar panels – Total 4kwh peak
SMA Sunnyboy 3800 inverter with Sunny beam wireless monitor

* Eco-energyUK and BSG Property Services