Home Monitoring (home made) – Aggregation

Aggregation of the data before uploading to PVOutput

Why do we need to aggregate the data?

The data collected by the meter reading Arduino will be sent to PVOutput on the addstatus service using the standard parameters. The hot water Arduino will be sending the data on the extended parameters (only available to donors), but you can only use the extended parameters if a v1, 2, 3 or 4 is also transmitted in the same request. Both of the Arduino’s transmit their readings at different times so in order to be able to use the extended parameters on PVOutput, the data needs to be aggregated.

The scripts and programs for aggregation

Both Arduino’s transmit their data initially to my server (called Pompeii in case you hadn’t noticed from the code). Pompeii then aggregates the data before calling PVOutput with the meter readings and additional data.

The first scripts in the chain are php scripts, simply because they were there first – however the new aggregation service can easily replace the php scripts. Their job is to add the date and time to the data received and re-post to Pompeii.

pvoutput-post.php

<?php
$dte = date('Ymd');
$tme = date('H:i');

$url = 'http://localhost:22010/pvoutput-post-consumption';

$import = $_POST["iw"];
$export = $_POST["ew"];
$msBetweenCalls = $_POST["msBetweenCalls"];

$myvars = 'd=' . $dte . '&t=' . $tme . '&v4=' . $import . '&v9=' . $export . '&msBetweenCalls=' . $msBetweenCalls;

//echo "Post Data: " . $myvars;

$ch = curl_init( $url );
curl_setopt( $ch, CURLOPT_POST, 1);
curl_setopt( $ch, CURLOPT_POSTFIELDS, $myvars);
curl_setopt( $ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, 1);
curl_exec( $ch );
?>

pvoutput-post-temp.php

<?php
$dte = date('Ymd');
$tme = date('H:i');

$url = 'http://localhost:21010/pvoutput-post-hotwater';

$temperature = $_POST["t"];
$immersion = $_POST["i"];
$myvars = 'd=' . $dte . '&t=' . $tme . '&v7=' . $temperature . '&v8=' . $immersion;

//echo "Post Data: " . $myvars;

$ch = curl_init( $url );
curl_setopt( $ch, CURLOPT_POST, 1);
curl_setopt( $ch, CURLOPT_POSTFIELDS, $myvars);
curl_setopt( $ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, 1);
curl_exec( $ch );
?>

The next program is a Mule application which is listening on two ports. Each time it receives a message it converts it to an object and puts it on a vm queue with a correlation id of the date + time. If two messages are received with the same correlation id, the next phase is to combine the data and post the data to PVOutput and also write the data to file. The data is written to file in case I decide to write my own graphing app on top of the recorded data.

pvoutputaggregator

There’s also an exception strategy in case second message is never received. If the second message is not received and the first message is a Meter reading, we can still send the result to PVOutput

There are only 6 class files for the entire app – I’ll post them on the next part in order to not clutter this page

Next Part

Part 6: Java Code

Example of a Spring 3.2 and Hibernate 4 repository base

The is a partial example of a spring 3.2 and hibernate 4 project repository base layer. It doesn’t necessarily compile or even do anything, but it’s an example…

aCtx-repository.xml

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context"
	xmlns:tx="http://www.springframework.org/schema/tx"
	xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
					    http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
					    http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx.xsd">

	<bean id="sessionFactory"
		class="org.springframework.orm.hibernate4.LocalSessionFactoryBean">
		<property name="packagesToScan" value="uk.co.vsf.penny.domain.impl" />
		<property name="dataSource" ref="dataSource" />
		<property name="hibernateProperties">
			<props>
				<prop key="hibernate.dialect">org.hibernate.dialect.HSQLDialect</prop>
				<prop key="hibernate.hbm2ddl.auto">create</prop>
				<prop key="hibernate.show_sql">true</prop>
			</props>
		</property>
	</bean>

	<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource"
		destroy-method="close">
		<property name="driverClassName" value="${jdbc.driverClassName}" />
		<property name="url" value="${jdbc.url}" />
		<property name="username" value="${jdbc.username}" />
		<property name="password" value="${jdbc.password}" />
	</bean>

	<context:property-placeholder location="jdbc.properties" />

	<bean id="transactionManager"
		class="org.springframework.orm.hibernate4.HibernateTransactionManager">
		<property name="sessionFactory" ref="sessionFactory" />
	</bean>

	<tx:annotation-driven transaction-manager="transactionManager"
		proxy-target-class="true" />

</beans>

aCtx-dao.xml

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">

	<bean id="baseDao" class="uk.co.vsf.penny.repository.dao.impl.BaseDao">
		<property name="sessionFactory" ref="sessionFactory" />
	</bean>

	<bean id="accountDao"
		  class="uk.co.vsf.penny.repository.dao.impl.AccountDaoImpl"
		  parent="baseDao">
	</bean>

	<bean id="transactionDao"
		  class="uk.co.vsf.penny.repository.dao.impl.TransactionDaoImpl"
		  parent="baseDao">
	</bean>

</beans>

aCtx-base.xml

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context"
	xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
					    http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd">

	<context:component-scan base-package="uk.co.vsf.penny" />
</beans>

BaseDao.java

public class BaseDao {

    private SessionFactory sessionFactory;

    protected SessionFactory getSessionFactory() {
        return sessionFactory;
    }

    public void setSessionFactory(SessionFactory sessionFactory) {
        this.sessionFactory = sessionFactory;
    }

    public Long totalTableCount(Class<?> clazz) {
        Criteria criteria = getSessionFactory().getCurrentSession().createCriteria(clazz);
        criteria.setProjection(Projections.rowCount());
        return (Long) criteria.uniqueResult();
    }
}

AccountDaoImpl.java

public class AccountDaoImpl extends BaseDao implements AccountDao {

    public void addAccount(Account account) {
        getSessionFactory().getCurrentSession().save(account);
    }

    public void updateAccount(Account account) {
        getSessionFactory().getCurrentSession().update(account);
    }

    public void deleteAccount(Account account) {
        getSessionFactory().getCurrentSession().delete(account);
    }

    @Override
    public Long countAccountsForUser(User user) {
        Criteria criteria = getSessionFactory().getCurrentSession().createCriteria(Account.class);
        criteria.add(Restrictions.eq("user", user));
        criteria.setProjection(Projections.rowCount());
        return (Long) criteria.uniqueResult();
    }
}

Account.java

@Entity
@Table
public class Account {

    @Id
    @GeneratedValue(strategy = GenerationType.AUTO)
    private long id;

    @Column(nullable = false)
    private String name;
    @Column(nullable = true)
    private String extraInfo;
    @Column(nullable = true)
    private Money startingBalance;

    @Column(nullable = true)
    @Temporal(TemporalType.DATE)
    private Calendar statementDate;

    @OneToMany(cascade = CascadeType.ALL, fetch = FetchType.EAGER, mappedBy = "account", orphanRemoval = true)
    private List<Transaction> transactions;

    @ManyToOne(cascade = CascadeType.ALL, fetch = FetchType.EAGER, optional = false)
    private User user;
}

org.hibernate.HibernateException: No Session found for current thread

I’m creating a little project at home to record some data and I’m using Spring & Hibernate. This problem took me a couple of hours to work out, so I thought I’d post my solution as it wasn’t listed in the 20 or so pages I visited to fix the problem.

So, you’ve just set up a DAO and wired the session factory, data source and transaction manager up, but your junit test is still reporting:

org.hibernate.HibernateException: No Session found for current thread

jdbc.properties

jdbc.driverClassName=org.hsqldb.jdbc.JDBCDriver
jdbc.url=jdbc:hsqldb:mem:testdb
jdbc.username=sa
jdbc.password=

application context

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context"
	xmlns:tx="http://www.springframework.org/schema/tx"
	xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
					    http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
					    http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx.xsd">

	<bean id="sessionFactory"
		class="org.springframework.orm.hibernate4.LocalSessionFactoryBean">
		<property name="packagesToScan" value="uk.co.vsf.repository.domain" />
		<property name="dataSource" ref="dataSource" />
		<property name="hibernateProperties">
			<props>
				<prop key="hibernate.dialect">org.hibernate.dialect.HSQLDialect</prop>
				<prop key="hibernate.hbm2ddl.auto">create</prop>
				<prop key="hibernate.show_sql">true</prop>
			</props>
		</property>
	</bean>

	<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource"
		destroy-method="close">
		<property name="driverClassName" value="${jdbc.driverClassName}" />
		<property name="url" value="${jdbc.url}" />
		<property name="username" value="${jdbc.username}" />
		<property name="password" value="${jdbc.password}" />
	</bean>

	<context:property-placeholder location="jdbc.properties" />

	<bean id="transactionManager"
		class="org.springframework.orm.hibernate4.HibernateTransactionManager">
		<property name="sessionFactory" ref="sessionFactory" />
	</bean>

	<tx:annotation-driven transaction-manager="transactionManager"
		proxy-target-class="true" />


	<bean id="baseDao" class="uk.co.vsf.repository.dao.impl.BaseDao">
		<property name="sessionFactory" ref="sessionFactory" />
	</bean>

	<bean id="readingDao"
		  class="uk.co.vsf.repository.dao.impl.ReadingDaoImpl"
		  parent="baseDao">
	</bean>

	<context:component-scan base-package="uk.co.vsf" />
</beans>

unit test

@ContextConfiguration(locations = { "classpath:aCtx-base.xml", "classpath:aCtx-repository.xml", "classpath:aCtx-dao.xml" })
@TransactionConfiguration(defaultRollback = true)
public class ReadingDaoTest extends AbstractJUnit4SpringContextTests {

    @Autowired
    private ReadingDao dao;

    @Autowired
    private SessionFactory sessionFactory;

    @Test
    public void readingsCount() {
        assertEquals(0, dao.totalReadings());
    }

    @Test
    public void saveReading() {

        assertEquals(0, dao.totalReadings());

        Reading reading = new ReadingImpl(BigInteger.valueOf(123));
        dao.addReading(reading);

        assertEquals(1, dao.totalReadings());
    }
}

What can possibly be wrong? Well as I say, I read solution after solution and still nothing fixed my problem, until I took another look at my unit test and realised I should be extending AbstractTransactionalJUnit4SpringContextTests!

Problem sorted – hope this helps someone else