Email Price Checker – Part 2: Code Verifying and Splitting

In part 1 we built the basics of the app – and by basics, it really was basic. In this part, code checking and sub code splitting will be added.

To start with we’ll add sub code splitting.

This is heavily dependent on the website code and for the website I’m price checking (CPC) it uses the following code structures. The first code is the code itself. E.g. LP04169
Then there are the code + a two digit subcode starting at 01 up to 99. E.g. LP0416901 to LP0416999

The first class needed here is a class to represent the Item being priced. The item being priced has a code, sub code (where appropriate), a flag to indicate whether it was available, and a price with and without VAT.

package uk.co.vsf.domain;

import java.math.BigDecimal;

import org.apache.commons.lang.builder.ToStringBuilder;

/**
 * Represents an item that's being priced checked.
 */
public class Item {

	private String code;
	private String subCode;
	private boolean available;
	private BigDecimal excVat;
	private BigDecimal incVat;

	public Item(String code, String subCode) {
		this.code = code;
		this.subCode = subCode;
	}

	public Item(String code) {
		this.code = code;
	}

	public String toString() {
		ToStringBuilder tsb = new ToStringBuilder(this);
		tsb.append(code);
		tsb.append(subCode);
		tsb.append(available);
		tsb.append(excVat);
		tsb.append(incVat);

		return tsb.toString();
	}

	public String getCode() {
		return code;
	}

	public String getSubCode() {
		return subCode;
	}

	public boolean isAvailable() {
		return available;
	}

	public void setAvailable(boolean available) {
		this.available = available;
	}

	public BigDecimal getExcVat() {
		return excVat;
	}

	public void setExcVat(BigDecimal excVat) {
		this.excVat = excVat;
	}

	public BigDecimal getIncVat() {
		return incVat;
	}

	public void setIncVat(BigDecimal incVat) {
		this.incVat = incVat;
	}
}

In order to create the list of items that will be priced, we need to transform the payload to create an item for the code and all sub codes. So, a custom transformer has been used.

package uk.co.vsf.pricechecker.transformer;

import java.util.ArrayList;
import java.util.List;

import org.mule.api.transformer.TransformerException;
import org.mule.transformer.AbstractTransformer;

import uk.co.vsf.domain.Item;

/**
 * Transforms the incoming code into X number of sub codes.
 */
public class SubCodeTransformer extends AbstractTransformer {

	private int subCodeQuantity;

	@Override
	protected Object doTransform(Object src, String enc)
			throws TransformerException {
		if (!(src instanceof String)) {
			return src;
		}

		List<Item> items = new ArrayList<Item>();

		addInitialItem((String) src, items);
		addSubCodeItems((String) src, items);

		return items;
	}

	private void addInitialItem(String code, List<Item> items) {
		Item item = new Item(code);
		items.add(item);
	}

	private void addSubCodeItems(String code, List<Item> items) {
		for (int i = 0; i < subCodeQuantity; i++) {
			String subCode = (i < 10 ? "0" : "") + i;
			Item item = new Item(code, subCode);
			items.add(item);
		}
	}

	public void setSubCodeQuantity(int subCodeQuantity) {
		this.subCodeQuantity = subCodeQuantity;
	}
}

The next step it to modify the flow to include the transformer and a property to specify how many sub codes.

price-checker-10

The payload after the transformer will be a list of Items which then needs to be split so that each can be processed individually.

price-checker-11

And at this point it’s worth testing what we’ve got! So a logger has been added to the end to print out the message payload.

price-checker-12

Having sent a basic test email to the app, this is the log result:

Payload after split: uk.co.vsf.domain.Item@3e8ad1[Abc1225,96,false,<null>,<null>]
Payload after split: uk.co.vsf.domain.Item@bb1ee[Abc1225,97,false,<null>,<null>]
Payload after split: uk.co.vsf.domain.Item@1ce835b[Abc1225,98,false,<null>,<null>]
Payload after split: uk.co.vsf.domain.Item@11201a1[Abc1225,99,false,<null>,<null>]

The other requirement was to filter received messages containing a valid code. In the case of the website I’m price checking, the codes are 2 characters, 5 numbers, which makes for a good regex \w\w\d\d\d\d\d (or another permutation which does the same). If the code is valid, the item will be price checked, otherwise the message could be terminated, but in this case, it’s more useful to send the sender a message to say that the code isn’t valid. So a choice element makes more sense. One route will go to the splitter and the other to an SMTP endpoint.

In order to make the flow easier to read once the choice element has been added, a sub flow has been added to contain the happy path after the filter logic.

price-checker-14

price-checker-15

The filter default SMTP endpoint needs the from address from the email in order to send a response back to the user. This was a little tricky to find, but by putting full trace on, you’ll be able to see the entire Email properties and pick the one that suits best. Alternatively use a simple transformer (etc) and debug the entry to see the message properties.

When I actually tried to send a filter exception response email I kept getting a 501 warning about a host being needed on the address. It turns out that the from address shouldn’t be encoded like the user account… (things like that annoy me)

The complete flow code now looks like:

<?xml version="1.0" encoding="UTF-8"?>

<mule xmlns:smtps="http://www.mulesoft.org/schema/mule/smtps" xmlns:tracking="http://www.mulesoft.org/schema/mule/ee/tracking"
	xmlns:core="http://www.mulesoft.org/schema/mule/core" xmlns:context="http://www.springframework.org/schema/context"
	xmlns:pop3s="http://www.mulesoft.org/schema/mule/pop3s" xmlns:smtp="http://www.mulesoft.org/schema/mule/smtp"
	xmlns:email="http://www.mulesoft.org/schema/mule/email" xmlns:pop3="http://www.mulesoft.org/schema/mule/pop3"
	xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
	xmlns:spring="http://www.springframework.org/schema/beans" version="EE-3.3.2"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="
http://www.mulesoft.org/schema/mule/pop3s http://www.mulesoft.org/schema/mule/pop3s/current/mule-pop3s.xsd 
http://www.mulesoft.org/schema/mule/smtps http://www.mulesoft.org/schema/mule/smtps/current/mule-smtps.xsd 
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd 
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd 
http://www.mulesoft.org/schema/mule/ee/tracking http://www.mulesoft.org/schema/mule/ee/tracking/current/mule-tracking-ee.xsd 
http://www.mulesoft.org/schema/mule/email http://www.mulesoft.org/schema/mule/email/current/mule-email.xsd">

	<pop3s:connector name="Pop3sConnector" doc:name="POP3"
		checkFrequency="${email.check.frequency}">
		<reconnect-forever frequency="${email.reconnect.retry.frequency}" />
	</pop3s:connector>

	<context:property-placeholder location="pricechecker.properties" />
	
	<flow name="CPCPriceCheckerInbound" doc:name="CPCPriceCheckerInbound">
		<pop3s:inbound-endpoint host="${email.host}"
			port="${email.port.pop3}" user="${email.account}" password="${email.password}"
			responseTimeout="${email.response.timeout}" connector-ref="Pop3sConnector"
			doc:name="POP3" />
        <payload-type-filter expectedType="java.lang.String" doc:name="Payload"/>
        <custom-transformer
			class="uk.co.vsf.pricechecker.transformer.ReplyToAddressTransformer"
			doc:name="Java">
		</custom-transformer>
		<set-payload value="#[message.payload.trim()]" doc:name="Set Payload"/>
        <custom-transformer
			class="uk.co.vsf.pricechecker.transformer.CodeTransformer"
			doc:name="Java">
		</custom-transformer>
		<choice doc:name="Choice">
			<when expression="#[message.outboundProperties['CpcMatch']]">
				<processor-chain>
					<flow-ref name="TransformAndSplitLogic" doc:name="Flow Reference" />
				</processor-chain>
			</when>
			<otherwise>
				<processor-chain>
                    <set-payload value="Code: '#[message.payload.toString()]' is not a valid CPC code" doc:name="Set Payload"/>
					<smtps:outbound-endpoint host="${email.host}" port="${email.port.smtp}"
						responseTimeout="10000" doc:name="SMTP" 
						user="${email.account}" password="${email.password}"
						from="${email.from}" replyTo="${email.replyTo}" subject="${email.subject}"
						to="#[message.outboundProperties['CleanedReplyToAddress']]" />
				</processor-chain>
			</otherwise>
		</choice>
	</flow>
	<sub-flow name="TransformAndSplitLogic" doc:name="TransformAndSplitLogic">
		<custom-transformer
			class="uk.co.vsf.pricechecker.transformer.SubCodeTransformer"
			doc:name="Java">
			<spring:property name="subCodeQuantity" value="${number.of.sub.codes}" />
		</custom-transformer>
		<splitter expression="#[message.payload]" doc:name="Splitter" />
		<logger message="Payload after split: #[message.payload]"
			level="INFO" doc:name="Logger" />
	</sub-flow>
</mule>

There is a problem with the response subject if the filter hasn’t been met being a copy of the original request subject rather than the overridden properties value.

Two new transformers were also needed.

One for the code checking.

package uk.co.vsf.pricechecker.transformer;

import org.mule.api.MuleMessage;
import org.mule.api.transformer.TransformerException;
import org.mule.transformer.AbstractMessageTransformer;

/**
 * Determines the pricing website the code is applicable for.
 */
public class CodeTransformer extends AbstractMessageTransformer {

	private String cpcRegex = "\\w\\w\\d\\d\\d\\d\\d";

	@Override
	public Object transformMessage(MuleMessage message, String outputEncoding) throws TransformerException {
		if (!(message.getPayload() instanceof String)) {
			return message;
		}

		String payload = (String) message.getPayload();
		boolean cpcMatch = payload.matches(cpcRegex);

		message.setOutboundProperty("CpcMatch", cpcMatch);

		return message;
	}
}

And another for the toAddress to reply back to in the case of a failure / price response.

package uk.co.vsf.pricechecker.transformer;

import org.mule.api.MuleMessage;
import org.mule.api.transformer.TransformerException;
import org.mule.transformer.AbstractMessageTransformer;

/**
 * Transforms the incoming message replyToAddresses into a clean email address.
 */
public class ReplyToAddressTransformer extends AbstractMessageTransformer {
	public static final String CLEANED_REPLY_TO_ADDRESS = "CleanedReplyToAddress";

	@Override
	public Object transformMessage(MuleMessage message, String outputEncoding) throws TransformerException {
		String replyToAddresses = message.getInboundProperty("replyToAddresses");
		if (replyToAddresses == null) {
			throw new IllegalArgumentException("Need a replyToAddress in order to return email");
		}

		String[] parts = replyToAddresses.split("<");
		if (parts.length > 1) {
			parts = parts[1].split(">");
			if (parts.length == 1) {
				message.setOutboundProperty(CLEANED_REPLY_TO_ADDRESS, parts[0]);
				return message;
			}
		}

		throw new IllegalArgumentException("No valid replyToAddress in email - " + replyToAddresses);
	}
}

There are more improvements to be made if this were production ready code. For example, the ReplyToAddressTransformer might well come unstuck if there was an unusual to replyToAddresses value. Also the CodeTransformer needs the regex moving into properties.

In part 3 I’ll clean up the CodeTransformer and add in address filtering to prevent spam and unwanted email from going through the price checking application.

Email Price Checker – Part 1: Initial Setup

This post will walk through a complete project from start to scratch. Previously I’d created a small Selenium price checking test which runs great, but means that my pc has to run the app and no one can use it except me. This new project will build on that but receive an email, check the prices (slowly) and return an email with the cheapest codes back to the sender – that way I can use it from work 😉

For this app, I’m using Mule IDE, Java 1.6+, a Ubunutu server to run the application once written and an email address dedicated to listening for the price checking requests.

Lets get started!

To start with, we need a new project. This is using 3.3.2 of Mule because that’s the version I’d downloaded during the Mule training course. There is a newer version available, but it’s not necessary for now.

price-checker-1

I tend to always use Maven as it makes it a lot easier to pull in dependencies.

price-checker-2

So, once the projects been setup, we get the visual flow page ready to start building the application.

price-checker-3

At this point it might be worth explaining what I want to achieve…
I need the application to:

  • Read emails from an email account.
  • Check they originate from a specific set of addresses to filter out rubbish.
  • Check they contain a code for the website we will be checking prices on (CPC).
  • Delay the message if there is already one in progress in order to prevent us being banned on the CPC website.
  • Each code contains 100 price sub codes – check the prices one at a time and delay between each sub code to prevent being banned.
  • Return the lowest codes and the price ex+inc vat to the sender.

First we need an email endpoint and connector for both sending and receiving of emails. I’m using POP3, but you could use IMAP. Additionally the body of the email will only contain the code needed to price check, so the email to string transformer has been chosen.

price-checker-4

price-checker-5

Or in code terms, that looks like:

<?xml version="1.0" encoding="UTF-8"?>

<mule xmlns:context="http://www.springframework.org/schema/context" xmlns:pop3s="http://www.mulesoft.org/schema/mule/pop3s"
	xmlns:smtp="http://www.mulesoft.org/schema/mule/smtp" xmlns:email="http://www.mulesoft.org/schema/mule/email"
	xmlns:pop3="http://www.mulesoft.org/schema/mule/pop3" xmlns="http://www.mulesoft.org/schema/mule/core"
	xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
	xmlns:spring="http://www.springframework.org/schema/beans" version="EE-3.3.2"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="
http://www.mulesoft.org/schema/mule/pop3s http://www.mulesoft.org/schema/mule/pop3s/current/mule-pop3s.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/smtp http://www.mulesoft.org/schema/mule/smtp/current/mule-smtp.xsd
http://www.mulesoft.org/schema/mule/email http://www.mulesoft.org/schema/mule/email/current/mule-email.xsd">

	<pop3s:connector name="Pop3sConnector" doc:name="POP3"/>

    <context:property-placeholder location="pricechecker.properties"/>

	<flow name="PriceCheckerFlow1" doc:name="PriceCheckerFlow1">
		<pop3s:inbound-endpoint host="${email.host}"
			port="${email.port}" user="${email.account}" password="${email.password}"
			responseTimeout="240000" connector-ref="Pop3sConnector" doc:name="POP3"/>
        <logger message="Message Received: #[message.payload]" level="DEBUG" doc:name="Logger"/>
	</flow>
</mule>

And the properties file mentioned contains the following:

price-checker-6

At this point, I’m going to test what I have by adding a logger to see what the payload is before I continue.

price-checker-7

Run the application by right clicking on the project –> Run As –> Mule Application.

At this point I spent the next 8 hours or more tearing my hair out as I could not get anything to connect to my email account… After installing 5 different Comodo SSL certificates and the one for my web hosting, I still couldn’t connect. These are some useful settings for anyone in the same position:

-Dmail.debug=true
-Djavax.net.debug=all

When I looked very carefully at the certificates being checked, I found that avast! (antivirus) was interfering with the certificate hashes meaning the connection wouldn’t authenticate!

When eventually you do get it communicating with your email box, having sent an email to the box…

price-checker-8

…you’d see something like this in the logs:

price-checker-9