Positive Reinforcement

I’m keen for my application to keep the employee interested and involved in their job, perhaps even in the gaming sense I’m keen for my application to reward the employee for their positive behaviour. Now while this could sound rather sinister, as does the creepy behavioural title of this post, that’s not my intention, so I’ll try to explain.

In my previous job we used an ageing piece of software called ClearDDTS to track defects and change requests. This had a very simple feature whereby it emailed you when the status of the defect or change-request you were working-on changed. So if the defect went through test successfully, you’d get a mail, or if the defect was finally closed once the customer had approved it, you’d get a mail. The email was useful for example if your defect had been re-opened so you knew to look at it again, but other than that I feel the potential was under-utilised.

I’d like to use similar techniques in my application, but not only to keep people informed of the status of their previous (or upcoming) work, but also to keep people interested. Ideas from modern social media crop-up here too – these days, staff are often spread around the globe and I know only too well how annoying it can be receiving an email or a call from someone you’ve never even heard of, asking you why something you’ve done works the way it does.

Before I go any further, I’m not building a social networking tool, nor am I building a defect tracking system. The software I’m focused-on is designed in part (it has many facets) to aid communication and collaboration within large enterprises, and part of the purpose is to track work done on (and changes made to) software.

As some examples of the kinds of things I’m thinking of: I’d like to see the user informed when work they’ve done goes into test, I’d like them to know who is doing that testing, what that person’s bio is and what they look like. I’d like to see the user informed when the work they’ve done is delivered to the customer, I’d like them to see any customer feedback, a log of what occurred during the installation and who performed it. I’d like to see the user informed when someone else in their team takes their changes to incorporate into their own work. I’d like the user to be able to see the work of those around them, so they can understand at a glance how it impacts them. Most of all I want this visualised in a gui. Part of this is a news feed (how very Facebook), but I’d also like a visual representation of it.

I’ve been thinking about this on and off for a while and it led to a 4:30 or so wake-up one morning with my brain buzzing randomly about the standard Organisation Chart. That led to these thoughts. The video may be slightly jokey, but my point is serious. The Org Chart tells management how many people they manage, but it doesn’t describe to the individual their daily communication, or their responsibilities, or their worth. Let’s face it, if you’re at the bottom of an Org Chart, where most are, you’d be forgiven sometimes for thinking that the work you’re doing is just a drop in the ocean.

If the Org Chart demonstrates to you how insignificant you are, how much benefit would there be to providing an employee with an ever-changing, constantly up-to-date diagram of where they sit in their organisation? What impact does what they do have? Lots, generally. When you fix a defect in a piece of software, how many other people work with what you do? How many people are impacted before it goes out the door? The fact that you introduced a bug, or forgot to document something clearly, how many people did this negatively impact? What were their names? What do they look like? Where did they go on holiday? (You can mention this when you contact them to apologise.) How much better would I feel if I could see visually the good that came of what I did today, and when it’s negative, perhaps it would improve me as an employee to know.

Some of this wasn’t a problem in the ‘old days’ when people generally only worked with others in the same room, but large companies now are run across many countries, and many of us work daily with two or three other countries. I think this is genuinely great for the individual, it’s one of the most wonderful things about my previous job, all the people from so many different countries that I worked with, it’s just that companies haven’t yet evolved to take any other advantage of this other than the reduced cost.

The Org Chart has no place in my application, but I feel a visual representation is important. I’m imagining a diagram that shows you, and your work, and your links to your peers (whether local or remote). I’d like this to be active and clickable so the user can drill down, or scroll around and find out a little more about their peers – who they are and what they are working on.

To my mind this will give an employee not only an idea of what is upcoming for them and what has happened to the work they’ve done previously, but it will also give them an understanding of what is going on around them, and a sense of worth and position in their organisation. So while the title of the post could be seen as creepy, I’d rather the emphasis was on the word positive rather than the word reinforcement, and the ideas thought of as a benefit to the everyday enjoyment of the employee, as well as, as a benefit to the organisation they work for.

Tagged , ,

An XA Filesystem, update

In a comment on my original post, Alessandro Apostoli asks for an example of how to use the XAFileResource class, and specifically how to pass the Xid.

Digging this out has proved an interesting exercise (the code didn’t work at all any more, as I had expected), and it has raised two points worth noting. The second one I believe may be the answer to Alessandro’s question.

The EJB test code is included at the bottom of this post. It was written for Glassfish v2ur2, and I’ve tested that it all works. If you would like a copy of my Eclipse workspace containing the EJB, drop me an email – mdneale on hotmail.com matt@matthewneale.net.

1. String representation of a Xid

As I wrote in the original post, I’m not actually using EJBs in my application, and I’m not using an application server, though I do have a standalone J2EE Transaction Manager. The original tests were done against an application server so that I could prove the theory worked before piling-in more unknowns.

As is noted in the comment at the top of the XAFileResource class, for it to work, the Xid class that is passed to it must have a toString() method which returns a String representation of the Xid. Sadly the Glassfish Xid implementation com.sun.jts.jtsxa.XID doesn’t – invoking toString() on it always returns null. This probably makes sense, as com.sun.jts.jtsxa.XID can’t really know which String form would be useful.

Going back to my original very rough implementation of the XAFileResource class, I had a method in there which converted a Xid to a String, to workaround this exact problem. The problem doesn’t occur in my application as my Xid implementation does provide a String representation when toString() is called.

How you design this is up to you, but for simplicity’s sake, you can just add the following xidToString() method to the XAFileResource class, and change all calls to xid.toString() to a call to this new method. You’ll probably also want to look at the other uses of ‘xid’ as the rest are in log4j debug statements, and you probably will want to see the String representation there too, rather than just “null”.

protected String xidToString(Xid xid) {
	String xidStr = null;

	if (xid != null) {
        	String globalTransactionIdStr = "";
        	String branchQualifierStr = "";

        	if (xid.getGlobalTransactionId() != null) {
                	globalTransactionIdStr = new BigInteger(1, xid.getGlobalTransactionId()).toString(16);
        	}

        	if (xid.getBranchQualifier() != null) {
                	branchQualifierStr = new BigInteger(1, xid.getBranchQualifier()).toString(16);
        	}

        	xidStr = globalTransactionIdStr + "." + branchQualifierStr;
	}

	return xidStr;
}

One word of warning – the Commons Transaction FileResourceManager uses the transaction Id passed to it (the string representation of the Xid) to create a directory on the filesystem, and therefore the String representation needs to be within the limits the filesystem imposes on filenames. The above works fine for me (Glassfish on OS X 10.5), but watch out for that on other Application Servers and Operating Systems.

2. Connection Class

In my original post I alluded to the need for a Connection class, but I didn’t want to include it as I didn’t want to confuse the original intent, which was to communicate the XAResource implementation.  However, looking back at the original test code I can see it could be confusing without a note about it.

The issue is that when the XAFileResource is enlisted in the transaction, the Transaction Manager passes it the Xid to use. However, outside the Transaction Manager and the XAFileResource class, the Xid isn’t visible, so how do we get hold of the Xid in order to pass it to further calls to FileResourceManager (for instance for creating files, writing to files etc)?

The way I dealt with this in my original EJB test case was to store it in a static – a quick dirty fix for the test case, but of no further use as it won’t work for more than one transaction.

The way I’ve dealt with this in my application is to write an XAFileConnection class, which wraps an XAFileResource, but also implements XAResource itself. The XAFileConnection is what is enlisted in the transaction, and thefore, when the start() method is called on it we can store the Xid for use later. The idea is that you keep one XAFileResource class per FileResourceManger, and one XAFileConnection per transaction. This means you need some kind of tracking to ensure one connection per transaction. In a J2EE environment this is normally taken care of for you, but here you’ll have to write some kind of DataSource.

I don’t want to reproduce my entire XAFileConnection code here, as it’s large and repetitive. You’ll get the idea from a simple example. The following is what is used now in the EJB test case:

package txtest;

import javax.transaction.xa.XAException;
import javax.transaction.xa.XAResource;
import javax.transaction.xa.Xid;

public class XAFileConnection implements XAResource {

	private XAFileResource xaFileResource = null;
	private Xid storedXid = null;

	public XAFileConnection(XAFileResource xaFileResource) {
		this.xaFileResource = xaFileResource;
	}

	/**
	 * The start method stores the Xid for this 'connection' and then invokes
	 * the XAFileResource.
	 */
	public void start(Xid xid, int flags) throws XAException {
		this.storedXid = xid;
		xaFileResource.start(xid, flags);
	}

	// The rest of the methods just pass through to the XAFileResource

	public void commit(Xid xid, boolean onePhase) throws XAException {
		xaFileResource.commit(xid, onePhase);
	}

	public void end(Xid xid, int flags) throws XAException {
		xaFileResource.end(xid, flags);
	}

	public void forget(Xid xid) throws XAException {
		xaFileResource.forget(xid);
	}

	public int getTransactionTimeout() throws XAException {
		return xaFileResource.getTransactionTimeout();
	}

	public boolean isSameRM(XAResource xares) throws XAException {
		return xaFileResource.isSameRM(xares);
	}

	public int prepare(Xid xid) throws XAException {
		return xaFileResource.prepare(xid);
	}

	public Xid[] recover(int flag) throws XAException {
		return xaFileResource.recover(flag);
	}

	public void rollback(Xid xid) throws XAException {
		xaFileResource.rollback(xid);
	}

	public boolean setTransactionTimeout(int seconds) throws XAException {
		return xaFileResource.setTransactionTimeout(seconds);
	}

	public Xid getXid() {
		return storedXid;
	}

	/*
	 * Better to do this, and hide the FileResourceManager inside XAFileConnection.
	 * This way you can remove the getXid() method.
	public void createResource(Object resourceId) throws ResourceManagerException {
	}

	public OutputStream writeResource(Object resourceId) throws ResourceManagerException {
	}
	*/

}

If you look at the EJB test code below you’ll see how it all hangs together. In my real code, I don’t return the Xid as I do here. What I’ve done is add methods to XAFileConnection to perform all the FileResourceManager’s tasks – i.e. createResource, writeResource etc. These don’t take a txid parameter any more, as it’s known to the XAFileConnection class. This way we’ve now completely abstracted the actual FileResourceManager.

3. The EJB test code

The one EJB java file is reproduced below with its remote interface. As I mentioned at the top of this post, if you would like an archive of my Eclipse workspace, just send me an email.

The EJB creates a file, and inserts a row into a table. The first argument to the method test() is used as the filename and the data inserted into the table. The second test() argument says whether we want to commit or not. If we don’t, then an exception is thrown and the container-managed transaction rolls-back.

You’ll notice the xidToString() method is duplicated here – this will go away if you hide the FileResourceManager inside the XAFileConnection class as mentioned above.

package txtest;

import java.io.IOException;
import java.io.OutputStream;
import java.io.PrintWriter;
import java.math.BigInteger;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.SQLException;

import javax.ejb.Stateless;
import javax.ejb.TransactionAttribute;
import javax.ejb.TransactionAttributeType;
import javax.ejb.TransactionManagement;
import javax.ejb.TransactionManagementType;
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import javax.sql.DataSource;
import javax.transaction.RollbackException;
import javax.transaction.SystemException;
import javax.transaction.Transaction;
import javax.transaction.TransactionManager;
import javax.transaction.xa.XAException;
import javax.transaction.xa.Xid;

import org.apache.commons.transaction.file.FileResourceManager;
import org.apache.commons.transaction.file.ResourceManagerException;
import org.apache.commons.transaction.file.ResourceManagerSystemException;
import org.apache.commons.transaction.util.Log4jLogger;
import org.apache.commons.transaction.util.LoggerFacade;
import org.apache.log4j.Logger;

@Stateless
@TransactionManagement(TransactionManagementType.CONTAINER)
public class XATestEJB implements XATestEJBRemote {

	private static final Logger logger = Logger.getLogger(XATestEJB.class);

	private static FileResourceManager fileResourceManager = null;

	static {
		LoggerFacade loggerFacade = new Log4jLogger(logger);

		fileResourceManager = new FileResourceManager(
				"/Users/mdneale/xa-test/base",
				"/Users/mdneale/xa-test/work",
				false,
				loggerFacade);

		try {
			fileResourceManager.start();
		} catch (ResourceManagerSystemException e) {
			logger.error(e);
		}
	}

	@TransactionAttribute(TransactionAttributeType.REQUIRED)
    public void test(String filename, boolean commit) throws XATestEJBException {
    	logger.debug("test(filename=" + filename + ", commit=" + commit + ")");

		try {
			insertRow(filename);
			createFile(filename);

			if (!commit) {
				throw new XATestEJBException("Forced Rollback");
			}
		} catch (Exception e) {
			logger.error(e);
			throw new XATestEJBException(e);
		}
    }

	protected void insertRow(String filename) throws SQLException, NamingException {
		Context context = new InitialContext();

		DataSource dataSource = (DataSource)context.lookup("jdbc/xatest");

		Connection connection = null;
		PreparedStatement statement = null;

		try {
			connection = dataSource.getConnection();

			statement = connection.prepareStatement("INSERT INTO TEST ( filename ) VALUES ( ? )");
			statement.setString(1, filename);

			statement.executeUpdate();
		} finally {
			if (statement != null) { statement.close(); }
			if (connection != null) { connection.close(); }
		}
	}

	protected void createFile(String filename) throws NamingException, SystemException, XAException, IllegalStateException, RollbackException, ResourceManagerException, IOException, XATestEJBException {
		Context context = new InitialContext();

		TransactionManager transactionManager = (TransactionManager)context.lookup("java:appserver/TransactionManager");

		Transaction transaction = transactionManager.getTransaction();

		// Create the XAFileResource
		XAFileResource xaFileResource = new XAFileResource(fileResourceManager);

		// Wrap the XAFileResource in an XAFileConnection
		XAFileConnection xaFileConnection = new XAFileConnection(xaFileResource);

		if (!transaction.enlistResource(xaFileConnection)) {
			throw new XATestEJBException("Failed to enlist resource in transaction");
		}

		fileResourceManager.createResource(xidToString(xaFileConnection.getXid()), filename);

		OutputStream outputStream = fileResourceManager.writeResource(xidToString(xaFileConnection.getXid()), filename);

		PrintWriter writer = new PrintWriter(outputStream);

		writer.print("HELLO");
		writer.flush();

		outputStream.close();
	}

	protected String xidToString(Xid xid) {
		String xidStr = null;

		if (xid != null) {
	        String globalTransactionIdStr = "";
	        String branchQualifierStr = "";

	        if (xid.getGlobalTransactionId() != null) {
                globalTransactionIdStr = new BigInteger(1, xid.getGlobalTransactionId()).toString(16);
	        }

	        if (xid.getBranchQualifier() != null) {
                branchQualifierStr = new BigInteger(1, xid.getBranchQualifier()).toString(16);
	        }

	        xidStr = globalTransactionIdStr + "." + branchQualifierStr;
		}

		return xidStr;
	}

}

The simple Remote interface for completeness:

package txtest;

import javax.ejb.Remote;

@Remote
public interface XATestEJBRemote {
	
    public void test(String filename, boolean commit) throws XATestEJBException;
    
}

Notes:
1. This test uses Glassfish v2ur2, MySQL 5.0, MySQL Connector/J 5.1.7 and Commons Transactions 1.2.
2. My database has one table CREATE TABLE TEST ( filename VARCHAR(50) NOT NULL ).
3. Filesystem directories are defined in the static block at the top of XATestEJB.
4. Nothing special exists in XATestEJBException, except that it is defined with the annotation @ApplicationException(rollback = true).

Tagged , ,

Did Java do us a favour, or no?

Gripe of the day alert!

In the old days, when I was a C programmer (when we were all C programmers), reading a null pointer was just about the worst thing you could do, apart from perhaps writing to one. Then along came Java – in many ways much nicer, it has a NullPointerException, so your program prints a nice message and doesn’t just core dump.

My gripe is that this fact has made people lazy. Whereas I used to be completely anal about checking pointers in C, I’ve become much more relaxed. I’ve noticed this laziness in my code recently and I’m going to change because…

A NullPointerException is not acceptable, EVER! (In my opinion of course). I hate getting them, whether in commercial software, in open-source software, in free software, or in my software. It tells you something is badly wrong, but rarely gives a user any insight into which of the million parameters that affect the running of the software is to blame.

One well-respected piece of open-source software was NPEing all over me recently and it was “right wracking me off”. Yes, I know I was fiddling around with it trying to get something obscure to work, and when I found out how to make it work, it didn’t NPE, but a well written piece of software doesn’t NPE, and if you’re calling an exposed API, then that API should be ultra-defensive.

Gripe off.

Tagged ,

Advent Calendar Disgrace!

Kinder Advent Calendar 2008

I had to spend a few minutes the other day explaining to my girlfriend, and consoling her -  it didn’t matter that the first door on the advent calendar (which I was opening) was so much larger than the second door (which she was opening), as over the 24 days they are bound to be organised so they even-out. If not they would be the cause of endless fights between children.

Oops. No: first door – large; second door – tiny; third door – large; fourth door – tiny; fifth door – large; sixth door – large (ooh); seventh door – large; eighth door – tiny…

There is one bonus for the person going second though – door number 24 is huge and most likely has a Kinder Surprise behind it! This got me thinking that actually the doors were cleverly organised and would even-out, and as I’ve obviously too much time on my hands, I decided to add up the door areas. [This is getting sad I know]. The results are:

2 Children

  • Child 1 opens 205.36 sq cm of door (41% more door than Child 2!)
  • Child 2 opens 145.57 sq cm of door

3 Children

  • Child 1 opens 106.36 sq cm of door
  • Child 2 opens 91.74 sq cm of door
  • Child 3 opens 152.83 sq cm of door

This doesn’t seem very fair.  I can hear the screams of a million disappointed children from here! When I was young, the three of us shared one advent calendar. Is this not the case any more? Perhaps we were an anomaly, and normally children had one each?

My final thought is – is it possible to order a reasonable number of different sized doors (say 3 sizes) on a calendar so that it wouldn’t matter whether 2, 3 or 4 children used the calendar, they would all get an equal total door size?  Call it a kind of Christmas puzzle!

Tagged ,

An XA Filesystem

A long post, sorry!  Most is pasted code though.

Update: If you get to the bottom of this post, you may be interested in this update.

One of the underlying design tenets of my application is that everything should happen, or nothing should happen. This covers all the database access, as well as file access, and there will be a lot of the latter. This means transactions for databases and transactions for filesystems, and since both database changes and filesystem changes will be required together, this means XA transactions across the database(s) and the filesystem(s).

There are two big problems with the above requirement: firstly, DDL is rarely transactional (PostgreSQL and Informix can do it, but since choice of RDBMS is my customer’s, not mine, I can’t rely on them); and secondly, filesystems aren’t transactional either.

The DDL issue is a major one, and one not easily resolved. Given that it won’t be transactional, is it worth pursuing transactions in the other areas? I think so – the effort of writing recovery code is so much more otherwise. The layer which abstracts the application from the database can be written to understand the capabilities of the RDBMS – so if transactional DDL is supported, then it can be used.

That’s for another day though, today I’m concerned with filesystem transactions. Luckily here some work has been done. Filesystem transactions in Java are supported by the Commons Transaction Apache project. I used this really nice step-by-step tutorial to get up and running.


package txtest;

import org.apache.commons.transaction.file.FileResourceManager;
import org.apache.commons.transaction.util.Log4jLogger;
import org.apache.log4j.Logger;
import org.junit.Test;

public class CommonsTxTest {
    private static final Logger logger = Logger.getLogger(CommonsTxTest.class);

    @Test
    public void test() {
        try {
            FileResourceManager fileResourceManager = new FileResourceManager(
                    "/Users/mdneale/txtest/store",
                    "/Users/mdneale/txtest/work",
                    false,
                    new Log4jLogger(logger));

            fileResourceManager.start();

            String txId = fileResourceManager.generatedUniqueTxId();

            fileResourceManager.startTransaction(txId);
            fileResourceManager.createResource(txId, "new.file");
            fileResourceManager.commitTransaction(txId);

            fileResourceManager.stop(FileResourceManager.SHUTDOWN_MODE_NORMAL);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

And it works. Running this short program creates the new file “new.file” in the folder /Users/mdneale/txtest/store. Now swap the line:

fileResourceManager.commitTransaction(txId);

for:

fileResourceManager.rollbackTransaction(txId);

Remove the file “new.file” that was created by the last run of the program, and run the program again. This time the file doesn’t appear. So far, so good.

The next step is XA. I’m going to explain here how to interface the Commons Transaction File Resource Manager to a J2EE Transaction Manager so that the Transaction Manager can control the resource as part of an XA Transaction.  It shouldn’t be difficult to convert it for a non-J2EE Transaction Manager if necessary.
Continue reading

Tagged , , , , ,

Book Meme

“Bourdelle had spent a decade working on the details, gradually building them up on a pivot that turned on a naked warrior whose outstretched left arm, it is true, defies the laws of anatomy – though photographs show that Bourdelle used his own left arm as a model.”  From the chapter on Montparnasse in Metrostop Paris by Gregor Dallas.

The chapter is about the French Sculptor Antoine Bourdelle and the sentence describes the memorial sculpture for those who died in the Franco-Prussian War that he was commissioned to produce for his home town of Montauban.

Book Meme:

  • Grab the nearest book.
  • Open it to page 56.
  • Find the fifth sentence.
  • Post the text of the sentence in your journal along with these instructions.
  • Don’t dig for your favorite book, the cool book, or the intellectual one: pick the CLOSEST.

Book Meme seen on Leah Culver.

An excellent book by the way, if you’re interested in the history of Paris. The subject matter of each chapter is wildly different, so some chapters may interest you more than others, but on the whole very interesting.

By the way, each day I feel a little more guilty about not writing anything here.  The excuse is that I’m very busy, but that’s always the excuse isn’t it? This will galvanise me into action…

Tagged

10 things I’ll remember about you…

Purely for amusement – ten slightly odd captures from around my desk and office in the weeks leading up to my last day at Comverse.  I wanted to be able to remind myself in future of what I saw around me every day.  Each photo has a description if you click through to the photos on flickr.


Desk Detail #1Desk Detail #2Desk Detail #3Desk Detail #4Desk Detail #5

Desk Detail #6SDG Musical ToysDesk Detail #7Goodbye Adelphi Building, John Adam StThe Last Muffin
Tagged ,

Who am I? What is this blog for?

It’s now a week until my employment at Comverse ends and my new (ad)venture begins. For somewhere between six months and a year I’m bootstrapping myself to flesh-out and build some technologies that have been rattling around in my head for some time. I don’t know at this point exactly where this will lead, the ideas are dragging me in a few directions, but it will be a challenge, and for that reason alone it’s worth undertaking.

After nearly fifteen years in the software industry, I’ve seen much of the good and bad. After the first few weeks of my first job in 1995 I was left wondering why, though we were writing exciting software to solve our client’s problems, we didn’t harness software in the same way to solve our own problems. Between that point and today the complexity of designing, developing, testing, configuring, shipping and maintaining software has got ever more complicated. Through all this increase in complexity, I haven’t seen software that looks inwards and solves the problems inherent with that increased complexity.

Now there has been some improvement, and the more I look, the more I see. Since I started thinking about this venture, I’ve uncovered huge numbers of really exciting things happening that I never knew about. So, perhaps the first thing I’ll use this blog for is to highlight some of those. My bookmarks are full of companies writing software tools that may revolutionise what we’re doing. However, in amongst all of them I still can’t find what I’m looking for.

So what is this blog for? There was one idea above, but other than that I’m not entirely clear. It’s definitely a page to represent me beginning something new, and I suspect that for the time being it may just be a source of amusement for myself. However over the next year I’m going to have to learn a lot – and maybe (if I have time) I’ll share some of it here.  I’ll also definitely need some feedback on what I’m doing, so having a place to document and explain what I’m doing will also be useful.

If you work in the software industry, whether in development, test, professional services or management and you’re interested in harnessing technology to improve and automate the software life-cycle, or if you’re part of a globally distributed team and want to improve collaboration within your team, maybe I’ll have something here for you.

I know I’ve not told you much, sorry for that. There will be more to come.

Follow

Get every new post delivered to your Inbox.

Join 113 other followers