Unnecessary data capture

Last week The Joseph Rowntree Reform Trust published a report us telling us that a quarter of all government databases are unnecessary and hold too much personal data, potentially breaching not only privacy laws but also human rights.

What’s the story behind this though? What drives governments to hold unnecessary information about us?

Like most large organisations, governments like to hold data ‘just in case’. For example, the UK Parliamentary Home Affairs Committee has strong reservations of the genuine need for more surveillance data being gathered by UK government agencies, but the defence here is that we might need it.

That is, there is no current need for holding the data, but there might be in the future. And because most of these systems take forever to change, preparing a system for ‘just in case’ saves time and money later, right?

Wrong.

The world’s leading manufacturing organisations have moved from a ‘just in case’ model to a ‘just in time’ model. Now, it’s far more complicated than that, but the these highly successful organisations do not just add features ‘just in case’. Instead, they have developed processes that allow them to develop products rapidly, allowing them to respond to changes in market conditions rather than hope that their best guesses work out.

In the software development world, this approach has been paralleled by the development and increasing adoption of ‘agile’ development practices. Scott Ambler’s Agile Unified Process (AUP) is a great starting place for businesses that are more comfortable with traditional rigid development methods and allows organisations to build their databases using a flexible framework that enables rapid adoption of new requirements when necessary.

This type of framework allows you to reduce the cost of delivery and maintenance, but allows you to respond quickly and add new features – even to highly complex systems – while maintaining and improving quality. It also bucks the rule that the later you add a requirement, the more it will cost. Agile software developments help to significantly reduce this ‘cost of escalation’ curve

So if it’s so good, why isn’t everyone doing it and succeeding at it?

Most large organisations have spent a great deal of time and effort developing consistent, repeatable delivery processes (most often waterfall methods) that capture learning form one project and make sure those lessons are applied to all future projects. On the face of it this sounds like they’re applying best practice. However, applying every lesson to every project is usually the wrong thing to do. Not every problem found in one project will re-occur in another one, and the delivery method can easily become a super heavyweight behemoth that frustrates every attempt to deliver at speed. I’ve worked in multiple large organisations where their fast track method took at least 9 months!

When faced with entrenched organisational culture that says 9 months is fast, the adoption of ‘agile’ methods is a real shock to the system. Many people struggle accepting that the new way is better, and this often leads to serious cultural clashes. Organisations seeking to reduce their delivery overheads and get to market quicker by adopting agile methods must do this in tandem with a top down cultural change programme, or the new methods will be seriously hampered.

And this is perhaps why governments still insist on capturing and keeping everything, because it’s too hard for them to introduce a change program that deals with the culture as well as the technology.

Other agile approaches like Lean Agile (my preferred approach when conditions allow), Scrum and Crystal Clear also have tremendous value (as well as AUP). Just remember that the full benefits of these methods will only come to fruition with strong executive support.