Figure 1 – Hitting the Data Quality ‘Bulls Eye’.

[source: Steve Nimmons]

Data is at the core of every business. Data Quality is a consequence of many factors.

Applications act on data, processes act on applications, people interact with applications via processes.

Figure 2 – Factors which impact Data Quality (abridged)

[source: Steve Nimmons]

People have varying abilities, interest levels, workloads, levels of business understanding and experience. Their role in the Information System is critical. In most circumstances they are the source of the data and responsible for its input into (or consumption from) the Information System. Their ability to fulfil this important role is variable and compounded by (not exhaustive):

• Business Process Complexity
• Training Levels
• Availability of exemplars
• Intuitive nature of processes
• Variability of processes
• Constraints within processes (e.g. number of paths, degree of control applied)
• Quality of online help
• Review processes and audits
• Compliance mandates and consequences of non-compliance
• Learning curves relating to processes and use of applications
• Clear definitions of roles and responsibilities
• Churn rates
• Change (e.g. legislation, process changes, application changes)
• Design quality of applications
• An agreed common language

The Data Quality Blame Game

Data Quality is a consequence of interacting variables. When Data Quality drifts:

• Application designers argue from a position of “Garbage in / Garbage out”
• Process and Application designers promote the “stupid or careless users” argument
• Users blame process complexity and lack of intuitive functionality in applications (as well as over-work and lack of training)
• Recipients of reports based on tainted data ‘panic’
• Managers commence a ‘quest for the Golden Hammer’ in an attempt to find a single simple fix (which is forlorn).

The Approach to Take

Tackling Data Quality issues requires an Information Systems focused approach (respecting key variables across People, Processes and Applications). Think about how best to formulate Data Quality countermeasures by focusing iteratively on the weakest links. High-level steps:

• Quantify the actual Data Quality issue (sometimes hearsay is a false amplifier of the real size of the problem)
• Prioritise data items where Data Quality is paramount (those linked to KPI’s, statutory reporting, financial reporting and forecasting for example). Non-business critical data items could be tackled in later phases
• Hypothesise cause of Data Quality drift for prioritised data items
• Substantiate hypotheses – e.g. check against actual circumstances (focus groups, user interviews, remediation programmes in place)
• Check for compound problems – e.g. lack of training and only very occasional use of a specific process will often lead to inconsistent outcomes
• Look for badly designed applications (semantic confusion, no input validation or mandatory fields)
• Check governance and compliance processes. Are there any penalties for non-compliance with that mandated?
• Check for badly integrated applications/processes – e.g. manual re-keying of data or disconnected processes (e.g. system A doesn’t notify system B of an important data change)

Most importantly, do not be drawn towards the simplistic conclusions of the “Data Quality Blame Game.” Information Systems work (much like an orchestra) when all parts are in harmony. The interaction of users with processes and applications is non trivial. Data Quality is a consequence, and hence a symptom of misalignment of variables listed above. Focus on what needs to be tuned, then tune again!

Rules, guidelines, laws and judgement

A few weeks back I attended an interesting debate and presentation about Business Process Management (BPM) in the Public Sector. In fact this was a Government Panel discussion led by a consultant who was ‘much singing’ the praises of BPM backed by enterprise rule systems (ERMS). If you’re not familiar with such patterns, the basic premise is the process is separated from the business logic which is crafted and maintained in a separate, dedicated system and ‘consulted’ as necessary in key decision steps in the business process.

It was a solid pitch, I certainly bought into most if not all of the approach (as long as we don’t go crazy with rule specification in the ERMS). My rule of thumb is put things in there that are a) volatile and/or b) complex. Legislation is often complex and volatile, so many aspects of decision making in Public Sector processes fit the mould nicely.

What surprised me was the Q&A session that followed. We were (you see) in a room full of community leaders, many of them Justices of the Peace. Think ‘Wisdom of Solomon’ meets immovable ‘binary’ automated decision making.

The mathematician / engineer in me took the view that ‘the rule is the rule’,  a guiding truth by which the decision is made based on real-time inputs. The ‘interpretation of the Magistrate’ however was much different. The ‘rule’ is the questionable mandate, the hand tying aspect that removes judgement from human aspects of the process. Is the rule ‘good’, ‘bad’, ‘well-intentioned’, ‘lawful’, ‘open to interpretation’, ‘backed by precedent’ all tumbled out as arguments against stringent rule driven process definition.

I was bemused. Surely the rule(s) brought order to chaos, uniformity to individualism, defensibility and intrinsic fairness. Judgement should ‘overrule the rule’ was the basic counterpoint. This principle has worked (with life-saving grace) during a number of aircraft emergencies when the pilot switches off auto-pilot, so I do not readily dismiss.

But at the heart of this is a personal feeling of control and influence. If there are human factors in the decision process, we must define the boundaries of constraints. Law, best practices, company goals and values, regulation all set the parameters. The nature of rules define their ‘elasticity’. Some are naturally more ‘bendable’ than others. Some tasks and job functions require more elasticity, while others are easily more ‘binary’. This brings me to the point, rules are themselves often of quite an organic nature. Fear of uptake might be borne in a misconception of system rigidity, a loss of control, contribution, judgement, experience. As ‘propeller heads’ of the rules world, we need to exercise caution in describing their flexibility to business users, and maybe even ‘the general public’. Rules can be soft, although some must be ‘hard’. It’s important however to represent diversity and be cognisant of perception when explaining the operational aspects of these systems to potential users.

The Business Change folks should be astute at handling ‘emotional uptake and acceptance’, but let’s not make their lives harder by speaking too much as ‘purists and absolutists’. Some see such rule driven systems as a form of intellectual castration. Judgement, knowledge, experience, gut feel, intuition, innovation might be hampered by a lack of elasticity.  As the presenter found, there is a time to appeal to human psychology and its inherent resistance when faced with a perceived loss of control.

Article originally published with the BCS in Nov 2008

Panning for gold,  Steve Nimmons looks at complex event processing.

There is an inherent complexity in understanding the relationship between what can appear to be seemingly unconnected events occurring in real or near-real time. I make the temporal distinction as there are sophisticated business intelligence and data mining solutions for pattern or trend discovery in previously captured business information.

These are proven and do a very solid job in specific circumstances. There are also interesting extensions emerging in terms of mash-ups that augment and enrich more fully-featured end-user driven business intelligence solutions. Analysing events in real-time can be exceptionally informative, adding to the overall utility of business intelligence and providing a mechanism for business processes to react advantageously.

I describe complex event processing (CEP) as gold panning the elemental binary soup. CEP is trying to amplify the signal of interesting information against the noise of your event driven architecture. What is interesting is generally termed a ‘concept’ and CEP is all about trying to instantiate concepts from the flow of events.

CEP is typically achieved using forward chaining rules engines, themselves based on the RETE algorithm. There are a number of examples of good COTS products, I myself had the privilege of working on one prototype system that has now been successfully productised.

The business problem we faced was the correlation of events in a distributed mobile telecoms network. Tracing failures throughout the user experience was complex and put a large burden on operations and customer service representatives (CSRs).

The solution was to decode large volumes of real-time data from RADIUS servers, WAP servers, MMS, MML, SMS, micropayment systems et al, and hunt for failure patterns in the datasets (and present these in a simple consumable format). The result was exceptional operational saving, increased user satisfaction and greater insight into systems failures (which were then addressed).

Having ‘done battle’ with early systems I table the following implementation recommendations:

1. Trial candidate CEP systems in a proof of concept unless you are familiar with the solution, product and approach.
2. It is complex and a mutual frustration to produce decent test data with known results. When configuring a CEP system to look for ‘needles in haystacks’ it is very handy to actually know where and what some of them are. False positives, false negatives and correlations that fall inside or outside of configurable tolerances will need to be tested. Start thinking about this very early in the project cycle and engage domain experts that know the respective systems inside out. Make this ‘slick’ and you will get a better result at much lower cost.
3. Tuning is extremely important. Make sure your key attributes can be configured. This often (although not exclusively) relates to timeouts.
4. Ensure, unless by explicit design, that you can handle disasters and fail-over. This may sound obvious, but engines running in memory for ultimate speed may not provide you with necessary state management. An inelegant solution is to run two engines performing the same work. If you need to run at very high throughput volumes this must be part of non-functional testing in a proof of concept.
5. CEP can become an arcane mess and can be quite ‘organic’ in nature. Black-magic systems are only fun for propeller heads with little commercial responsibility, so avoid the ‘allure of mystery’.
6. Make sure the golden nuggets CEP expose are used. This can often be a feedback loop into other business processes. The key point is new information might be disruptive to existing processes, so plan ahead.
7. Depending on the implementation, presenting results to a ‘human’ through a worklist (workflow / BPM) is essential. This might be in the validation of suspect movements, purchases, funds transfers etc. Badly tuned CEP systems with inappropriate thresholds overwhelm the human tasks. Conversely a system that is ‘too loose’ can allow suspect events (potentially of a very serious nature) to pass undetected.

Solutions and industries in which I see increased CEP interest and uptake include: transport, financial services (fraud detection) and fault monitoring systems (cross industries). The latter case is very interesting and systems are being developed that seek to provide early warning of developing faults that generate event streams with tell-tale characteristics.

The definition of ‘tell-tale’ is of course rather slippery, with practically limitless variation between circumstances. This is at the heart of what makes CEP both interesting and challenging.