Figure 1: A (simplified) Big Data Ecosystem

[source: Steve Nimmons]

In terms of ‘forces’ affecting the CIO Agenda, Information Strategy and Enterprise Architecture, Big Data is increasingly important. This is due to explosive growth in number of data source types: applications, digital media, mobiles, users, customers, unstructured data sets, sensors, emails, blogs etc. Data is complex and in mixed formats (text, video, audio), on-demand infrastructure scalability (including massively scalable storage) is needed to deliver Big Data capabilities, as are robust analytics and visualisation tools and techniques for distributed, parallel systems. Increasing bandwidth availability has also led to exponential data growth rates and capabilities e.g. social networks, video and microblogging.

Where do you start in formulating a reference architecture for Big Data and sourcing suppliers for a Big Data ecosystem?

Should you believe the Hype?

The Gartner Hype Cycle places Big Data on ‘the upslope’ towards the ‘peak of inflated expectations’. Big Data is of course already underpinning many of the web giant’s architectures (typically because necessity has been the mother of invention).

Figure 2: Gartner Hype Cycle for Emerging Tech (2011)

[Source: Gartner]

• Facebook uses Hadoop to store copies of internal log and dimension data sources and as a source for reporting/analytics and machine learning. There are two clusters, a 1100-machine cluster with 8800 cores and about 12 PB raw storage and a a 300-machine cluster with 2400 cores and about 3 PB raw storage.
• Yahoo! deploys more than 100,000 CPUs in > 40,000 computers running Hadoop. The biggest cluster has 4500 nodes (2*4cpu boxes w 4*1TB disk & 16GB RAM). This is used to support research for Ad Systems and Web Search and to do scaling tests to support development of Hadoop on larger clusters
• eBay uses a 532 nodes cluster (8 * 532 cores, 5.3PB), Java MapReduce, Pig, Hive and HBase
• Twitter uses Hadoop to store and process tweets, log files, and other data generated across Twitter. They use Cloudera’s CDH2 distribution of Hadoop. They use both Scala and Java to access Hadoop’s MapReduce APIs as well as Pig, Avro, Hive, and Cassandra.

Other Hadoop users include:  1&1, A9.com, About.com, Amazon.com, American Airlines, AOL, Apple, Booz Allen Hamilton, Cerner, ChaCha, comScore, EHarmony, Federal Reserve Board of Governors, foursquare, Fox Interactive Media, Freebase, Hewlett-Packard, IBM, InMobi, ImageShack, ISI, Joost, Last.fm, LinkedIn, Microsoft, Meebo, Mendeley, Metaweb, Netflix, The New York Times, Ning, Outbrain, Playdom (now part of Disney Interactive Media Group), Powerset (now part of Microsoft), Rackspace, Razorfish, StumbleUpon and Twitter.

Hadoop Overview

Figure 3: Hadoop Overview

[source: Steve Nimmons]

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using a simple programming model. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.

Hadoop has Commons, MapReduce and Distributed File System capabilities (HDFS) as well as sub-projects: HBase, Cassandra, Avro, Hive, Mahout, Pig, ZooKeeper and Chukwa.

Given the pervasive nature of Hadoop, this is a strong contender for any Big Data implementation. HBase is the Hadoop database. Cassandra is also a NoSQL database. Mahout is a data mining and machine learning component, Hive and Pig are querying components, Zookeeper a coordination component.

Hadoop Distributions, such as that from Cloudera, bundle Apache Hadoop with other Open Source tools to create a more feature rich ‘platform’. The Cloudera distribution is definitely one to evaluate.

A simple Reference Model

In terms of implementing ‘Big Data’ architectures there are a number of choices, particularly in the visualisation and analytics space (refer to Figure 1). A simplified reference model is provided in Table 1. This will be expanded in a series of future posts on architectures for Big Data, exploring key features and design trade-offs.

Table 1: Simplified Big Data Reference Model

[source: Steve Nimmons]

Data Loaders (e.g. Sqoop) and log management (e.g. Flume, Scribe) could also be included in the reference model / ecosystem.

Further Reading and Interesting Tools

• Processing: http://processing.org. advanced visualizations
• Protovis: http://vis.stanford.edu/protovis/ 
• Gephi: http://gephi.org/ focused on Social Network Analysis
• Tableau: http://www.tableausoftware.com/public/
• ManyEyes: http://www-958.ibm.com/software/data/cognos/manyeyes/, from IBM.

VPEC-T Enterprise Architecture Mind Map (click to enlarge)

[source: Steve Nimmons]

VPEC-T is a Systems Thinking framework with 5 dimensions (Values, Policy, Events, Content and Trust).

VPEC-T is useful in the context of Enterprise Architecture, particularly in terms of exploring the full breadth of the Information System and related challenges.

Architectural Frameworks such as TOGAF will be very familiar to Enterprise Architects, VPEC-T perhaps less so. How then does VPEC-T’s dimensions knit together with familiar and traditional architectural concepts such as the views produced in the TOGAF ADM?

A high-level view is given in the MapMap above.

The PEC Dimensions

Unsurprisingly (and recognised in the book “Lost in Translation”) the PEC dimensions of Policy, Events and Content have a close mapping against Enterprise Architecture concerns such as Governance, Codification, Business Architecture, Information Architecture, Application Architecture, Technical Architecture, Security Architecture, and Integration Architecture.

The mappings below are intentionally high-level and therefore non-exhaustive. The purpose is to highlight the mapping between VPEC-T and high level architectural concepts.

The Policy Dimension maps naturally to:

• Governance
• Information Architecture in terms of Data Retention policies, Information Handling Models etc.
• Security Architecture in terms of security policy and legislation, Confidentiality, Integrity and Availability requirements and business continuity
• Business Architecture in terms or organisational design and roles and responsibilities.

The Events Dimension Maps Naturally To:

• Integration Architecture (SOA, REST etc.)
• Event Driven Architecture
• Information Architecture (Real-time analytics, Complex Event Processing, Pattern Based Strategy)
• Security Architecture – Protective Monitoring, SIEM, Fraud Detection, Alerts
• Strategy in terms of external events and reaction to market forces.

The Content Dimension Maps Naturally To:

• Codification (Patterns, Anti-Patterns, Reference Architecture and Principles)
• EA Artefacts (Models, Meta-Models, Semantics)
• Security Architecture (Protective Markings, Information Handling Models, Aggregation, Confidentiality, Integrity, Availability)
• Information Architecture (Canonical, Logical, Physical Data Models, Data Flows, Formats, Brokering, Transformation, Taxonomy, Big Data, Data Dictionary, Data Quality, BI)
• Technical Architecture concerns such as storage, scalability, sizing, availability, device independence, mobile access, consumerisation
• Strategy – reporting strategy, KPIs, tolerances

The V & T Dimensions

The Values and Trust dimensions are equally useful.

The Values Dimension Maps Naturally To:

• Values Analysis (Value Stream Analysis, Value Network Analysis etc.)
• Strategy (Ethics, Sustainability)
• Governance
• Target Operating Model (how do we want to do business and what do we stand for)
• Business Architecture – organisational design, corporate values and behaviours and how this affects the People and Process dimensions of the Information System.

The Trust Dimension Maps Naturally To:

• Security Architecture (Trust Domains, Identity Management, Registration, Enrolment, Authentication, Authorisation, Non-repudiation, digital signatures)
• Business Architecture (optimising organisational design, Social Network Analysis)
• Technical Architecture (Delivering Security Enforcing Functions in the infrastructure and application designs)
• Risk Management
• Data Architecture (data quality, provenance etc.)
• Stakeholder and Communications Management

The Beauty of VPEC-T

VPEC-T has many virtues. I think some of the most important are:

1. It maps naturally onto all of the principal Enterprise Architecture concerns (as described above)
2. It is simple to tailor for small / large problem domains
3. Problem solving is easily geared towards a particular dimension. This is a useful for gaining new perspective on old problems, or tackling system weaknesses
4. It focuses thinking on ‘soft’ and ‘hard’ aspects of problems. This ensures holistic Information Systems thinking, not just Information Technology ‘solutioning’
5. It is intuitive and has a shallow learning curve.

Further Reading and Thanks

Thanks to Nigel Green (VPEC-T co-creator) for providing comments on an early draft of the MindMap.

Some other resources for further study:

• http://taotwit.posterous.com/vpec-t-a-thinking-framework-presented-to-scio (by Nigel Green)
• http://ingenia.wordpress.com/2010/07/10/using-vpec-t-and-archimate/
• http://ingenia.wordpress.com/2010/12/08/how-to-elaborate-a-business-model-with-enterprise-architecture/
• http://servicefab.blogspot.com/2010/06/chris-bird-applying-p-e-c-sabre.html (PEC led Design, by Chris Bird)
• http://servicefab.blogspot.com/2006/08/problem-with-processes.html (the blog post that eventually gave birth to VPEC-T and Lost in Translation).

What ever became of Linda?

The back story

At university, Steve studied Computer Science and had a wide range of extra-curricular interests in cognition, economics, psychology, literature and philosophy.

1. 20 years on, Steve is a successful Technologist.
2. 20 years on, Steve is a successful Technologist and a Fellow of several professional institutions concerned with natural history, antiquaries and the arts.

The back story portrays Steve as a Computer Scientist and something of a (perhaps precocious) polymath. Statements 1 and 2 tell us that Steve still majors in Technology and has not become a full-time economist or philosopher (although we cannot rule out, that this is part of his career plan!). The hypothesis in Statement 2 (if correct) is richer than that in statement 1.

Is Statement 1 or 2 more probable?

The Conjunction Fallacy

The back-story ‘tempts’ a conjunction fallacy, which occurs when it is assumed that specific conditions are more probable than a single general one.

A frequently cited example of this fallacy comes from Amos Tversky and Daniel Kahneman:

Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations.

Which is more probable?

1. Linda is a bank teller.
2. Linda is a bank teller and is active in the feminist movement.

85% of those asked chose option 2, whereas the probability of two events occurring together (in “conjunction”) is always less than or equal to the probability of either one occurring alone.

This seems to occur because of the Representativeness Heuristic, in each case the more detailed second statement seems more representative of the person described in the back-story.

This goes beyond being an interesting anecdote. In Pattern Based Strategy, pattern detection and analysis (as well as hypotheses and probabilities of occurrence) need to be taken into account. To get ‘fooled by the back-story’ and the basics of probability (i.e. unless we know with absolute certainty that Steve is a serial Fellow, or Linda is an active feminist), then statement 2 in each case must be less probable.

Fooled by Representativeness

Nassim Taleb introduced us to ‘Fooled by Randomness’. There is also merit in not being fooled by detail, Representativeness and flawed understanding of probability. Pattern seeking could be skewed by conjunction fallacies and specious hypotheses.

“It was a sewing machine when I borrowed it.”

Kettle Logic is a type of informal fallacy in which multiple arguments are used to defend a point, but the arguments themselves are inconsistent.

Kettle Logic takes its name from a story related by Freud, of a man who was accused by his neighbour of having returned a kettle in a damaged condition. The neighbour offered three inconsistent arguments:

1. That the kettle had been returned undamaged
2. That the kettle was already damaged when it was borrowed
3. That the kettle had never been borrowed it in the first place

The inconsistency of argument is of course problematic, it would have been better if the neighbour had only chosen one.

Argument of this nature will be familiar to landlords (“it may have been on the inventory, but it was never in the property”, “it was damaged before I arrived”, “are you sure it was a kettle”, “perhaps it was a different kettle”).

Kettle Logic is a real issue in business. Freud’s example presents three clearly inconsistent arguments concurrently. Business problems tend to present inconsistent arguments over elongated periods with subtle (and usually (but not always) accidental) introduction.

This highlights a need for a Kettle Logic Trap to collate and detect contradictions in requirements, problem statements, hypotheses, business goals, strategy, risks and issues and organisational tensions.

VPEC-T is a systems thinking framework which guides problem solving using five dimensions, namely Values, Policy, Events, Content and Trust. The Values and Trust dimensions are particularly interesting in relation to Kettle Logic. The former, in the sense of personal and organisational values, and Trust in the sense of the nature of the contradictory argument (deliberate, accidental, concealment, openness etc.).

Trust in VPEC-T is concerned with the trusting (or otherwise) relationship between all parties engaged in a value system where Trust will be based on intimacy of the parties, one party’s credibility in the eyes of another, and the risks involved. Trust values change with time and circumstances. [source: Wikipedia]

The neighbour presenting the contradictory arguments above will quickly lose credibility and trust. Using VPEC-T filters in ‘real-world’ problem solving will help with detection of contradictory arguments emanating from interest groups, coalitions, and individual stakeholders.

Kettle Logic (as applied above) attempts to shift blame and introduce confusion. Contradiction and confusion may equally be introduced accidentally. Without a ‘detect and resolve’ mechanism there is a danger that Kettle Logic will linger, possibly trapping the problem solver into dealing with the contradiction in system design (solving the wrong problem) rather than identifying and removing fallacious argument.

The Process Fixation Anti Pattern

This is a common Anti Pattern, and in the world of Enterprise Architecture is a bedfellow of ‘Framework Abuse’. Certain Enterprise Architects become fixated with process, convincing themselves that meticulous implementation of every conceivable aspect and dimension of the framework they are using (TOGAF ADM or otherwise) is somehow practical and worthwhile. Of course this quickly turns into a waste of time, effort and money. ‘Modelling for Modelling’s sake’ snowballs, with increasingly ingenious reasons as to why the darkest recesses of the architecture need further elaboration. The great fallacy at hand is a belief that, ‘if we follow the process, we cannot go wrong’, ergo ‘if we follow the process to increasing levels of detail’ we can only increase our chances of success. When projects and architectural initiatives fail, adherence to the process is often rewarded with ‘blame amnesty’. Worst still is another fallacious conclusion that the reason for failure was lack of rigour and detail in the application of the process. This fuels more folly in future endeavours.

This Anti Pattern rears its head as closely related variants, such as: ‘the process becomes the deliverable’ or ‘the plan becomes the deliverable’.

The Anti Pattern Codified

Anti Pattern Name [Process Fixation (or sometimes ‘Seduced by the Process’).]

Type: [Process, Management]

Problem: [Process fixation seduces architects into what might be pejoratively referred to as ‘doing the framework’. Execution of the method becomes the deliverable, not the business critical deliverables themselves. They risk becoming‘near-accidental’ by-products.]

Context: [Inexperience (Seduced by the Process), The Defence of ‘I Followed the Process’ – i.e. Protecting one’s Ass(ets).]

Forces: [Regulation, policies and adherence mandates, inability to distinguish between what is essential, what is ‘nice to have’ and what level of detail is required and when. Amnesty for following a good process in the wrong way.]

Resulting Context: [Expensive and bloated architectural models that few understand or use, by-product deliverables, ‘hit and miss’ quality.]

Solution(s): [Use of Pragmatic Frameworks (e.g. PEAF), over-sight of the process by experienced pragmatists, ‘Agile’ mindset.]

This post uses the Anti Pattern template (with some modifications) from c2.com as its structural basis.

Practical advice for managers on how the Web and social media can help them to do their jobs better

[source: Amazon]

I first heard Euan Semple speak about Social Media at a BCS (British Computer Society) ELITE event at BT Tower (in London) back in 2008. What differentiated him from others writing and speaking about the subject?

• Experience: he has a very credible background in collaboration and communications, formerly at the BBC and latterly as an ‘independent consultant’ with blue chips and niche players.
• Hype realism: a recognition of the need to drive real value from social media, delivering business outcomes, not ‘digital noise’.
• Adoption complexity: it takes ‘10 seconds’ to sign up on Twitter, and less again to start using it in an ineffective and potentially damaging way. Forces such as consumerisation and social web have created mind shifts in business. Euan sets out simple, effective, engaging and sensible advice which will inform CxOs, marketers and communications professionals alike.

If you have an interest in the social web and optimisation of communications using social media, this book is a must buy.

Further Info

[source: Amazon]

Today′s managers are faced with an increasing use of the Web and social platforms by their staff, their customers, and their competitors, but most aren′t sure quite what to do about it or how it all relates to them. Organizations Don′t Tweet, People Do provides managers in all sorts of organizations, from governments to multinationals, with practical advice, insight and inspiration on how the Web and social tools can help them to do their jobs better. From strategy to corporate communication, team building to customer relations, this uniquely people–centric guide to social media in the workplace offers managers, at all levels, valuable insights into the networked world as it applies to their challenges as managers, and it outlines practical things they can do to make social media integral to the tone and tenor of their departments or organizational cultures.

• • A long–overdue guide to social media that talks directly to people in the real world in which they work
  • Grounded in the author′s unparalleled experience consulting on social media, it features eye–opening accounts from some of the world′s most successful and powerful organizations
  • Gives managers at all levels and in every type of organization the context and the confidence to make better decisions about the social web and its impact on them

Euan Semple is one of the few people in the world who can turn the complex world of the social web into something we can all understand. And, at the same time, learn how to get the most from it.

Ten years ago, while working in a senior position at the BBC, Euan was one of the first to introduce what have since become known as social media tools into a large, successful organisation. He has subsequently had five years of unparalleled experience working with organisations such as Nokia, The World Bank and NATO.

He is a one-man digital upgrade option for us all to download.

This world is changing fast, but he makes sense of it because he understands that the core basics remain the same: community, learning, and interaction. He is a master story-teller who offers a host of practical tales about how this new world can work for real people in the real world.

Maxims encoding the wisdom of the Scots.

David Russell:

The hardest thing in life is to know which bridge to cross and which to burn.

Robert Louis Stevenson:

To travel hopefully is a better thing than to arrive.

Andrew Carnegie:

People who are unable to motivate themselves must be content with mediocrity, no matter how impressive their other talents.

James Matthew Barrie:

When the first baby laughed for the first time, the laugh broke into a thousand pieces and they all went skipping about, and that was the beginning of fairies.

William Shenstone:

Anger is a great force. If you control it, it can be transmuted into a power which can move the whole world.

David Ogilvy:

Develop your eccentricities while you are young. That way, when you get old, people won’t think you’re going gaga.

Gilbert Highet:

These are not books, lumps of lifeless paper, but minds alive on the shelves.

Robert Burns:

I pick my favourite quotations and store them in my mind as ready armour, offensive or defensive, amid the struggle of this turbulent existence.

The Wicked Problem

Rittel and Webber’s formulation of wicked problems specifies ten characteristics:

10 Characteristics of Wicked Problems

1. There is no definitive formulation of a wicked problem. It’s not possible to write a well-defined statement of the problem, as can be done with an ordinary problem.
2. Wicked problems have no stopping rule. You can tell when you’ve reached a solution with an ordinary problem. With a wicked problem, the search for solutions never stops.
3. Solutions to wicked problems are not true or false, but good or bad. Ordinary problems have solutions that can be objectively evaluated as right or wrong. Choosing a solution to a wicked problem is largely a matter of judgment.
4. There is no immediate and no ultimate test of a solution to a wicked problem. It’s possible to determine right away if a solution to an ordinary problem is working. But solutions to wicked problems generate unexpected consequences over time, making it difficult to measure their effectiveness.
5. Every solution to a wicked problem is a “one-shot” operation; because there is no opportunity to learn by trial and error, every attempt counts significantly. Solutions to ordinary problems can be easily tried and abandoned. With wicked problems, every implemented solution has consequences that cannot be undone.
6. Wicked problems do not have an exhaustively describable set of potential solutions, nor is there a well-described set of permissible operations that may be incorporated into the plan. Ordinary problems come with a limited set of potential solutions, by contrast.
7. Every wicked problem is essentially unique. An ordinary problem belongs to a class of similar problems that are all solved in the same way. A wicked problem is substantially without precedent; experience does not help you address it.
8. Every wicked problem can be considered to be a symptom of another problem. While an ordinary problem is self-contained, a wicked problem is entwined with other problems. However, those problems don’t have one root cause.
9. The existence of a discrepancy representing a wicked problem can be explained in numerous ways. A wicked problem involves many stakeholders, who all will have different ideas about what the problem really is and what its causes are.
10. The planner has no right to be wrong. Problem solvers dealing with a wicked issue are held liable for the consequences of any actions they take, because those actions will have such a large impact and are hard to justify.

Classic examples of wicked problems include economic, environmental, and political issues. A problem whose solution requires a great number of people to change their mindsets and behavior is likely to be a wicked problem. Therefore, many standard examples of wicked problems come from the areas of public planning and policy. These include global climate change, natural hazards, healthcare, the AIDS epidemic, pandemic influenza, international drug trafficking, homeland security, nuclear weapons, and nuclear energy and waste.

In recent years, problems in many areas have been identified as exhibiting elements of wickedness – examples range from aspects of design decision making and knowledge management to business strategy. [Source: Wikipedia]

Implications

1. To address the first characteristic of Wicked Problems, it is necessary to collect a wide range of views of the problem space. An Open Innovation, crowd-sourcing or think-tank based approach (which could be internal ideation, or a mixture of the aforementioned) has promise. In the ‘definition formulation stage’ there will be ‘many’ contradictions, agreement and disagreement between stakeholder groups, terminology problems and nuances. Facilitating and filtering outputs from this phase presents interesting challenges. VPEC-T has a place in this, particularly in dealing with the complexity of eclectic values.
2. The second characteristic of Wicked Problems highlights the importance of solution hypotheses and a means by which to prototype representative solutions and measure their utility. An ability to prototype many solution hypotheses in parallel may be achieved with Open Innovation, particularly challenge driven Open Innovation where a competition model is used. Characteristic 5 implies that prototyping is not viable in the context of Wicked Problems. I think this is somewhat misleading. I agree that once commitment is made to a solution it is a ‘one shot’ operation with consequences, but populating a set of initial solution hypotheses and understanding the scope of the Wicked Problem (particularly where it is ‘interspersed’ with ‘traditional problems’) will help get the right definition of the problem and the right level of focus on its key facets.
3. The third characteristic of Wicked Problems will again benefit from the application of Open Innovation and VPEC-T. Open Innovation in the sense of rapid development of solution hypotheses and a mechanism to source improvement ideas from a wide range of participants (including disruptive thinkers from other markets, industries or geographies). VPEC-T comes into play in the filtering process. Open Innovation is useful in the population of the funnel of candidate solutions. VPEC-T is a useful filter to select preferred options which fit best with the Values and Trust dimensions of the company, government or country attempting to solve the Wicked Problem. In certain Wicked Problems, the Values dimension will need to focus on ethics and cultural acceptability and the Policy dimension on relevant laws and restrictions.
4. Wicked Problems will not be solved through application of design patterns. Characteristics 4 and 7 above rule this out. The generation of unexpected consequences in Characteristic 4 indicates potential for the application of Pattern Based Strategy (in terms of signal detection, and making sense of unanticipated events via correlation and causation analysis).
5. Refer to implication 2.
6. Characteristic 6 reinforces that Open Innovation has potential in terms of sourcing solution hypotheses and enriching these hypotheses with a range of opinion. As outlined in Implication 1, facilitation and filtering is important and VPEC-T has an important role to play.
7. As stated in Implication 4, this rules out the application of design patterns, the solution to a Wicked Problem being unique.
8. Characteristic 8 makes Wicked Problems particularly Wicked. A three pronged attack on this characteristic with Open Innovation, VPEC-T and Pattern Based Strategy has value. Open Innovation in the sense of collectively working on how Wicked Problems are entwined with other problems, VPEC-T in terms of filtering and facilitating analysis, and Pattern Based Strategy in terms of correlation and causation analysis.
9. Characteristic 9 is a real sweet spot for VPEC-T, which excels at surfacing the Values and Trust dimensions of Wicked Problem Solving.
10. VPEC-T (and other thinking frameworks) has an important role in dealing with Characteristic 10. The problem solver(s) eventually need to put their reputations on the line, and must therefore have explored the problem space methodically. Certainty and Wicked Problem solving do not go ‘hand in hand’, and systems and strategic thinking methods are useful in driving out as much uncertainty as may be considered reasonable in highly-complex environments.

Further Reading

“Wicked Problems, Righteous Solutions, A Catalogue of Modern Software Engineering Paradigms” by Peter DeGrace and Leslie Hulet Stahl.

[Source: Amazon]

Figure 1 – Complex and brittle point to point integration

[source: Steve Nimmons]

This is a real ‘bread and butter’ Anti Pattern and has largely been resolved through evolutions of hub and spoke integration architectures (Figure 2), the Enterprise Service Bus, SOA and RESTful architecture. When connections between applications are ‘point to point’, integration is brittle and difficult to modify. Changes to an application have significant downstream impact on applications/systems with which it is integrated.

Figure 2 – Hub and Spoke Integration Architecture

[source: Steve Nimmons]

The Anti Pattern Codified

Anti Pattern Name [Point to Point / Tightly Coupled Integration.]

Type: [Technical / Integration.]

Problem: [Legacy application integration made significant use of the point-to-point Anti Pattern, largely due to the lack of maturity or availability of alternative integration technologies. As depicted in Figure 1, this integration style creates a mesh which is brittle and complex. Applications and systems are also tightly coupled in terms of awareness of integrated applications, and familiarity with integrated application APIs and data models.]

Context: [Legacy integration.]

Forces: [Lack of integration tooling (historical), cost of implementing hub and spoke integration architectures with high-end broker solutions (historical), failure to appreciate the downsides of point-to-point integration (largely historical).]

Resulting Context: [High cost of change, invasive integration, limited or no abstraction, applications responsible for translating data from other applications, no central audit or security functions, difficult systems to manage, monitor, scale, secure and troubleshoot.]

Solution(s): [Hub and spoke patterns, latterly the Enterprise Service Bus pattern (solving the integration mesh problem) and the Canonical Model pattern (solving the application data translation problem).]

This post uses the Anti Pattern template (with some modifications) from c2.com as its structural basis.

Anti Pattern at Work?

It is really interesting how people react to terminology, and ‘Anti Pattern’ is an emotive term, especially in managerial or behavioural contexts. In many organisations the presence of Anti Patterns is significant. Highlighting them without good cause and due care, and without a clear plan for their discontinuation, is an abuse.

Framing is also key, and focus needs to be on replacement of the Anti Pattern with a positive alternative. When I write and talk about Anti Patterns, I highlight the villainous characteristic of the design or behaviour in ‘the abstract’ (parables work well). Anti Pattern analysis is very important, as is the manner in which the conclusions of the analysis are applied and communicated.

Putting up ‘Anti Pattern at Work’ warning signs is not likely to be helpful or conducive to long-term harmonious relations. Anti Pattern Abuse is an Anti Pattern in itself.

The Anti Pattern Codified

Anti Pattern Name: [Anti Pattern Abuse.]

Type: [Management, Technical, Communications.]

Forces: [Have Anti Pattern Catalogue, will use.]

Context: [An over-zealous codifier abuses the use of an Anti Pattern catalogue and points out a slew of negative design, managerial and behavioural characteristics with no consideration of impact and the need for a positive and motivational action plan.]

Resulting Context: [Acrimony, demoralisation, negativity.]

Solutions: [Codify and analyse Anti Patterns. Look for Patterns that eliminate the Anti Pattern(s) and frame any changes in positive terms. Remember the words of Ralph Waldo Emerson: “In my walks, every man I meet is my superior in some way, and in that I learn from him.” Apply the wisdom in your Anti Pattern catalogue with humility and tact.]