Figure 1: A (simplified) Big Data Ecosystem

[source: Steve Nimmons]

In terms of ‘forces’ affecting the CIO Agenda, Information Strategy and Enterprise Architecture, Big Data is increasingly important. This is due to explosive growth in number of data source types: applications, digital media, mobiles, users, customers, unstructured data sets, sensors, emails, blogs etc. Data is complex and in mixed formats (text, video, audio), on-demand infrastructure scalability (including massively scalable storage) is needed to deliver Big Data capabilities, as are robust analytics and visualisation tools and techniques for distributed, parallel systems. Increasing bandwidth availability has also led to exponential data growth rates and capabilities e.g. social networks, video and microblogging.

Where do you start in formulating a reference architecture for Big Data and sourcing suppliers for a Big Data ecosystem?

Should you believe the Hype?

The Gartner Hype Cycle places Big Data on ‘the upslope’ towards the ‘peak of inflated expectations’. Big Data is of course already underpinning many of the web giant’s architectures (typically because necessity has been the mother of invention).

Figure 2: Gartner Hype Cycle for Emerging Tech (2011)

[Source: Gartner]

• Facebook uses Hadoop to store copies of internal log and dimension data sources and as a source for reporting/analytics and machine learning. There are two clusters, a 1100-machine cluster with 8800 cores and about 12 PB raw storage and a a 300-machine cluster with 2400 cores and about 3 PB raw storage.
• Yahoo! deploys more than 100,000 CPUs in > 40,000 computers running Hadoop. The biggest cluster has 4500 nodes (2*4cpu boxes w 4*1TB disk & 16GB RAM). This is used to support research for Ad Systems and Web Search and to do scaling tests to support development of Hadoop on larger clusters
• eBay uses a 532 nodes cluster (8 * 532 cores, 5.3PB), Java MapReduce, Pig, Hive and HBase
• Twitter uses Hadoop to store and process tweets, log files, and other data generated across Twitter. They use Cloudera’s CDH2 distribution of Hadoop. They use both Scala and Java to access Hadoop’s MapReduce APIs as well as Pig, Avro, Hive, and Cassandra.

Other Hadoop users include:  1&1, A9.com, About.com, Amazon.com, American Airlines, AOL, Apple, Booz Allen Hamilton, Cerner, ChaCha, comScore, EHarmony, Federal Reserve Board of Governors, foursquare, Fox Interactive Media, Freebase, Hewlett-Packard, IBM, InMobi, ImageShack, ISI, Joost, Last.fm, LinkedIn, Microsoft, Meebo, Mendeley, Metaweb, Netflix, The New York Times, Ning, Outbrain, Playdom (now part of Disney Interactive Media Group), Powerset (now part of Microsoft), Rackspace, Razorfish, StumbleUpon and Twitter.

Hadoop Overview

Figure 3: Hadoop Overview

[source: Steve Nimmons]

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using a simple programming model. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.

Hadoop has Commons, MapReduce and Distributed File System capabilities (HDFS) as well as sub-projects: HBase, Cassandra, Avro, Hive, Mahout, Pig, ZooKeeper and Chukwa.

Given the pervasive nature of Hadoop, this is a strong contender for any Big Data implementation. HBase is the Hadoop database. Cassandra is also a NoSQL database. Mahout is a data mining and machine learning component, Hive and Pig are querying components, Zookeeper a coordination component.

Hadoop Distributions, such as that from Cloudera, bundle Apache Hadoop with other Open Source tools to create a more feature rich ‘platform’. The Cloudera distribution is definitely one to evaluate.

A simple Reference Model

In terms of implementing ‘Big Data’ architectures there are a number of choices, particularly in the visualisation and analytics space (refer to Figure 1). A simplified reference model is provided in Table 1. This will be expanded in a series of future posts on architectures for Big Data, exploring key features and design trade-offs.

Table 1: Simplified Big Data Reference Model

[source: Steve Nimmons]

Data Loaders (e.g. Sqoop) and log management (e.g. Flume, Scribe) could also be included in the reference model / ecosystem.

Further Reading and Interesting Tools

• Processing: http://processing.org. advanced visualizations
• Protovis: http://vis.stanford.edu/protovis/ 
• Gephi: http://gephi.org/ focused on Social Network Analysis
• Tableau: http://www.tableausoftware.com/public/
• ManyEyes: http://www-958.ibm.com/software/data/cognos/manyeyes/, from IBM.

Practical advice for managers on how the Web and social media can help them to do their jobs better

[source: Amazon]

I first heard Euan Semple speak about Social Media at a BCS (British Computer Society) ELITE event at BT Tower (in London) back in 2008. What differentiated him from others writing and speaking about the subject?

• Experience: he has a very credible background in collaboration and communications, formerly at the BBC and latterly as an ‘independent consultant’ with blue chips and niche players.
• Hype realism: a recognition of the need to drive real value from social media, delivering business outcomes, not ‘digital noise’.
• Adoption complexity: it takes ‘10 seconds’ to sign up on Twitter, and less again to start using it in an ineffective and potentially damaging way. Forces such as consumerisation and social web have created mind shifts in business. Euan sets out simple, effective, engaging and sensible advice which will inform CxOs, marketers and communications professionals alike.

If you have an interest in the social web and optimisation of communications using social media, this book is a must buy.

Further Info

[source: Amazon]

Today′s managers are faced with an increasing use of the Web and social platforms by their staff, their customers, and their competitors, but most aren′t sure quite what to do about it or how it all relates to them. Organizations Don′t Tweet, People Do provides managers in all sorts of organizations, from governments to multinationals, with practical advice, insight and inspiration on how the Web and social tools can help them to do their jobs better. From strategy to corporate communication, team building to customer relations, this uniquely people–centric guide to social media in the workplace offers managers, at all levels, valuable insights into the networked world as it applies to their challenges as managers, and it outlines practical things they can do to make social media integral to the tone and tenor of their departments or organizational cultures.

• • A long–overdue guide to social media that talks directly to people in the real world in which they work
  • Grounded in the author′s unparalleled experience consulting on social media, it features eye–opening accounts from some of the world′s most successful and powerful organizations
  • Gives managers at all levels and in every type of organization the context and the confidence to make better decisions about the social web and its impact on them

Euan Semple is one of the few people in the world who can turn the complex world of the social web into something we can all understand. And, at the same time, learn how to get the most from it.

Ten years ago, while working in a senior position at the BBC, Euan was one of the first to introduce what have since become known as social media tools into a large, successful organisation. He has subsequently had five years of unparalleled experience working with organisations such as Nokia, The World Bank and NATO.

He is a one-man digital upgrade option for us all to download.

This world is changing fast, but he makes sense of it because he understands that the core basics remain the same: community, learning, and interaction. He is a master story-teller who offers a host of practical tales about how this new world can work for real people in the real world.

Figure 1 – The Johari Window devised by Joseph Luft and Harry Ingham

The Johari Window is a model for describing personal awareness types and human interaction.

Quadrant A: encapsulates personal awareness and a wish to share information with others, for the purposes of simplicity assume this means publicly.

Quadrant B: encapsulates personal awareness of a different type. The motivations for concealment are plentiful (bad habits, competitive advantage, Machiavellianism, protection of personal interests etc.). The size of this box tends to diminish as trust relationships expand, however I contend: a) there are many types and levels of concealment implied here and b) many different levels of trust in different social circles.

Quadrant C: encapsulates weak personal awareness and misinterpretation (we assume others see us as we see ourselves, but this is not the case). This quadrant (in the context of Social Networking) provides an interesting opportunity for introspection and awareness development from social feedback, Social Network Analysis and sentiment analysis. This is a box full of brambles!

Quadrant D: Donald Rumsfeld’s infamous Known Knowns speech of 2002 sums up this quadrant.

A Prophetic View

Just under two years ago I wrote a somewhat prophetic article concerning Privacy and Social Networks in which I argued for the need for additional privacy controls and multiple walled gardens within social networks. Facebook lists were a crude approximation, but Goolge+ Circles now excel at delivering the concept. A sister post in February 2010 discussed Social Search and the Integrity of the Social Graph, concluding that Google was heading (with purpose) into the Social Networking space.

What I said back in January 2010:

Visualisation of Social Network privacy controls is poor. The granularity of access controls is too coarse. My solution would be creation of (either my privacy “Onion model”) or perhaps more simply a ‘radar’ or quadrant model on which connections could be placed within ‘trust zones’ (by dragging and dropping them onto the appropriate region). Configuration is half the battle, and visualisation of the resultant privacy controls effect is essential. This is where current controls are weakest. I also want multiple walled gardens to play with (where I could segregate user groups) and ensure no (uncontrolled) information leakage between…

A trust and privacy ‘radar’ would be equally interesting, with those closest to the centre having the greater trust relationship and access to more personal data.

The Johari Window and Google+ Circles

Figure 2 – The Google+ Circle Model

I have a number of Circles within Google+: Friends, Family, Acquaintances, Scientific Community, Social Media, Politics, Techies etc. There is also a ‘Public category’ which maps neatly onto Quadrant A of the Johari Window.

Quadrant B maps neatly to the different circles (Friends, Family etc.). This creates controlled separation, where I can isolate various topic discussions. This helps prevent Family members from being bored by discussions about Social Network Analysis or Social Psychology! Equally it saves Scientific Community colleagues reading my latest views on the European Union. There is a great deal more depth to this than simple ‘separation of interests.’ Despite what we may think, as multi-dimensional beings, we do not necessarily want everyone in cyberspace or our social sphere having a complete 360 degree view of our personality, interests or social connections.

Quadrant C could make for a ‘fun’ social network game – tell me something about myself that I don’t know, but you do know. Play at your own risk!

Quadrant D is ripe for Reality Mining as long as there is a digital footprint.

The Johari Window provides an interesting thinking framework on which to base an approach to online privacy protection and information sharing across social groups.

Extending the Johari Window for Privacy and Reputation Protection

I propose an extension to the Johari Window (as depicted in Figure 3). As information flows into a Circle we lose control of it. We must assume that we have chosen Circle members well and that each member will understand (and abide) by our privacy wishes in respect of that information. The obvious drawback however is that there is no adequate meta-data associated with the shared information to indicate to Circle members what is ‘allowable’. Perhaps Google will introduce ‘Circle Contracts’ to stipulate between parties what is acceptable!

Adding an A+ box (Figure 3) recognises that there will be information which I am happy to be disclosed by people acting as relays between Circles with no restrictions.

Box B+ recognises information disclosed to certain Circles must stay within that Circle or may be selectively disclosed to other Circles (not under my ownership) which meet certain membership/privacy criteria. There is currently however no way to express this (or manage disclosure across ‘logically chained Walled Gardens’).

Box C+ recognises that there is information about myself of which I am unaware, and would be happy about being disclosed. If it is information which may be publicly disclosed, it fits within box A. If it requires restriction per ‘Walled Garden’ or Circle, it fits within box B.

Box C++ recognises that there is information about myself of which am I unaware, and would be unhappy about being disclosed. This box is ripe for Reputation Protection.

Boxes C+ and C++ are interesting as I would be theoretically unaware of my privacy requirements until the information is disclosed (of course heuristics could be employed).

Boxes B, B+, C, C+ and C++ all have potential for information leakage. As Circles and Networks are highly interconnected, chances are the information could reach parties which you would rather not see it.

Extending the Johari Window and applying this thinking technique to online privacy within Social Networks is useful in terms of surfacing complexity and also challenging personal views of requirements for information management.

Figure 3 – Extending the Johari Window

[source: Steve Nimmons]

Article originally published by Evaluation Centre / Conspectus, Summer 2008

Steve Nimmons warns of the hidden threat to corporate privacy and reputation lurking within Web 2.0.

The Historical Problem

I recall (approximately eight years ago) reading an interesting poster on social engineering at a well-known electronics company in California. This wall-chart communicated sensible advice for dealing with unsolicited phone calls, ‘chance’ conversations and the importance of discretion when discussing corporate matters on planes, trains and automobiles.
Topics such as tail gating, the ‘risk of gallantry’, the social and psychological tricks used by experienced practitioners to project ‘belonging’, the need for discretion and vigilance in public spaces and of course ‘clear desk policies’ were explained in concise, relevant and accessible language.

In this way, workforces across this and other enterprises were equipped to deal with the primary aspects of corporate social manipulation. Using in-house and industry standards, they shared the wisdom of primary threats, expected behaviours and above all encouraged staff training and awareness.

I visited many technology start-ups during this time. Their social engineering concerns centred mainly on leakage of financial data and intellectual property. With looming IPOs (initial public offerings) these companies had a lot to lose; the wrong information entering the market at the wrong time could potentially damage earnings.

Intellectual property was naturally their core competitive differentiator and was suitably protected, including legally through patents and non-disclosure agreements. It was clear what they feared, why they feared it and that they were being proactive in terms of minimising their overall exposure to risk. Their perimeter defences, with clear corporate boundaries and technological barriers, tamed Web 1.0.

The Problem Develops

Fast-forward eight years to the introduction and exponential uptake of Web 2.0 and it is fascinating (indeed crucial) to explore the need for similar protection and advice today. The Web 2.0 revolution essentially involves the removal of technological barriers to content publication. Blogs, wikis, forums, social bookmarking and social networks are just some of the means by which individuals can share and debate views (single click, no safety catch).

As we have discovered (or perhaps suffered) in the past few years, the web provides ideal conditions for libel, defamation (perhaps creating internal conflict or damaging partner relationships), careless divulgence of information and the association of the individual and corporations with unflattering and potentially damaging material.

These are arguably Web 2.0 ’s most concerning corporate side-effects. Worryingly, the individual is the power-broker of Web 2.0 and with microblogging (particularly Twitter) tipped for meteoric success, we will see even less control exercised over what are essentially globally distributed sound-bytes.

Pseudonyms provide anonymity, personally or corporately identifiable profiles ‘should’ engender a greater spirit of caution and present an opportunity for positive self and corporate marketing (for example, through blogging and thought leadership initiatives). But what needs to be understood clearly is that the search engines with their omnipresence discover our sins. In print media, yesterday’s news wrapped today’s fish and chips. But in the electronic age, opinion has an almost immortal quality. Search engines have a unique ability to discover and neatly present information that we may prefer remained hidden. Meanwhile, the Web and blogosphere contain a cacophony of voices inside which they are the eavesdroppers and intelligence agents.

There is an adage that Web 2.0 profiles are like tattoos – something you do when you are young and live to regret. But with appropriate controls, education and consideration, companies can seek to accentuate the positives and in sophisticated cases utilise them in personal branding and corporate marketing strategies.

Where once scraper and ‘shill’ sites were padded with ‘pointless’ copies of the Open Directory Project (an old trick to create thousands of pages to bloat a website that was then packed with affiliate programs and click-through advertising), they are now extracting content from RSS feeds, quite a number scraping via Technorati tags that simply mirror their underlying site’s (content) taxonomy. I use Technorati tags to categorise content for improved searching and user experience. I am often amused to see how my articles are ‘aggregated’ onto these sites totally against copyright and any sense of appropriate ownership and control. In some cases the use of such content may be beneficial (eg, offsite advertising), but consider the potential for widespread distribution of commentary.

Keep in mind the traditional political and broadcasting advice to ‘treat every microphone as if it were live’. Something said is difficult to retract in Web 2.0 ’s publishing model. This could affect your personal reputation, privacy, cause corporate embarrassment or perhaps worse. Social engineers are astute, so be careful of being drawn into electronic conversations that should be avoided. Solutions to some of these issues are emerging – including online reputation protection services such as Reputation Defender, ClaimID and Naymz – suggesting both the commercial and personal need to clean up ‘digital litter’.

Digital litter is all those nuggets of information personally linked to you – and be under no illusion that this body of
information is being pored over by fraudsters and marketing companies, and in the corporate realm by researchers and
competitors. Information, of course, is not as volatile as might be imagined. Simply deleting it from the original source is no guarantee of its destruction, with scraper sites, search indexes and historical web caches adding to the complexity. Reputation protection may only dilute some of the problems rather than completely remove them.
We must of course accept freedom of speech and the right of fair criticism. In the Web 2.0 domain, our ‘complaints’ may well be beyond any reasonable bounds of control.

Corporate reputation is also tightly coupled with customer satisfaction, shareholder value, innovation and similar attributes. A key addition to the advice from 2000 is therefore to minimise personal and corporate risk from a system of worldwide electronic publishing where everyone can act as content producers.

Corporate boundaries

As well as shifting the content producer-to-consumer ratio, Web 2.0 has removed some traditional corporate boundaries. In Unified Communications we talk about edgeless enterprises. Web 2.0 warrants a special mention as it has ‘eroded the edge’ through its technological simplicity, whilst also prompting a radical reappraisal of the psychology of home and work.LinkedIn, Facebook, Plaxo, MySpace and associated groups today provide a rich hunting ground for the social engineer. Companies can be significantly profiled. Their names, departments, reporting structures, nature of business, personal links and networks can be mined and prioritised for further attack.

It is relatively easy to comb sites for information to use in ‘impersonation attacks’, extracting additional detail through email, telephony and other channels. With no identity management (ie, no established trust) it is simple to create fake pages, groups and details and use these to link the unwitting.

I am opposed to companies blanket banning access to social networks. Bans of this nature have one glaring weakness – they end when employees go ‘off the clock’. They also restrict the business benefit that could be derived from appropriate use. It’s vital to understand your risk exposure and develop appropriate security policies, best practices and employee education.

Parental education is a recurring theme in the recent Byron Review (established in 2007 to study the online safety of children) and I draw parallels with employee and employer education. Threats are ‘evolutionary’ and social engineering is enjoying an upsurge in the volume and quality of unsolicited, freely and legally attainable information. Reputation protection faces new challenges due to the speed of content production and distribution, a mechanism of such
simplicity and attractiveness that bewildering numbers have embraced it across previously untouched demographics.
There is a strong case for placing the onus on site providers to better protect privacy, but personal accountability must be advocated above all.

Key Points

So the key points that go on my updated wall-chart for 2008 are:

• Explain risk exposure in terms of information leakage, and libellous, defamatory or brand-damaging activities through the employee base. Public comments from identified staff are potentially detrimental to business reputation and relationships.
• Understand and (in many cases) limit the volume of available corporate data on personnel, roles, responsibilities and professional activities – the social engineering goldmine.
• Marketing and IT security play an expanding role in meeting new threats and opportunities.
• Introduce Web 2.0 savvy security policies and training plans. It is no understatement that the proliferation of Web 2.0 opens a sizeable number of holes in the sieve of corporate intelligence (take the recent Facebook security leaks and social worms like Secret Crush as examples). Educated staff can make informed decisions and can better manage their own digital footprint as well as that of their employer. It vital is therefore for modern security training to cover the fundamental dangers of Web 2.0.
• Companies should understand the mechanics of auditing, measuring and defending their online reputation. Web intelligence solutions are particularly useful but the ability to manage remedial action is still fairly undeveloped.
• With appropriate selection, guidance, motivation and controls, there is an opportunity to use the publishing power of Web 2.0 for extremely positive personal and corporate gain.

It is important not to be overtaken or overrun by technological advances. I recently advised a company following its discovery of some unofficial social networking groups (bedecked with company name and logo). These groups were innovative and well intentioned (if naïvely established) and such discoveries indicated corporate IT were losing touch with talented, motivated and active networkers.

Simple editorial control and content audits were set up. It is however important to reflect on the potential for damage as well as the potential for gain if the same enthusiasm were harnessed through focused and ‘moderated’ corporate initiatives.

Summary

Information leakage has reached the point where in April 2008 the Israeli Defence Force (IDF) was compelled to issue a
statement warning that “Facebook was a threat to national security”. At the heart of this was the ‘free and easy’ manner in which members of the IDF were posting personal information, identifying themselves as members of the security services, pictured at sensitive installations and discussing sensitive subjects.

In other words, the problems we face are so potentially damaging that they are now ‘on the radar’ of government security services. Online advertising models deserve an article in their own right, but I would briefly mention privacy concerns over Phorm and the highly publicised ‘Beacon disaster’ championed by an ‘unwisely zealous’ Facebook. These add an additional twist to the complex world of Web 2.0 security.

The best way to respond to these threats is to shape, cultivate, educate and empower your employees. Develop an
understanding of your risk exposure and Web 2.0 ’s threats. The blinkers of a ‘9 to 5’ blackout may be unworthy – but be warned, you lose control of your employees, your personal or corporate reputation in Web 2.0 at your peril.