Contained Chaos and Big Data

Sandeep Sachdeva - Inspiration comes in different forms. Today it came in the form of two resources.  Both are very important in what they bring in concept to Big Data.

As such putting my thoughts in front of you. We keep hearing that Big Data is different in what it brings to the table and as such it needs to be treated differently.  It also needs a different environment, different people and different as in bigger challenges so that you can see Big Data shine in what it does best. The first inspiration is a book - that I am about to introduce to you - that explains some of those concepts in a very articulate way although it is completely unrelated to technology. I have had the opportunity to meet with some of you in person and you know me as an avid book reader. Very rarely have I recommended a book, I am making an exception today. We will save the other inspiration in the form of photographs for later.

First the disclaimer

This is not a review of the book, nor is it an assessment of the book. This is really an opinion on how some of the principles and the thinking applies to Big Data. Why it is important that one should consider these in earnest.

The Chaos Imperative

So let’s get started. If you have readThe Chaos Imperativegreat. If  not, may I suggest that you do and here’s why.

There are many reasons for why Big Data took hold like it did, one of the reasons was due to the fact that someone along the way realized that there is significant value in terms of intelligence and insight in ALL data. Even when the data was just logs or periodic measurements. If I may take this one step further, it may also be due to the fact that the data came into the hands of people who were able to see or imagine the patterns that may result from the data that is not being used.

Now.. you should also take into account the fact that decision makers were fully supportive of the fact that this type of exploration WILL result in gains for the company. In the sense that there was an unusual intersection of the presence of vast amounts of data, allowing the imagination to run wild in terms of what is possible, and having the people who could imagine the possibility of patterns and insights that would turn into actionable items either directly or indirectly.

That in a gist is what the book explains very well but completely unrelated to technology.

Contained Chaos

It brings forward the notion of “Contained Chaos” among other things and how to isolate some of the elements. In here the authors talk about – whitespace, unusual suspects and organized (or planned) serendipity that we will discuss below.

A wise man once said “What you do not say is equally important as what you do say”. Similarly, what you do not do is equally important as what you do. When applied to whitespace, it is the unstructured time that allows the mind to wander freely, to take into account what is possible, what may be possible, what may become possible with what you have and what you don’t have. Whitespace as it applies to Big Data is really the need to allow the Data Scientists to think freely in their choice to take or not take a certain path. I call it the space that allows creation of magic moments within a company, an organization, a business unit, a program, a project or a task.

Unusual suspects as it relates to Big Data is explained really well in my previous post “Why Sherlock Holmes is the ultimate Data Scientist?”. It is that unique inquisitive mind, the endless wanderer, the hungry relentless seeker and the immensely powerful analytical capability which is required for Big Data. Yes I know it is not easy to find such a person, but man has always done well as a team. Look where we are now and how we continue to progress together. Our thought in the form of a satellite has crossed the solar system. So build the team of people that would personify a single Sherlock Holmes. That will be your team of unusual suspects needed for a Big Data program.

Organized Serendipity

Next is Organized serendipity. This is key, as it brings out unusual insights and breakthrough results. The authors ask that you bring in everyone that you can into a single session as ideas and thoughts can come from any point. These ideas and thoughts then challenge existing assumptions and constraints, bring about the contained chaos through unusual discussions. Here every one is equal and free to express what is on their mind. In an organized serendipity session the group can be challenged with a set of questions or allowed to come up with a set of questions that is important to them. Both of these approaches are beneficial and will be radical in what they would bring to the company.

I will leave you here with the rest of the book to enjoy.  If you have not already read my previous posts  “Why Big Data is like a Movie?” and “Are you the Hero in the Big Data Movie?” it may be worth your time. Especially after reading the book.

To the Authors Mr Ori Brafman and Mr Judah Pollack I say – Thank you for a wonderful book.

Visit the BI & IM Symposium

Toghether with OVUM and AutomaterisingsGids, Sogeti Netherlands organizes the 14th edition of the BI & IM Symposium. This two days event at 24 and 25 November, locaiton: 'het Spant!' in Bussum (The Netherlands) is the biggest business intelligence event of the Benelux. Do you like to visit this event, please click on the button "Aanmelden BI Symposium" below and sign up.

Aanmeld knop BI Symposium