Unstructured Data and Data Hoarding: don't let the ogre scare you

 http://digital.asiaone.com/digital/features/hoarders-anonymous-unstructured-data

 

This article in the local newspapers piqued my interest since it’s about data hoarding.

Step 1: "Discover users and resources - Determine what is important by rolling up your sleeves and digging through the piles of data" is probably the most critical step in the process. But who can tell what is useful, and especially what will be useful. 

There are 2 major contributors to volume of data: what to keep and for how long. (Embedded in ‘what to keep’ is also ‘in how much detail’ but let’s skip that for now)

For much analysis, a certain volume of data over time is required. You do not want to invest huge resources in a fad, especially if your time to market is not that rapid. Basically, to me, the length of time you need for analysis depends on how fast the environment you are playing in changes. Thus the length of time that we need to keep data is linked to the business you are in.

In terms of what to keep, there is much more debate. For example, how many people would have, a few years ago, decided that the stream of data that is constantly being output by sensors in a manufacturing process were worth keeping? They were consumed immediately and discarded. But nowadays, predictive maintenance is something that is relatively easy to do using precisely the huge volume of accurate sensor data that has been kept. And it is worth keeping this data since doing maintenance before a breakdown, even if it involves taking some components off-line, is much less costly that having to fix a broken machine and the associated impact on the production line.

Another example would be the huge volume of emails that employees engage in over time, especially changes in the patterns of these emails, in terms of frequency, direction, content, sentiment… The classic Enron email analysis is a clear example. How many organisations were analyzing employee email for more than flagging insider trading or breaches of corporate policy ‘now’, as opposed to understanding the patterns and detecting malpractice and collusion? Today this is a component of Human Resource Analytics.

I picked these 2 examples precisely because many people consider these (machine logs, body of emails) to be cases of unstructured data. While these may have been considered too difficult to use in the past, these are routinely used nowadays (hence some people prefer the term ‘semi-structured’). Today the frontier may lie at audio/video files, but I am sure everyone knows of cases where such data can be made very useful (and easily searchable in the case of audio files).

In sum, I would caution against underestimating Step 1. Discovering what is important is not a trivial task. The beauty of discovery is that we are only limited by our imaginations. Hence I would say: hoard as long as you think is relevant to your environment/market, and as much as possible, and don’t let the ogre of ‘unstructured’ scare you.