Experimentation and Design of Experiments
Experimentation
is a ‘new’ buzzword, some even claiming “Big Data” demands experimentation.
I
agree, and disagree.
The
need for experimentation has always been here – why would you launch a full scale
campaign costing hundreds of thousands when you can spend some time refining
and enhancing your offer and targeting?
It’s
just that, with “Big Data”, data is available at a more granular level and
hence we are able to detect more subtle events and sequences of events that can
be seen as precursors to various courses of action. Today, organisations can
quite easily know where their customers are, and allied with their recent
behaviour, can create tailor-made offers. If I just had lunch at a fried chicken
restaurant, and my bank has an offer at a yogurt bar nearby, it might be the
exact thing to balance my tummy. Hence the bank could send me an offer for the
yogurt bar.
Technology
has helped in 2 further ways. Firstly, not only is “Big Data” available for
analysis at a very granular level, but we are able to do so without having a
Computer Science Degree. Hence the number of potential experiments that can be
attempted has increased. Secondly, technology has also increased the ability to
execute these experiments; apps, text messages are all avenues where we can be
reached.
But
what, to me, is critical is the design of the experiments. Just like with “Big
Data” analytics, while technology has put the ability to analyse data in the
hands of many, there are fundamentals that put probability on your side. Proper
experimental design is critical and the costs of not doing so can be
catastrophic.
The
articles linked below highlight some failures in the medical field that have
huge implications and that could be avoided simply with proper experimental
design. The sad part is that the failures are so basic that a little thought
would have avoided them:
Ensure large enough a sample to
conduct experiments (power tests)
Do not bias the experiment by
selecting groups to affect success probabilities
(random selection)
Do not bias the experiment by
your own behaviour (double blind tests)
By
all means experiment, but please do so intelligently.
