Analytics: Getting the basics right – an example of the importance of proper data preparation

I recently saw an infographic that seemed to show that travelling by Singapore Airlines was, on average, cheaper than travelling by Malaysian Airlines. Actually according to the infographic, the per km cost of Singapore Airlines is lower than that of Malaysia Airlines. (and by almost 50%) The full article can be found here: A deep data-driven drive into airline prices:

25 most popular airlines on Rome2rio and the median price per kilometre for their fares 

I was puzzled and decided to look a little bit deeper into the issue.

To me, unless doing a pure discovery piece, an analyst/’data scientist’ should start by asking him/herself “what am I trying to figure out?” Then after the data supports an answer, the question should be “how can I best convey this information graphically?” Ideally, the infographic should be self-explanatory, a picture is worth a thousand words. And to remain in the realm of clichés, I’m saying that a piece of non-discovery analytics should ‘begin with the end in mind’, the end being the answer, not the infographic.

Let’s say I want to find out what airlines are the cheapest. What metric should I use?

The median cost per kilometre as used in the article is a great one. Using the median is a quick way of getting rid of outliers caused by last minute bookings for example. Looking a bit deeper into the article, the authors mention using only economy fares and removing outliers by only including prices that are less than twice the minimum fare. Using the distance as a denominator is a nice way of making the airlines comparable, you know, the apples and oranges thing...

Now, let’s take a look at the data we have on hand. The analysis was done using millions of bookings on Rome2rio.com. Rome2rio is a website that allows door-to-door booking, using not only airplanes, but other means of transport such as trains, cars... 

So the first question is: can the rome2rio data be used as is to represent the world?

The first thing that looks odd is the “most popular” airlines as per Rome2rio. Looking at the IATA figures:

There is a huge difference between the IATA list of airlines and Rome2rio’s. Even when we consider only Asian airlines:

It looks like that data that Rome2rio has is very different from worldwide or Asian-wide numbers. The dataset therefore does not allow conclusions to be made on for air travel in general, but only on the specific segment of travel that Rome2rio customers are part of. And there is nothing wrong with that.

What I am talking about is a form of self-selection. People who are looking for door-to-door type travel and to/from ‘exotic’ geographical areas (Kenya, Russia, Jordan, Iceland...) are different from others, say like me who just wants a flight and I will settle the rest on site. 

The title of the infographic does state that it’s based on Rome2rio data, just that it would be nicer to state the difference rather than, in the article. stress the millions of data points which tend to give an impression of universal use. Basically the dataset is biased. My previous post talks about biased training data, and how we can attempt to remedy this. I won’t go there in this one.

So now I realise that given the dataset I have, I can only make conclusions for Rome2rio travelers. I have removed all the oranges of non-Rome2rio type travelers. 

The infographic is re-interpreted as saying that, for Rome2rio-type travelers, Singapore Airlines is cheaper than Malaysian Airlines. Still it’s puzzling.

A possible solution to the puzzle can be found in the article itself. The authors focus on a selected group, flights within different geographies: Australia, and also on airlines serving local routes in US, Turkey, Russia, Indonesia, Thailand, India. And this makes more sense in many ways.

Basically you need to ensure you are comparing like with like; even among apples, there are green apples and red apples and you wouldn’t think of them as similar, would you?

 

Hence breaking down the analysis regionally makes sense from a usability point of view; after all, if I am travelling in Asia, I wouldn’t care if American Airlines is cheaper than Singapore Airlines, American Airlines is irrelevant to my route. But is that enough from an analysis point of view?

To be able to compare stuff meaningfully, they should be as similar as possible. In terms of airline ticket prices, apart from the class of travel that has already been accounted for, there are other important considerations to the cost. 

One of the most important is the number of stops, especially given the way airline hubs are usually in their home countries. Aircraft fuel consumption peaks at take-off, so it wouldn’t make sense to compare Singapore to London via Singapore Airlines and the same route via Malaysian Airlines (although still on that route Malaysian Airlines seems cheaper). 

Another consideration would be the distance traveled. Airlines usually pick planes for the routes they travel so as to take into account the costs of fuel and the potential load they are able to carry. It’s a balancing act, on one hand the more fuel you carry the further you can fly, but on the other, the heavier you are and the more fuel you burn (especially at takeoff when the load is at the maximum). Hence there is a tipping point beyond which it is better to take have a stop-over and refuelling and that tipping point varies depending on the type of aircraft.

Airline ticket prices are a function of not just the cost but also the level of competition on routes. For routes to/from remote areas, the prices are likely to be much higher simply because of lack of competition; it does not make sense to look at oligopolistic markets. These should be excluded from the analysis as they are outliers that can unnecessarily skew the analysis (since we are looking for a price in general).

Similarly the timing of the flights also has to be taken into account. The flights should be in the same time period or in a time period that accounts for seasonality and weather conditions.

The basic idea is that we should ensure we are comparing red apples with red apples, and green apples with green apples.

Basically there should be a lot of thinking that takes place before data is analysed. The data preparation stage should not always follow a purely mechanical approach. To me an analyst/’data scientist’ should get involved in data preparation because the way you prepare the data can have a huge impact on the outcome of the analysis, and as you work with the data, you sometimes want to go back to the way the data was prepared and change things, hence I prefer to get my hands on raw data and prepare it as the situation requires.

Human Resource Analytics: is machine learning the answer to removing biases from the hiring process?

Human Resource Analytics is one of the more exciting applications of ‘analytics’/’Big Data’/’Data Science’. A very straight forward application is to predict who is likely to leave within a certain time frame, and also, to some degree, what can be done to retain the employee if the organisation wishes to do so. (Not all employees who want to leave are worth keeping). I’ve built this type of model and they work quite well, and I only used the employee’s behaviour at work.

This can be even more interesting in the context of “Big Data”, with the increasing use of social networking and online presence, most of us leave traces for more behavioural clues to be harvested. Recently there has been a trend to use machine learning more in Human Resource, especially in terms of hiring. It is sold as finally unbiased hiring. But how accurate is that? 

Quite a few start-ups are promising unbiased hiring, getting you the best candidate by using proprietary algorithms, very often machine learning algorithms.

Basically, in machine learning, the machine is fed a whole bunch of data, in the context of HR this is likely to be CVs of applicants, any test answers, interview notes, and in the case of supervised algorithms, whether the person was hired or not (and may be the subsequent performance of the people hired). The machine then trawls through the data, learns, and when fed a new bunch of candidates will pick a list of top candidates based on what it has learnt.

My first reaction to this is that, if the data you feed into the algorithm is the data collected by your own organisation within the hiring period, then the machine is likely to replicate what the organisation has been doing. Let’s say that most of your hiring managers come from a specific school, they might be biased in favour of that school, and hire people accordingly. The machine will, in an unbiased way, learn this behaviour and apply it. Training a machine on biased data, is likely to lead to the machine churning out biased outcomes.

Of course there are a few ways round this. 

One way is not to limit the training dataset to a single organisation, but to take a broader view of the market, basically trying to remove or at least dilute the bias. May be your organisation is populated by graduates from school A, your competitor by those from School B, so if you train your algorithm on the combined dataset, you are likely to be reducing the bias. This is the idea behind credit bureau scores, and is likely what quite a few of the start-ups in the HR space are trying to do. 

In this case, the organisation-specific biases are likely to be mitigated, as long as the training dataset is diverse enough. However, unless you have the universe, you still run the risk of being biased. Ideally you should weigh your training set; it’s like stratified sampling in a way, but how will you know how to stratify? You’d need to know the biases.

Another approach is to repair the training dataset, for example: http://www.datasociety.net/output/hiring-by-algorithm/ Basically, the output of the algorithm has to be analysed, and any biases uncovered before implementation, and the training dataset modified to remove that bias. It is not as easy as it seems. For example, imagine if the true reason why people are rejected from the hiring process is not directly apparent in the data, let's say due to criminal record. If some section of society, say people with blue hair are over represented in the population of people with criminal record, then it is possible that the algorithm will be less likely to recommend people with blue hair for hire based on the fact that a smaller proportion of them have been hired in the past, not recognising the real reason. Basically blue hair becomes sort of a proxy for criminal record. 

To go a level deeper, even an algorithm trained on the whole population is unlikely to be free of societal biases https://www.theguardian.com/technology/2016/aug/03/algorithm-racist-human-employers-work . For example if women have not been employed in C-level positions, it is likely that an algorithm would pick that up and not recommend women for these positions, or show people online whom they know to be women, lower paying jobs: http://www.andrew.cmu.edu/user/danupam/dtd-pets15.pdf  . 

To make matters worse, the algorithms often are black boxes. Unless the results of the algorithm are analysed and an attempt is made to find the possible drivers by supervised techniques, it would be almost impossible to uncover the actual drivers. (There are vendors who do that, but clients need to bear in mind that these are not necessarily the real drivers, and to me, it kind of defeats the purpose) 

As an example, imagine a story about a person committing crimes, getting caught by the police, and getting sent to jail is told to a classroom. Everyone agrees that the outcome (ending up in jail) is undesirable. However, what was learnt from the story: do not commit crimes, or do not get caught? A teacher can find out by simply asking the students, but a black box is unlikely to answer.

This is what is called algorithmic transparency, one of the solutions that the ford foundation posits to solving the biases in algorithms. https://www.fordfoundation.org/ideas/equals-change-blog/posts/can-computers-be-racist-big-data-inequality-and-discrimination/ but by definition, black boxes are not transparent.

For those of you who are interested in reading more about the topic I’d recommend the following paper by Barocas and Selbst: Big Data’s disparate impact http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2477899 . For an overview of a method to uncover the hidden biases and try to remove them from the training dataset bearing in mind the trade-offs http://arxiv.org/pdf/1412.3756v3.pdf. Basically it takes a lot of effort and if the actual predictors used are not visible, even more so.

In summary, I’d say that achieving total impartiality in a Human Resource function such as hiring by using machine learning is not as easy as it seems. Humans are generally biased, even societies have biases, and this bias often taints the machines too, especially via the data the algorithms are trained on. And if the algorithms are black-boxes, it makes it that much harder to remove the biases.