This post is part of a series on how the 9 Laws of Data Mining from Tom Khabaza can be applied to analytics. You can find previous posts here.
Law #8: “Value Law” – The value of data mining results [and analytics] is not determined by the accuracy or stability of predictive models
In analytics, we evaluate accuracy by defining how often a model predicts values correctly. Stability refers to the ability of a model to predict correctly on a consistent basis when the data sample is changed within a given population. If a model is stable, then the predictions will not change much when the data sample is changed for the same population. Both accuracy and stability seem essential to the modeling process, so why does the 8th Law tell us that the value of data mining should not be determined by them?
The answer is that the value is not derived from the process, but from the end result. The 8th Law tells us that the value of the data mining process results from 1) the models ability to improve action (more effective business processes) and 2) the models ability to give insight that leads to an improved business strategy (better decisions).
A model that is overly complex may not have as much business value as a model that is less accurate. The reason for this is that the cost associated with gathering the survey data for the calibration of the model could be very high; whereas a simpler model might lead us to the same business conclusion. Khabaza encourages us to ask the question: “Is the model predicting the right thing, and for the right reasons?” This relates back to Law #2 about business knowledge. We can only determine whether a model is predicting the right thing for the right reasons if we know the context of the business problem that we are trying to solve. In addition, we can only determine the value of analytics if we think about value in terms of risk and reward.
The chart above encapsulates the goal of analytics investment in relation to the value (the desired net return) from the investment. 4CGeoWorks created this chart based on a discussion with Bill Huber, Owner/President of Quantitative Decisions. Bill and I both strive to guide companies to make good business decisions that are based on good data and solid analytics. We try to determine where a business falls along the curve and then how to get them to the optimum.
Some companies get lucky. They spend a moderate amount on analytics and they reap huge rewards. It is important to remember that there aren’t many companies in this category. (The probability curve is not in your favor here.) Getting value from analytics typically takes money and a lot of effort. If a company happens to be lucky in their analytics, then they probably don’t need our services (at least not until Law #9-the Law of Change-kicks in).
There are some companies (mostly very large businesses) that understand the value of analytics and they are willing to spend a lot of money to achieve their goals. Those who have figured out how to make analytics work for their business have the enviable position of “Everybody Wins!” Again, the percentage of businesses that fall into this category is likely to be very small.
A survey in 2012 indicated that only about 5% of businesses were using data analytics on a daily basis and considered it to be a core competency despite a majority of businesses using some form of data analytics. Although that study is several years old, there is no indication that these percentages have changed dramatically. A smaller survey in 2015 of 316 executives of large, global companies found that only 8% of data scientists felt that their use of analytics was “best of breed” despite the fact that 90% of the large companies were using data analytics.
Other large businesses may spend a lot of money, but they don’t get a good return on their investment. Most likely, they don’t follow the 9 Laws of Data Mining and ensure that the analytics are based upon business knowledge and objectives. It could be a problem of hiring the wrong people or any number of internal accountability failures that cause the low return on investment. For many of these companies, they don’t even know they have a problem, so it may be difficult for us to help them. If a company does recognize that they are not getting the best results from their analytics, we can assist in optimizing their approach.
Most small to midsize companies either do not invest in or spend very little on analytics because they don’t fully recognize the value that it could bring to their business. By only spending a small amount on analytics, the company reduces the risk of a failed analytics project, but as a result they also are likely to derive fewer benefits which may put them at a competitive disadvantage. They could be missing out on returns that far outweigh their investment which would take their business to the next level. A consultant can provide real value for this company in educating them on the potential benefits from analytics and helping them establish best practices to ensure a good return while keeping investments closer to the optimum point. So, where do you fall on the analytics value chart?
