Intersection of Data & Human Experience

With the coming of Big Data—and the opportunity it gives for providing nuanced, ever-more personalized analysis, interpretations, and solutions to problems—it can at times seem that all future decisions will only be taken inside the ‘black box’ of the processing unit, hidden from human view and comprehension.

Increasingly we are seeing why this is not best practice. For me, all analytics should start with the human—not, as is often the case, with the data.

Asking the Right Questions

Business analytics should offer a structured, systematic way for leaders and teams to approach business problems. There should be systems that allow us to leverage the underlying data to support the intuition of individuals, and certainly not replace the decision-making capabilities and experience of domain experts who should be guiding the process.

What decisions are we trying to improve? What are we trying to understand about our customers? What data do we need to achieve this? What data do we currently have? Can we gain a better understanding of that data? What additional data would we like to have? Can we innovate around the type of data we might want to collect—in order to be able to answer the questions we really care about?

It is in this symbiosis of creating a team of data experts and domain experts—or finding that very rare individual who has mastered both—that the real value in data analysis is created. For while machines are masters at creating solutions from data, they still lag some way behind humans in identifying the right problems to solve, and so too in the relevant questions we should be asking of that data.

Day-to-Day Data Decisions

It is not only organizations that need to have expertise in evaluating and analyzing data. Data-driven decisions are now so prolific that we all, as individuals, have an interest in better understanding how these processes and systems operate.

Historically, we have always used data and statistics to inform and shape the big, strategic decisions that we make. Data was hard to come by, however, and took time to work through. Today data is collected and parsed in real time, which means we are seeing data shaping tactical, minute-by-minute decisions.

‘Surge pricing’ on Uber is an obvious example, where short-term increases in demand provoked by a rainstorm or the end of a large football match, for instance, result in instant but temporary hikes in journey prices. This approach is also seen in booking hotel rooms and on the likes of Airbnb. Retailers can update pricing decisions frequently, especially in the context of e-commerce, where the price of updating decisions is very low, and we are seeing it trialled in the grocery market too, where electronic price signage allows prices to change during the day depending on demand.

Data-driven analytics are shaping decisions in inventory control, as well. How much stock Amazon carries is crucial to its success, for example—and in assortment decisions, such as how many varieties a retailer should offer.

It is in these more nuanced decisions that an understanding beyond the pure numbers, becomes so valuable. Artificial Intelligence will learn patterns of human behavior over time, given sufficient quantities of data, but an expertise in understanding consumer psychology and biases is also needed to ensure incorrect solutions are not generated and implemented. There is a lot of research that shows customers can be overwhelmed by too much choice, for example.

We have to take into account these types of human biases when we are trying to design good business analytics, and this is where a holistic approach is key to understanding these systems and not looking at them as black boxes. Business leaders must challenge and understand the principles underlying the mechanisms that lead to particular outcomes, within these systems and processes.

Expert Partnerships

There is a common misconception that recruiting an expert in data analytics or data science is going to lead to quick solutions to every problem you have. Very often the new talent with data expertise does not have the domain expertise required to fully engage with and understand these problems to begin with. Not, at any rate, at as broad and deep a level as you would need in order for them to bring about optimal and innovative solutions.

To unlock the real value of these new data analytics hires, they should be partnered with your domain experts—people with a very sharp intuition about the right questions to ask and the issues your team or organization ought really to be caring about.

From an organizational perspective then, try pairing these individuals together in teams—data science expert with domain expert. One operating without the other can in fact be badly limiting. It would be extremely difficult for a data science expert, for example, to be as creative and imaginative—in identifying what new or currently untapped data might be out there—as the domain expert who knows the industry inside out. There is a symbiosis that occurs when you bring these two skill-sets together.

So the call to arms is, yes—bring in data analysis expertise, but make sure you marry it with real sector expertise too. People who understand the domain dynamics can ask the right questions of the data—and help shape and run the correct experiments to improve decisions.