Dealing with Big Data: From Quants to Smarts

For those tracking the world of Big Data and Analytics, you’ve probably heard of “Quants.” Just to be sure, a quant is the epithet of a person who is an expert in analytics and “quant”-itative analysis – thus the moniker “quant.” Quants are the people who businesses rely upon to illuminate the gnarliest analytical, mathematical, and numerical problems.

Quants play a crucial role in many industries and functional areas from health care and manufacturing to banking and retail supporting supply chain, finance, marketing, and more. As long as there are streams of data to understand, quants are here to stay. As the streams of flow into larger data lakes, especially with the Internet of Things (IoT) promising to generate gazillions of bytes more of data, quants have unparalleled job security. My former CEO at AMD, Rory Read, used to say that management’s role is to “torture data until the truth surfaces.” This role is relegated to data savants, a.k.a. The Quants!

“Torture data until the truth surfaces!”  Rory Read

One important area of focus for a company’s Business Intelligence (BI) activity is to identify the areas where more quantitative data analysis is needed. Many companies find themselves using high-performing employees mired in the laborious and unproductive practice of aggregating, rationalizing, cleaning, and reporting on data. BI and analytics programs drive to reduce the time needed to get information while allowing additional time for employees to dig deeper for fundamental, root-cause analysis capability, develop data models, simulate scenarios, and provide decision support.

Once the business achieves data cleanliness and stability, the focus changes from reporting and informing about past performance to anticipating and predicting future potential outcomes. Driving the business by looking exclusively through the rear-view mirror does not allow companies to avoid dangers lurking ahead. Once the ability to anticipate is in place, business discussions move from “what happened” to “what may happen” – scenario planning and modeling.

The migration to a forward-looking analytics model compels organizations to ask, “Are we ready for the ramifications that these BI development programs have on individual skills, management expectations, organization structure, and communications across the enterprise?” As the company evolves toward more anticipation-oriented and proactive decision-making based on data, the skills and disciplines required on the teams change. The questions management should be asking change, too. How will this actually work?

Teams must frame the decision scope and ensure clarity of the business problem(s) or opportunity(ies) being addressed. The quantitative analysis that follows will include reviewing historical data, establishing a hypothesis, gathering data, and performing the analysis. At the back-end of the process, decision makers must evaluate the results from the story that the data analysis conveys.

Trust is vitally important in both the data and the individual/teams doing the quantitative work. Verification never ceases. Critical statistical validation and thorough inquiry should always be the norm. Transparency of the analytical approach and assumptions is central to successful decision-making. If the models are used by upper management to guide and inform decisions, the underlying assumptions and algorithms used to provide the core information must be visible and understood by all involved.

“In the end, the science of analysis must be married with the art of intuition and experience to make BI programs bring the anticipated results.”

Thomas H. Davenport highlighted a number of key questions that everyone involved in the process should be asking in a Harvard Business Review article from the July-August, 2013 edition “Keep Up with Your Quants.” These questions included:

  • What was/were the source(s) of the data?
  • How well does the sample represent the population?
  • Were there outliers and did they affect the results?
  • What assumptions are in your analysis and models? Do any conditions render this invalid?
  • Why did you choose the specific analytical approach? What options did you consider?
  • How are you certain of causality vs. coincidental correlation of the outcomes with the variables used?

This type of inquiry has not been common or pervasive until recently. Management must understand the fundamentals and assumptions used to establish institutional confidence in the information and guidance that comes out of the analytics and decision support models.

In the end, the science of analysis must be married with the art of intuition and experience to make BI programs bring the anticipated results. Artificial Intelligence (AI) will play a more significant role in the future. In the meantime, the behaviors identified above are fundamental to the success of any analytics and BI program. It is necessary to keep strong and open relationships between the quants and the decision makers.

Do you want to be a quant? There are myriad opportunities across all industries and functional areas for individuals to delve into the world of quantitative analysis. As the data streams continue to grow exponentially with IoT to become tsunamis, the need will increase further. Quants will no longer be a luxury for business success, they will be necessary. The future will bring more AI into play – people working on data and models now are likely to grow and evolve to develop new AI algorithms.  Their role will expand to not only help evaluate data with known models, but also to create and maintain models that are intelligent and adaptive to update themselves. It’s an exciting future in Big Data, BI, Analytics, and AI!

Who’s your quant?

 

“I never guess. It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.” Sir Arthur Conan Doyle (Sherlock Holmes author)

 

Michael Massetti is an Executive Partner with Gartner who really does enjoy being a supply chain professional! Seriously.

All opinions are my own.