Are Companies Making Smarter Decisions with Big Data?

The Evolution of Big Data: From Volume to Velocity and Insight

Back in December 2014, the challenge centered around extracting valuable wisdom from the growing mass of big data. Significant progress has been made in data storage, processing capabilities, and overall digital data management. However, reliably identifying pertinent information that drives positive business results remains a persistent difficulty for human analysts.

The "Moneyball" Effect: Baseball as a Data Laboratory

Baseball provides a compelling illustration of data's potential impact on business strategy. The story of Billy Beane and the Oakland Athletics, popularized by the book and film “Moneyball,” demonstrated the power of advanced statistical analysis over traditional scouting reports.

Today, data analysts play a role in baseball as significant as that of seasoned professionals. However, a question arises: is it possible to have too much data available?

The Expanding Coaching Staff: A Symptom of Data Overload

Alex Spier, writing for the Boston Globe, recently highlighted a notable shift in the Boston Red Sox organization. Manager Alex Cora now has 11 coaches supporting 26 players. This contrasts sharply with Terry Francona’s staff of six coaches for 25 players in 2011.

Spier attributes this increase, in part, to the expanding volume of data collected by teams. This necessitates a larger team to observe, interpret, and translate data into actionable plans. As Spier noted, teams are increasingly employing multiple hitting coaches and expanding their staffs to effectively process the “mountains of information” available.

Baseball serves as a valuable testing ground for advanced statistical methods, offering lessons for businesses navigating expanding datasets.

From Big Data to Fast Data: The Need for Speed

Organizations invest heavily in data lakes, machine learning models, and data-driven decision-making. Yet, they continue to confront fundamental decisions involving their ever-growing data resources. While technology assists in sorting through this information, the prevalence of ineffective advertising and email marketing demonstrates that substantial improvements are still needed.

Deepak Jeevankumar, managing director at Dell Technologies Capital, emphasizes a key shift since 2014. Companies are now prioritizing the rapid delivery of insights to those who require them. “Big Data has become less relevant than ‘fast data’,” Jeevankumar explained. Consumers and businesses alike demand quick insights and knowledge.

Real-Time Analysis and Investor Interest

Jeevankumar believes data should be analyzed both during streaming and immediately after a query. Startups providing solutions for real-time data analysis, for both companies and sports teams, are poised for success.

Startups focused on analyzing and extracting insights from fast data are attracting significant investor attention, with companies like Confluent and Databricks serving as successful examples.

Decision Intelligence: A New Approach to Data Analysis

Just as baseball coaches must distill data for their players, business analysts and managers need to provide employees with accurate, timely information for efficient decision-making.

Pam Baker, a data journalist, identifies both progress and challenges in today’s data landscape. “It is harder, much harder,” she states. However, this doesn’t necessitate less data, but rather, new and improved methods for uncovering answers. This has led to a shift from data mining towards decision intelligence.

Querying for Answers, Not Just Digging for Nuggets

Decision intelligence allows users to search for specific answers, rather than sifting through vast amounts of data hoping to find something useful. This involves formulating effective questions and generating meaningful results.

While targeted questioning carries the risk of confirmation bias, Baker distinguishes between bias and sound analysis. The goal is to gather data relevant to the research question, differentiating between data-driven and decision-driven organizations.

The Limitations of Predictive Analytics

Baker notes that while tooling has improved, some areas haven’t progressed significantly since 2014. Predictive analytics, for example, often makes flawed assumptions about consumer behavior. Suggesting an engagement ring purchase based on a previous spring purchase demonstrates this limitation.

“Predictive analytics project a future outcome based on past outcomes and assumes nothing has changed,” Baker explains. “Human behaviors just aren’t that clean and neat.” Building nuance and context into analytics remains a challenge, raising privacy concerns about the extent of data collection.

The Human Element Remains Crucial

Over the past seven years, big data analysis has undeniably advanced. The underlying infrastructure, technology, and analytical tools have all improved. However, the problem isn’t fully solved. As baseball illustrates, it’s easy to become overwhelmed by data details and lose sight of the fact that, ultimately, success depends on human execution. Maintaining this human focus is paramount.