Data & AI: The Importance of People in a Data-Driven World

The 80-20 Rule and Data Science Challenges

The Pareto principle, frequently referred to as the 80-20 rule, proposes that approximately 80% of effects originate from 20% of causes, implying a significant disparity in impact.

The Data Scientist's 80-20 Reality

Within the field of data science, a common interpretation of this rule suggests that data scientists dedicate around 80% of their working hours to data cleaning and preparation, rather than actual analysis or insight generation. Consider a short commute significantly extended by unexpected congestion – a similar principle applies.

While the actual percentage may vary, a substantial portion of a data scientist’s time is consumed by transforming raw, disorganized data into a usable, analytical dataset. This involves tasks like identifying and removing duplicate entries, ensuring consistent formatting, and performing other essential preparatory steps.

Time Allocation in Data Preparation

Recent surveys indicate that, on average, this data preparation phase accounts for roughly 45% of the total time invested in a project. Earlier research, such as a poll conducted by CrowdFlower, estimated this figure to be as high as 60%, with numerous studies reporting similar results.

The Importance of Data Quality

It’s crucial to understand that data preparation is not a trivial pursuit. The principle of “garbage in, garbage out” is fundamental in computer science and holds true for data science as well. Poor data quality can lead to inaccurate results.

At best, flawed data may cause a script to fail, preventing calculations. At worst, it can result in business decisions based on unreliable insights. For example, an error might occur when attempting to calculate average spending due to a customer’s entry being incorrectly formatted as text.

The Cost of Inefficient Data Handling

A key consideration is whether dedicating a highly compensated data scientist’s time to repetitive data reformatting is the most efficient use of resources. Estimates place the average data scientist’s salary between $95,000 and $120,000 annually.

Assigning such an expert to mundane, non-specialized tasks represents a loss of both their valuable time and the company’s financial investment. Furthermore, the timeliness of data is critical; prolonged collection and processing can render a dataset obsolete before analysis can commence.

Wasted Effort Beyond the Data Science Team

The pursuit of data often extends beyond the data science team, frequently involving the time of personnel whose primary responsibilities lie elsewhere. Employees may be asked to gather or produce data, diverting them from their core duties.

Compounding this issue, over half of the data collected by organizations often remains unused, indicating a significant waste of time and resources across the board, leading to operational delays and associated financial losses.

Data Overload and Underutilization

Even when data is collected, it is frequently concentrated within a dedicated data science team that is already overburdened and unable to fully explore all available information.

Data Accessibility: A New Imperative for Businesses

Many organizations, excluding the initial data innovators like Google and Facebook, are still navigating the transition to a data-driven operational model. A significant challenge lies in the process of data acquisition and preparation, where data scientists often spend considerable time cleaning data while those involved in initial collection frequently don't benefit from the resulting insights.

We are still in the nascent stages of comprehensive data transformation. The achievements of technology leaders who prioritized data within their core strategies have ignited a movement that is only now gaining momentum. Despite currently varied outcomes, this indicates that businesses are still developing a data-centric mindset.

The Value of Data and the Human Element

The inherent value of data is widely recognized by businesses, evidenced by the high demand for AI specialists across diverse industries. However, success hinges on proper implementation, and a crucial aspect of this is prioritizing people alongside artificial intelligence.

Data has the potential to improve nearly every facet of an organization’s structure. While the idea of a machine learning solution for each business process is appealing, it’s not currently a necessary step. The primary objective for companies seeking to leverage data is to efficiently move it from its origin to the individuals who require it for informed decision-making.

Democratizing Data Access

It’s important to note that the recipient of this data doesn’t necessarily need to be a data scientist. It could be a manager optimizing workflow, an engineer identifying manufacturing defects, or a UI designer conducting A/B testing. These individuals require consistent, readily available data for insightful analysis.

Investing in employees and providing them with fundamental analytical skills allows them to effectively utilize data, mirroring the capabilities of machine learning models. In this context, accessibility is paramount.

Beyond the Buzzword: Practical Data Analytics

Some may dismiss “big data” as mere corporate jargon, but robust analytical capabilities can demonstrably improve a company’s financial performance, provided there’s a well-defined strategy and realistic expectations. The initial focus should be on ensuring data is accessible and user-friendly, rather than simply accumulating large volumes of it.

Ultimately, cultivating a comprehensive data culture is as vital for an enterprise as its underlying data infrastructure.

• Focus on data accessibility over sheer volume.
• Invest in employee training for basic data analysis.
• Ensure data reaches the individuals who need it for decision-making.