Stop Self-Fulfilling Prophecies: AI for Small Data

The Data Challenge in the Age of AI

The proliferation of digital technologies over the last ten years has resulted in an abundance of data. This presents numerous opportunities, particularly regarding the potential for artificial intelligence to revolutionize business operations.

However, within the B2B sector – an area with which I have significant experience – a scarcity of data persists. This is primarily due to the comparatively lower volume of transactions when contrasted with B2C environments. Consequently, for AI to fulfill its promise of transforming enterprises, it must effectively address these challenges associated with limited datasets.

The Pitfall of Self-Fulfilling Prophecies

A common issue arises from data scientists employing flawed methodologies, inadvertently creating self-fulfilling prophecies. This diminishes the efficacy of AI when dealing with small data scenarios and ultimately restricts its broader impact on enterprise advancement.

The concept of a “self-fulfilling prophecy” is recognized in fields like psychology and finance. In data science, it manifests as simply “predicting the obvious.” This occurs when organizations identify a model that reinforces existing successes, sometimes even intentionally, and then apply it to unrelated situations.

For example, consider a retailer discovering that customers who add items to their online shopping carts are more inclined to complete a purchase. Focusing marketing efforts solely on this group is predicting a known outcome!

A more effective strategy would be to concentrate on optimizing areas that currently underperform – specifically, converting first-time visitors who haven’t yet added anything to their carts. By focusing on the non-obvious, the retailer is more likely to drive sales growth and attract new customers, rather than simply retaining existing ones.

A Five-Step Process for AI in Small Data Environments

To circumvent the creation of self-fulfilling prophecies, adhere to the following process when applying AI to small data problems:

1. Enrich Your Existing Data

When faced with limited data, the initial step is to enhance the information you already possess. This can be achieved by integrating external data sources and employing look-alike modeling. The success of recommendation systems utilized by platforms like Amazon, Netflix, and Spotify demonstrates this principle.

Even with minimal purchase history, these platforms leverage extensive product and customer data to generate remarkably accurate predictions. If your B2B organization currently categorizes deals using a broad metric – such as “large companies” – emulate Pandora’s approach by dissecting each customer based on granular details.

The more comprehensive your data, the more insightful and powerful your predictive models become. You can transition from low-dimensional data with limited predictive power to high-dimensional knowledge capable of driving impactful recommendations.

2. Model the Future, Not Just the Past

Data science can be approached in two fundamental ways. The empirical method, used when lacking initial direction, allows the data to reveal patterns. The scientific method, conversely, involves formulating a hypothesis and then designing tests to validate it.

Relying exclusively on either method is insufficient. Over-reliance on the empirical method leads to self-fulfilling prophecies. This is particularly evident when launching a new product with no historical data. Validation requires a designed test.

Combining hypothesis creation with data-driven testing provides a more accurate view of potential future outcomes.

3. Add Semantic Meaning to Your Data

Once a hypothesis is established, define the relationships between different data points. Analyzing a shopper who purchases diapers, bottles, and nursery decor can lead to “slicing the data too thin” if you don’t recognize the underlying pattern – a new baby.

External knowledge and human expertise can significantly improve results through semantic modeling, providing context and accelerating machine learning, especially when dealing with limited data. Building a robust taxonomy is crucial for success.

For example, a large medical device company with millions of SKUs relies on human experts to develop a taxonomy that understands product families and customer patterns, ultimately improving predictive modeling.

4. Prioritize Speed and Control

As you approach the final stages, return to the data, now prepared to support your hypothesis and testing. If feasible, establish a controlled lab environment where you can introduce new variables and rapidly conduct A/B tests to gain insights.

This approach is particularly effective in marketing campaigns where feedback on lead conversion isn’t delayed by lengthy sales cycles. Designing a “control” group is critical for uncovering the complete truth, especially when historical data is limited.

The COVID-19 vaccine serves as an example: focusing solely on vaccinated individuals getting sick presents an incomplete picture. Comparing this group to an unvaccinated control group reveals the model’s overall effectiveness.

5. Focus on Business Metrics, Not Just Past Performance

Continuously using past successes to predict the future limits potential growth. Marketing reports may highlight impressive lead generation, but if those leads don’t translate into closed deals, the model remains ineffective.

With enriched data, a clear hypothesis, semantic understanding, controlled testing, and trials in place, measure everything against tangible business results – namely, revenue.

A B2B company that experienced rapid growth during the pandemic recognized the need to avoid self-fulfilling prophecies as the landscape shifted. Staying focused on the bottom line, rather than being distracted by misleading conversion metrics, proved crucial.

The Future of AI and Data

It’s tempting to believe that AI is unattainable without sufficient data. However, applying correct data science practices and avoiding pitfalls like self-fulfilling prophecies is the key to unlocking AI’s potential, even with limited datasets.

Therefore, whether you’re a B2B company or launching a new product, AI can still be a valuable asset.

When AI can effectively address both small and large data challenges, it will usher in a new era of scientific and technological advancement.