AI Forgetfulness: How Losing Memory Can Improve Performance

The Challenge of Artificial Forgetfulness in Machine Learning

Contemporary machine learning systems, despite exhibiting characteristics akin to artificial intelligence, lack genuine comprehension of the data they process. Consequently, they often retain even insignificant data indefinitely.

Researchers at Facebook have proposed a concept termed “structured forgetfulness” as a mechanism for AI to streamline its processes. This approach aims to enhance performance and move closer to the functionality of the human mind.

Human vs. AI Data Retention

The researchers illustrate the problem by contrasting how humans and AI agents might tackle a comparable task.

Consider a scenario involving ten doors, each a different color. If you are instructed to select the yellow door, you proceed accordingly and subsequently forget the colors of the remaining doors. This is because the specific colors of the other doors – whether red, plaid, or walnut – are irrelevant; only the identification of the yellow door matters.

Your brain efficiently discards this unnecessary information almost instantly.

However, an AI might retain the colors and positions of all nine non-yellow doors in its memory. This occurs because it lacks the intuitive understanding of the problem and the data itself, leading it to preserve all information used in its decision-making process.

The Impact of Excessive Data Storage

While this isn’t problematic with small datasets, modern machine learning algorithms routinely manage millions of data points and process terabytes of images or text, particularly during training.

Their architecture necessitates constant comparison of new data with existing knowledge. The inability to discard unimportant information results in performance degradation due to continuous referencing of irrelevant or outdated data.

Expire-Span: A Solution to Structured Forgetfulness

The solution developed by Facebook researchers involves enabling the AI to determine the duration for which a piece of data needs to be retained during its initial evaluation.

Image Credits: Facebook

“Each individual memory is associated with a predicted expiration date, and the scale of the memory depends on the task,” explains Angela Fan, a Facebook AI researcher involved in the Expire-Span project.

“The amount of time memories are held depends on the needs of the task – it can be for a few steps or until the task is complete.”

Returning to the door example, the colors of the non-yellow doors are only relevant until the yellow one is found. After that, this information can be safely discarded. The retention period can vary depending on the number of remaining doors to be checked.

(A more practical example could involve forgetting faces that don’t match the target face once it’s identified.)

Contextual Memory Retention

When analyzing extensive text, the relevance of specific words or phrases might extend until the end of a sentence, a paragraph, or even longer. This depends on the task at hand, such as identifying the speaker, the chapter, or the genre.

This approach enhances performance by reducing the amount of information the model needs to process.

Without the ability to forget, the system retains all data, increasing its size and decreasing its speed, as it assumes the other doors might be important.

Performance and Efficiency Gains

Fan reports that models trained using Expire-Span demonstrated improved performance and efficiency, requiring less memory and processing time. This is significant during both training and testing, which can consume thousands of hours of computational resources.

Even a modest improvement translates to substantial benefits, and it also enhances the end-user experience by making tasks faster and more energy-efficient. Real-time image processing, for instance, becomes more feasible.

AI and Human Cognition

While the ability to forget brings AI processes closer to human cognition, it still falls short of the nuanced and intuitive ways our minds function. Interestingly, the capacity to control what is remembered and for how long represents a significant advantage over human memory, which often operates with seemingly random parameters.