AI Efficiency Technique Drawbacks

The Approaching Limits of AI Quantization

A prevalent method for enhancing the efficiency of AI models, known as quantization, is encountering inherent limitations. The industry may be nearing the point where further gains become increasingly difficult to achieve.

Understanding Quantization in AI

Within the realm of artificial intelligence, quantization involves reducing the number of bits required to represent information. These bits are the fundamental units processed by computers.

Think of it this way: when asked for the time, a response of “noon” is sufficient, rather than a highly specific “twelve hours, zero minutes, one second, and four milliseconds.”

This simplification – or quantization – maintains accuracy while reducing complexity. The necessary level of precision is always dependent on the specific application.

How Quantization Impacts AI Models

AI models are comprised of numerous components suitable for quantization, notably their parameters. These parameters are the internal variables used by the model to generate predictions and make decisions.

Given the millions of calculations performed during model execution, reducing the bit-depth of these parameters significantly lowers computational demands.

It’s important to note that quantization differs from distillation, which is a more complex process involving selective parameter pruning.

Potential Trade-offs and Future Challenges

Recent findings suggest that quantization may involve more substantial trade-offs than initially believed.

Further advancements in quantization techniques may yield diminishing returns, necessitating exploration of alternative methods for improving AI model efficiency.

The industry is actively researching these alternatives to overcome the limitations of current quantization approaches.

The Trend Towards Smaller AI Models

Research conducted collaboratively by Harvard, Stanford, MIT, Databricks, and Carnegie Mellon indicates that the performance of quantized models can be negatively impacted if the original, unquantized model underwent extensive training with a large volume of data. This suggests that, beyond a certain point, developing a smaller model from the outset may prove more effective.

This finding presents a potential challenge for AI firms that prioritize training exceptionally large models – a practice known to enhance response quality – and subsequently quantize them to reduce serving costs.

Recent observations support this conclusion. Reports surfaced several months ago detailing that the quantization of Meta’s Llama 3 model often resulted in more detrimental effects compared to other models, potentially linked to its training methodology.

Tanishq Kumar, a Harvard mathematics student and lead author of the study, explained to TechCrunch, “In my assessment, the primary expense for all AI stakeholders, now and in the future, is inference. Our research demonstrates that a key method for reducing this cost will not remain perpetually effective.”

The Cost of Inference vs. Training

A common misconception is that AI model training is the most significant expense. However, running a model – such as when ChatGPT provides a response – often incurs greater aggregate costs. For instance, Google’s training of a flagship Gemini model is estimated to have cost $191 million.

Conversely, utilizing the same model to generate concise, 50-word responses to half of all Google Search queries would result in an annual expenditure of approximately $6 billion.

The Scaling Paradigm

Leading AI laboratories have consistently adopted the strategy of training models on massive datasets, operating under the assumption that “scaling up” – increasing both data volume and computational resources – will yield increasingly sophisticated AI capabilities.

As an example, Meta’s Llama 3 was trained on a dataset comprising 15 trillion tokens. (A token represents a unit of raw data, with 1 million tokens equating to roughly 750,000 words.) This represents a substantial increase from the 2 trillion tokens used to train the previous generation, Llama 2.

In early December, Meta unveiled Llama 3.3 70B, asserting that it “enhances core performance while significantly lowering costs.”

Current data indicates that scaling eventually yields diminishing returns. Reports suggest that both Anthropic and Google have recently trained very large models that did not meet internal performance benchmarks.

Despite these indications, the industry shows limited inclination to deviate from its established scaling practices.

The Question of Precision in AI Models

A central question arises: if training models on limited datasets presents challenges, can methods be employed to enhance their resilience against performance decline? Research suggests a potential avenue. Kumar and his associates discovered that employing “low precision” during model training can contribute to increased robustness.

In this context, “precision” denotes the number of significant digits a numerical data type can accurately represent. Data types define the kinds of values that can be stored and the operations that can be performed on them; for instance, the FP8 data type utilizes only 8 bits for representing a floating-point number.

Currently, most models undergo training at 16-bit, often referred to as “half precision,” and are subsequently “post-train quantized” to 8-bit precision. This involves converting certain model components, such as parameters, to a lower-precision format, potentially sacrificing some accuracy. This is akin to performing calculations to several decimal places and then rounding to the nearest tenth, often achieving a balance between efficiency and accuracy.

Companies specializing in hardware, like Nvidia, are advocating for reduced precision in quantized model inference. Their latest Blackwell chip supports 4-bit precision, utilizing a data type known as FP4, which Nvidia promotes as beneficial for data centers with limited memory and power resources.

However, excessively low quantization precision may not always be advantageous. Kumar indicates that unless the initial model possesses a substantial number of parameters, precisions below 7 or 8 bits could result in a discernible reduction in quality.

The technical nature of these concepts is acknowledged, but the core idea is that AI models are not fully understood, and computational shortcuts effective in other domains may not apply here. Just as stating “noon” as the start time of a 100-meter dash would be incorrect, the principles governing AI model behavior are not always intuitive.

“The central finding of our research is that certain limitations cannot be easily circumvented,” Kumar stated. “We aim to contribute a more nuanced perspective to the ongoing discussion that frequently favors increasingly low precision as a default for both training and inference.”

Kumar recognizes that the study conducted by him and his team was conducted on a relatively small scale, with plans for future testing involving a wider range of models. Nevertheless, he believes that at least one key insight will remain valid: reducing inference costs is not without trade-offs.

“The level of bit precision is significant, and it comes at a cost,” he explained. “Indefinite reduction of precision will inevitably lead to model performance degradation. Given the finite capacity of models, I believe greater emphasis will be placed on careful data curation and filtering, ensuring that only the highest-quality data is used to train smaller models. I am hopeful that novel architectures specifically designed to stabilize low-precision training will prove crucial in the future.”

This article was originally published on November 17, 2024, and was updated on December 23 with additional information.