Pruna AI Open Sources AI Model Optimization Framework

Pruna AI Open Sources AI Model Optimization Framework

Pruna AI, a European startup specializing in compression algorithms for AI models, is set to release its optimization framework as open source this Thursday.

Framework Capabilities

The framework developed by Pruna AI incorporates multiple efficiency techniques. These include caching, pruning, quantization, and distillation, all applied to a given AI model.

Standardized procedures for saving and loading compressed models are also a key feature. The framework facilitates the application of combined compression methods and provides evaluation of the compressed model’s performance post-compression.

Performance Evaluation

A crucial aspect of Pruna AI’s framework is its ability to assess potential quality degradation following model compression. It also quantifies the resulting performance improvements.

According to Pruna AI co-founder and CTO John Rachwan, the framework aims to replicate the standardization achieved by Hugging Face with transformers and diffusers. “We are focused on establishing similar standards, but specifically for efficiency methods,” he stated.

Industry Context

Compression techniques are already widely utilized by major AI labs. OpenAI, for example, leverages distillation to create faster iterations of its core models.

The development of GPT-4 Turbo, a more rapid version of GPT-4, likely benefited from this approach. Similarly, the Flux.1-schnell image generation model is a distilled version of Black Forest Labs’ Flux.1 model.

Distillation Explained

Distillation involves transferring knowledge from a large, established AI model – the “teacher” – to a smaller model – the “student.”

Developers submit queries to the teacher model and record the responses. Accuracy is often verified against a dataset. These outputs then serve as training data for the student model, guiding it to mimic the teacher’s behavior.

Value Proposition

“Larger organizations typically develop these tools internally,” Rachwan explained. “Open-source solutions often concentrate on single methods, such as a specific quantization technique for LLMs or a caching method for diffusion models.”

“However, a comprehensive tool that integrates all methods, simplifying their use and combination, is currently unavailable. This is where Pruna AI delivers significant value.”

Model Support and Focus

While Pruna AI supports a wide range of models – including large language models, diffusion models, speech-to-text models, and computer vision models – the company’s current emphasis is on image and video generation models.

Current Users and Offerings

Existing users of Pruna AI include Scenario and PhotoRoom. Alongside the open-source version, Pruna AI provides an enterprise edition featuring advanced optimization capabilities, including an optimization agent.

The Compression Agent

“Our upcoming compression agent is particularly exciting,” Rachwan revealed. “Users simply provide their model and specify desired speed improvements, along with an acceptable accuracy loss threshold – for instance, no more than 2%.”

“The agent then autonomously identifies the optimal combination of techniques and delivers the optimized model, requiring no manual intervention from the developer.”

Pricing Model

Pruna AI’s professional version is priced based on usage time. “It functions similarly to renting a GPU on platforms like AWS or other cloud services,” Rachwan clarified.

Optimized models can lead to substantial cost savings in AI infrastructure. Pruna AI has successfully reduced a Llama model’s size by a factor of eight with minimal performance impact, positioning its framework as a self-funding investment.

Funding

Pruna AI recently secured $6.5 million in seed funding. Investors include EQT Ventures, Daphni, Motier Ventures, and Kima Ventures.