Apple AI Models: Performance Falls Short of Expectations

Apple Intelligence Model Performance: A Detailed Look

Apple recently revealed updates to the artificial intelligence models that underpin its Apple Intelligence features, spanning operating systems like iOS and macOS. However, internal evaluations suggest these models don't quite match the performance of older models developed by competing technology companies, such as OpenAI.

Text Generation Capabilities

According to a blog post published on Monday, human evaluators found the quality of text produced by the new “Apple On-Device” model – designed for offline operation on devices like the iPhone – to be on par with, but not exceeding, that of comparable models from Google and Alibaba.

Further assessment revealed that Apple’s more powerful “Apple Server” model, intended for use within the company’s data centers, lagged behind OpenAI’s GPT-4o, which was released last year.

Image Analysis Performance

In separate testing focused on image analysis, human reviewers expressed a preference for Meta’s Llama 4 Scout model over Apple Server. This finding is noteworthy.

It's somewhat unexpected, considering Llama 4 Scout often demonstrates lower performance than leading AI models from organizations like Google, Anthropic, and OpenAI in numerous tests.

Implications for Apple's AI Strategy

These benchmark results corroborate existing reports indicating challenges within Apple’s AI research division in keeping pace with the rapidly evolving AI landscape.

Apple’s advancements in AI have, in recent years, been considered underwhelming, leading to a postponement of a planned Siri upgrade.

Legal action has even been initiated by some customers, alleging that the company has promoted AI functionalities for its products that are not yet fully realized.

Technical Specifications and Improvements

The Apple On-Device model, approximately 3 billion parameters in size, powers features such as text summarization and analysis. (The number of parameters generally correlates with a model’s capacity for problem-solving; larger models typically outperform smaller ones.)

As of Monday, external developers can access this model through Apple’s Foundation Models framework.

Enhanced Features and Multilingual Support

Apple states that both Apple On-Device and Apple Server exhibit improved tool utilization and efficiency compared to their previous iterations.

They also support understanding approximately 15 different languages.

This enhancement is partially attributable to an expanded training dataset encompassing image data, PDFs, documents, manuscripts, infographics, tables, and charts.