Meta Exec Denies Llama 4 Benchmark Boosting

Meta Denies Claims of Benchmark Manipulation for New AI Models

A Meta executive refuted allegations circulating on Monday concerning the tuning of the company’s recently released AI models. The claim suggested that these models were intentionally optimized for specific benchmarks, while simultaneously masking their inherent limitations.

Response from Meta's VP of Generative AI

Ahmad Al-Dahle, Vice President of Generative AI at Meta, addressed the rumor directly via a post on X. He stated unequivocally that it is “simply not true” that Meta’s Llama 4 Maverick and Llama 4 Scout models were trained utilizing “test sets.”

In the context of AI benchmarks, test sets represent curated datasets employed to assess a model’s performance following its training phase. Training a model on a test set can artificially elevate its benchmark scores, creating a potentially deceptive impression of its actual capabilities.

Origin of the Allegations

The unsubstantiated rumor regarding artificially inflated benchmark results surfaced over the weekend, spreading across platforms like X and Reddit. The initial source appears to be a post on a Chinese social media platform, attributed to an individual claiming to have resigned from Meta due to disagreements with the company’s benchmarking procedures.

Performance Discrepancies and LM Arena

Reports indicating underwhelming performance from Maverick and Scout in specific tasks contributed to the spread of the rumor. Furthermore, Meta’s choice to employ an experimental, unreleased iteration of Maverick to achieve improved results on the LM Arena benchmark raised concerns.

Researchers have noted significant behavioral variations between the publicly available Maverick model and the version hosted on LM Arena.

Acknowledging Varied Model Quality

Al-Dahle did acknowledge that some users are experiencing “mixed quality” in the outputs generated by Maverick and Scout across various cloud providers.

Ongoing Optimization and Bug Fixes

“Given that the models were released promptly after completion, we anticipate a period of several days will be required for all public implementations to be fully optimized,” Al-Dahle explained. He further assured that Meta remains committed to addressing bug fixes and facilitating partner onboarding.

Meta will continue to refine the models and work with partners to ensure optimal performance.