Cohere's Aya: A New Best-in-Class Vision AI Model

Cohere Unveils Aya Vision: A New Multimodal AI Model

Cohere For AI, the nonprofit research arm of the AI startup Cohere, has this week introduced Aya Vision, a multimodal AI model. The lab asserts that this new model represents a leading advancement in its class.

Capabilities of Aya Vision

Aya Vision is capable of performing a diverse range of tasks. These include generating descriptive captions for images, providing answers to questions posed about photographs, translating text between languages, and creating concise summaries. It supports 23 major languages.

Cohere is also offering access to Aya Vision at no cost via WhatsApp. This move, according to the company, signifies “a substantial stride toward broadening access to cutting-edge technical advancements for researchers globally.”

Addressing Linguistic Gaps in AI

Despite considerable progress in the field of Artificial Intelligence, a noticeable disparity remains in model performance across different languages. This gap is particularly evident in multimodal applications involving both textual and visual data.

As Cohere explained in a blog post, Aya Vision was specifically designed to mitigate this issue and help bridge the existing linguistic divide.

Model Variations and Performance

Aya Vision is available in two versions: Aya Vision 32B and Aya Vision 8B. The more advanced version, Aya Vision 32B, is claimed to establish “a new benchmark” in performance.

It reportedly surpasses models that are twice its size, including Meta’s Llama-3.2 90B Vision, on specific visual understanding assessments. Furthermore, Aya Vision 8B achieves superior results on certain evaluations compared to models ten times larger, according to Cohere’s data.

Availability and Licensing

Both models are accessible through the AI development platform Hugging Face. They are released under a Creative Commons 4.0 license, incorporating Cohere’s acceptable use addendum. It’s important to note that commercial applications are prohibited.

Training Methodology: Leveraging Synthetic Annotations

The training of Aya Vision utilized a “diverse collection” of English datasets. These datasets were then translated and employed to generate synthetic annotations.

Annotations, also known as tags or labels, are crucial for helping models understand and interpret data during the training process. An example would be markings around objects in an image or captions describing the people, places, or objects within it.

Cohere’s adoption of synthetic annotations aligns with a growing trend within the industry. While potential drawbacks exist, companies like OpenAI are increasingly utilizing synthetically generated data to train models as the availability of real-world data diminishes.

Gartner estimates that 60% of the data used for AI and analytics projects in the previous year was synthetically created.

Efficiency and Resource Optimization

According to Cohere, training Aya Vision with synthetic annotations allowed for reduced resource consumption while maintaining competitive performance levels.

“This demonstrates our commitment to efficiency and achieving more with less computational power,” Cohere stated in its blog. “It also provides increased support for the research community, which often faces limitations in access to computing resources.”

Introducing AyaVisionBench: A New Evaluation Suite

Alongside Aya Vision, Cohere has also released a new benchmark suite called AyaVisionBench. This suite is designed to assess a model’s capabilities in “vision-language” tasks.

These tasks include identifying discrepancies between two images and converting screenshots into executable code.

Addressing the “Evaluation Crisis” in AI

The AI sector is currently experiencing what some refer to as an “evaluation crisis.” This stems from the widespread use of benchmarks that provide aggregate scores which often do not accurately reflect proficiency in tasks that are most relevant to AI users.

Cohere contends that AyaVisionBench represents a step towards resolving this issue, offering a “comprehensive and challenging” framework for evaluating a model’s understanding of multiple languages and modalities.

The hope is that this new benchmark will provide a more accurate assessment of AI capabilities.

“The dataset functions as a robust benchmark for evaluating vision-language models in multilingual and real-world scenarios,” Cohere researchers wrote on Hugging Face. “We are making this evaluation set available to the research community to foster further advancements in multilingual multimodal evaluations.”