Automated Video Game Highlight Detection: 3 Methodologies
The Evolution of Gaming Through Livestreaming
The advent of livestreaming has fundamentally altered gaming, transforming it from a simple pastime into a recognized platform for both entertainment and competitive play.
Since being acquired by Amazon in 2014, Twitch has experienced substantial growth, with its average concurrent viewership increasing from 250,000 to over 3 million. Similar upward trends are being observed with competing platforms such as Facebook Gaming and YouTube Live.
The Expanding Streaming Ecosystem
This surge in viewership has stimulated the development of a supporting network of products. Contemporary professional streamers are consistently demanding more from technology to enhance the quality of their broadcasts.
They are also seeking ways to automate the more routine elements of the video production process.
The Demands of Full-Time Streaming
Professional online streaming is a demanding undertaking. Dedicated content creators often dedicate eight to twelve hours daily to their performances.
Extended, 24-hour marathon streams are frequently employed as a strategy to capture and retain viewer attention.
Beyond the Broadcast: Maintaining a Stream's Presence
However, time spent actively streaming represents only a portion of the overall commitment. Consistent engagement on social media and platforms like YouTube is crucial for channel growth.
This sustained activity attracts viewers to live streams, where opportunities for revenue generation exist through subscriptions, donations, and advertising.
The Challenge of Content Editing
Extracting the most engaging segments – typically five to ten minutes – from lengthy streams of eight or more hours requires significant time and effort.
While top-tier streamers can afford to employ video editors and social media managers, emerging and part-time streamers often lack the resources or time to effectively manage this crucial aspect of their work.
Balancing footage review with other personal and professional obligations presents a considerable challenge.
Computer Vision Analysis of Game UI
Automated tools are increasingly being utilized to pinpoint significant moments within extended broadcasts. A number of startups are currently vying for leadership in this developing market segment. The differing strategies employed in tackling this challenge are what set these competing solutions apart.
Many of these approaches are rooted in a traditional computer science principle: the trade-off between hardware and software solutions.
Athenascope was among the first companies to successfully implement this concept on a large scale. Supported by $2.5 million in venture capital and a team comprised of experienced professionals from major Silicon Valley technology companies, Athenascope created a computer vision system designed to identify highlight clips within longer recordings.
The underlying principle is similar to that used in autonomous vehicles. However, instead of interpreting road signs and traffic signals through camera input, this tool analyzes the gamer’s screen and identifies indicators within the game’s user interface that signify important in-game events, such as kills, deaths, goals, saves, wins, and losses.
These visual signals are the same cues that players traditionally rely on to understand the unfolding action within a game. Modern game UIs are designed to present this information in a high-contrast, clear, and unobstructed manner, typically in consistent locations on the screen.
This predictability and clarity makes these interfaces particularly well-suited for computer vision techniques, including optical character recognition (OCR) – the process of extracting text from images.
The potential consequences of errors are also less severe than in self-driving car applications. A false positive in this system simply results in a less exciting video clip, rather than a potentially dangerous situation.
However, a computer vision methodology does present certain limitations. The artificial intelligence required is computationally intensive, potentially exceeding the capacity of an average user’s system while simultaneously rendering a modern game at high resolution and frame rates, and encoding a live video stream.
Consequently, the AI must operate in the cloud. Raw video data is uploaded to Athenascope’s server infrastructure – known as “Athena” – processed, and then the resulting highlights are delivered to the user for download. Maintaining these powerful video analytics servers represents a significant operational expense for Athenascope.
Furthermore, the process of uploading and downloading raw video introduces latency and potential quality degradation.
Startups such as Clip It, where we have a founding role, are working to overcome these drawbacks by optimizing the image processing AI to run locally on the user’s computer. This approach offers faster results and reduces the infrastructure costs for the company.
Accessing Game Memory
The inherent challenges associated with computer vision techniques prompt an alternative strategy for tackling similar problems. Instead of relying on rendered video pixels as input, a program can directly examine a game's raw memory during execution.
This approach bypasses video rendering altogether, enabling direct access to the game's internal representation of notifications and events in their most fundamental form.
Overwolf stands as the leading innovator in this specific area. Established in 2010 with an initial seed investment of $100,000, the company recently unveiled a $50 million fund to support creators leveraging its platform.
This platform is fundamentally built upon the principles of direct memory access. Unlike Athenascope, which functions as a direct-to-consumer service, Overwolf generates revenue by licensing its technology to other developers.
Direct memory access offers advantages in both speed and reliability compared to computer vision. It eliminates the need for computationally intensive image analysis, and the data acquired is immediately usable.
However, inspecting the running memory of a third-party application exists within a legal and security gray area. This is precisely the technique employed by many cheating programs, such as aimbots used in first-person shooters, which violate game terms of service.
Consequently, significant resources within the game development industry are dedicated to preventing this type of access. Game memory is frequently obfuscated or encrypted.
Furthermore, the anti-cheat systems in numerous popular competitive games actively monitor for and block any unauthorized memory access attempts.
In June, an update to Call of Duty’s anti-cheat system incorrectly identified Overwolf as malicious, resulting in a block. It required over a month of collaboration between Overwolf and the Call of Duty development team to establish a manual exception.
This restored functionality for Overwolf’s user base. Beyond these security concerns, any modification to the game’s code that alters the internal memory structure will also disrupt compatibility for programs reliant on memory access.
The specific memory locations utilized by these programs may shift, leading to a fragility that necessitates continuous development and, inevitably, some degree of customer downtime with each game update.
Essentially, the maintenance costs associated with cloud infrastructure for computer vision are exchanged for ongoing development expenses to maintain synchronization with game updates.
This also includes direct communication and negotiation with game developers when required. Playstream.gg provides another illustration of the internal memory method in practice.
Their distinctive offering centers around automated in-game challenges, rather than video clip recording.
GPU-Integrated Software Development Kits
A third, separate technique is observable within the realm of automated highlight detection systems.
This methodology was pioneered by NVIDIA with its NVIDIA Highlights feature, previously known as Shadowplay. NVIDIA’s specialized role within the computer graphics process provides it with immediate access to video information directly on the GPU – a distinction from Athenascope, which relies on receiving video data via an internet stream.
Consequently, this results in exceptionally rapid and high-fidelity video recording. Unlike alternative systems, NVIDIA empowers the games themselves to manage clip creation; they provide a software development kit (SDK) enabling game developers to integrate with an NVIDIA GPU and request a clip of the preceding 15 to 30 seconds as needed.
This shifts the responsibility for implementation to each game’s development team, meaning NVIDIA Highlights functionality is only available in games where developers have specifically added support for it.
A further prerequisite is the presence of an NVIDIA GPU in the user’s system; AMD users are unsupported, as AMD’s replay recording tools lack automatic highlight recording capabilities. Support for console platforms (Xbox, PlayStation, and others) is also absent. Conversely, computer vision techniques are universally applicable, and the source platform of the video does not impact product compatibility.
A hybrid solution, merging the GPU-accelerated video capture of NVIDIA Highlights with the computer vision methods of Athenascope, could offer a compelling balance of advantages. This could provide the responsiveness of Overwolf alongside the platform independence of a computer vision system, potentially with the machine learning processes also executed on the GPU. Currently, no application employs this specific combination.
Analyzing Gameplay: A Comparative Overview
A key factor distinguishing competitors within this field centers on the point in the video processing sequence where the analysis is performed.
Athenascope represents one end of this range, conducting its analysis on the completed video output. This occurs following capture, encoding, the addition of overlays or filters, and subsequent upload to Athenascope’s servers.
Clip It shifts the analytical process nearer to the user, employing a similar method but executing it in real-time directly on the user’s machine.
Both Overwolf and Playstream further refine this proximity by examining the game’s memory while it is actively running. At the opposite extreme lies NVIDIA Highlights, which directly accesses video data from the GPU’s core, initiated through SDK integrations within the game’s code.
These nuanced differences in methodology form the core of competition within the automated video game highlight detection and capture market. Increasingly advanced computer vision AI is making software-based solutions more competitive with the performance advantages offered by hardware-level approaches.
