OpenAI Codex: Agentic Coding Tools - An Overview

OpenAI's Codex and the Rise of Agentic Coding

Last Friday marked the introduction of Codex by OpenAI, a novel coding system engineered to tackle intricate programming assignments based on instructions given in natural language. This development positions OpenAI within a burgeoning field of agentic coding tools that are only now beginning to emerge.

From Autocomplete to Autonomous Agents

Existing AI coding assistants, such as GitHub’s Copilot, Cursor, and Windsurf, largely function as highly advanced autocomplete systems. These tools typically operate within an integrated development environment, requiring direct interaction with the AI-generated code by the user. Completely assigning a task and retrieving the finished product remains a significant challenge.

However, a new generation of agentic coding tools – including Devin, SWE-Agent, OpenHands, and OpenAI Codex – are being designed to operate without requiring the user to directly view the code. The intention is to emulate the role of an engineering team manager, delegating tasks through platforms like Asana or Slack and receiving updates upon completion.

The Next Step in Automation

For proponents of advanced AI, this represents a logical progression in the ongoing automation of software development processes. The increasing capabilities of AI are poised to take on more and more of the workload.

Kilian Lieret, a researcher at Princeton and a member of the SWE-Agent team, explains the evolution: “Initially, code was written entirely by manual keystrokes.” He continues, “GitHub Copilot introduced genuine auto-completion, representing a second stage. While developers remain actively involved, shortcuts become available.”

Moving Beyond the Developer Environment

The ambition for agentic systems is to transcend traditional developer environments. The goal is to present coding agents with a problem and allow them to resolve it independently. Lieret states, “We are shifting focus to the management level, where a bug report is assigned and the bot attempts a fully autonomous fix.”

Achieving this is a complex undertaking, and initial results have been mixed.

Early Challenges and Criticism

Following its general availability in late 2024, Devin faced substantial criticism from online commentators, alongside a more nuanced assessment from an early client at Answer.AI. A common observation among experienced users was that overseeing the models often requires as much effort as completing the task manually, due to frequent errors.

(Despite a somewhat challenging launch, Devin’s potential has been recognized by investors – Cognition AI, Devin’s parent company, reportedly secured hundreds of millions of dollars in funding at a $4 billion valuation in March.)

The Importance of Human Oversight

Even those optimistic about the technology emphasize the need for human supervision in the coding process. Unsupervised “vibe-coding” is discouraged.

Robert Brennan, CEO of All Hands AI, which develops OpenHands, notes, “Currently, and for the foreseeable future, human code review is essential. I’ve observed instances where automatically approving all agent-generated code leads to significant issues.”

Addressing Hallucinations

Hallucinations remain a persistent problem. Brennan recounts an instance where the OpenHands agent, when queried about a recently released API, fabricated details of a non-existent API that aligned with the description. All Hands AI is actively developing systems to detect and prevent these hallucinations, but a definitive solution remains elusive.

Benchmarking Progress

The SWE-Bench leaderboards provide a valuable metric for evaluating progress in agentic programming. Developers can test their models against a collection of unresolved issues from open GitHub repositories. OpenHands currently leads the verified leaderboard, successfully resolving 65.8% of the problems.

OpenAI asserts that codex-1, a model powering Codex, achieves a higher score of 72.1%, as announced in their release – however, this score has not been independently verified and came with certain qualifications.

Benchmark Scores vs. Real-World Application

A key concern within the tech industry is that high benchmark scores do not necessarily equate to truly hands-off agentic coding. If agentic coders can only resolve three out of four problems, substantial human oversight will be required, especially when dealing with complex, multi-stage systems.

The Future of Agentic Coding

As with most AI tools, continuous improvements to foundation models are anticipated, ultimately enabling agentic coding systems to evolve into dependable developer tools. However, effectively managing hallucinations and other reliability concerns will be critical to achieving this goal.

Brennan concludes, “There’s a trust barrier to overcome. The question is, how much responsibility can we delegate to these agents, ultimately reducing the workload for developers?”