CockroachDB: The Resilient Database

The Convergence of Art and Engineering: The Genesis of CockroachDB

The intersection of engineering and artistic creation can yield remarkable results. This synergy was demonstrably evident in the case of Spencer Kimball and Peter Mattis, who, while students at Berkeley, developed the highly acclaimed open-source graphics application, GIMP.

The impact of their initial project was significant. Upon joining Google in 2002, Kimball and Mattis received a personal visit from Sergey Brin and Larry Page. The founders expressed their appreciation for GIMP, detailing its role in the creation of Google’s original logo.

A Trajectory of Advancement

Receiving such direct acknowledgement from Google’s leadership invariably sets a career on an upward course. Kimball and Mattis rapidly ascended within the company, becoming key figures on the Infrastructure Team.

A long-term, financially secure future at Google was readily attainable for them. However, they harbored aspirations beyond conventional employment.

The Founding of Cockroach Labs

Alongside fellow Google employee Ben Darnell, Kimball and Mattis sought to establish their own venture. This led to the formation of Cockroach Labs, and the development of their flagship open-source database, CockroachDB.

The question now arises: can these exceptionally skilled former Google engineers disrupt the established database landscape, a market historically characterized by failed ventures and unrealized potential?

Our investigation aims to determine whether CockroachDB possesses the capacity to redefine the future of data storage.

Berkeley Software Distribution

During the early to mid-1990s, Mattis and Kimball were computer science majors and roommates at Berkeley. Beyond their coursework, they actively participated in the eXperimental Computing Facility (XCF), a student organization characterized by a strong and dedicated interest in computer science.

The XCF Environment

The XCF operated through a project-based system. Members would select a project, then convene regularly to share progress and receive feedback. This environment was highly competitive. As Walter Rader noted in a 2003 publication, “Ideas that lacked merit were challenged, and ultimately, discarded – a process akin to Darwinian selection.”

Creation of GIMP

It was within this rigorous setting of the XCF that Mattis and Kimball developed GIMP in 1995. GIMP was conceived as an open-source alternative to Adobe Photoshop, a widely used photo editing application. Kimball later reflected on their collaboration, stating that GIMP represented his contribution to the free software community.

He felt a sense of obligation after benefiting from tools like emacs, gcc, and Linux, which significantly influenced his development as a computer scientist.

Parallels with CockroachDB

The development of GIMP foreshadowed their future work on CockroachDB. Kimball explained that both projects stemmed from a similar need. “We created GIMP because a sophisticated image manipulation program for Unix was unavailable, so we built one to address that gap.”

In 2015, when considering the creation of CockroachDB, they held a comparable perspective: a clear requirement existed, a suitable solution was lacking, and the potential impact could be substantial.

GIMP's Lasting Impact

GIMP ultimately achieved considerable success. Kimball recounted that the software continued to have a positive influence, even impacting the experiences of others. He described GIMP as “the gift that keeps on giving,” referencing encounters with figures like Larry and Sergey.

Early Scalability Challenges at Google

Mattis completed his studies at Berkeley in 1995, while Kimball had two further years of coursework remaining. They maintained contact and participated in various ventures during the peak of the dot-com boom. Subsequently, in 2002, Mattis secured a position at Google. He was highly satisfied with the company and successfully facilitated Kimball’s employment there three months later. Ben Darnell, the third co-founder of Cockroach Labs, also joined the search engine company around this period.

Google experienced rapid expansion and encountered difficulties in data management. The AdWords service, in particular, generated unprecedented volumes of data requiring storage and processing. While AdWords proved to be a significant revenue source, it simultaneously strained Google’s database systems.

Kimball directly observed these challenges. He stated, “Upon joining the AdWords team, I witnessed substantial database issues. They were employing a sharded MySQL architecture, which was struggling to cope with the demands.” He continued, “The system evolved from one shard to two, then four, eight, sixteen, and ultimately thirty-two. Google eventually discontinued this sharding approach, which is noteworthy given that many Silicon Valley startups at the time embraced it. Google instead committed to developing a superior solution.”

Their initial efforts, however, led to an unproductive path.

Google’s first attempt involved implementing a NoSQL solution. Unlike traditional SQL databases that organize data into distinct tables, NoSQL databases store data as collections of documents. A significant limitation of NoSQL databases at that time – including Google’s implementation – was the lack of support for transactions.

Transactions are crucial for reliable data modification. A common example is a travel booking involving airplane tickets, hotel reservations, and car rentals. Typically, data for these components resides in separate systems. Therefore, a successful booking requires the synchronized storage and updating of all three data elements.

As illustrated in Figure 1, transactions ensure data consistency. The process begins with purchasing the airplane ticket, followed by securing the hotel reservation, and concluding with the car rental. If all steps are completed successfully, the transaction is confirmed. Should an issue arise at any stage, such as a car rental database outage, the system will cancel the airplane ticket and hotel reservation, and report the transaction as failed.

The primary advantage of transactions is guaranteed data integrity at the storage level, allowing developers to concentrate on application functionality. Conversely, they can significantly reduce database performance due to potential table locking, restricting access for other users. With the rapid growth of internet users, supporting concurrent access across applications became paramount.

Despite this drawback, Google continued to develop its NoSQL technology. Kimball explained, “Feedback from Google’s application developers was consistent. They acknowledged the scalability benefits of NoSQL but lamented the absence of transactions.” The AdWords team specifically stated, “We manage 500 tables with complex relationships, and the sophistication of SQL databases is essential for replicating previous capabilities.”

Persistent concerns from developers prompted Google to create a relational database in 2006, designed to support transactions on a global scale. This technology, named Spanner, remains a fundamental component of Google Cloud today. Kimball noted, “They reintroduced SQL because they aimed to migrate AdWords to this system.” He added, “By the time I departed [Google], Spanner underpinned the majority of Google’s operations.”

This infrastructure debate within Google would eventually extend throughout the expanding internet ecosystem. Kimball remarked, “It felt like peering into the future, identifying the essential characteristics of a database that users would demand in 2020.” He continued, “I understood this in 2010, as Google was actively addressing this question, and I was present during that process.”

Ultimately, all three CockroachDB founders left Google. Darnell departed in 2009 to join FriendFeed, which was acquired by Facebook in 2009 and subsequently discontinued. Kimball and Mattis left Google in 2012 to establish Viewfinder. Darnell, then working at Dropbox, joined them at Viewfinder a few months later.

The Genesis of CockroachDB: From Viewfinder to Square and Beyond

The necessity for what would eventually become “CockroachDB” became strikingly clear to a team of engineers while working at Viewfinder, and wasn't limited to Google’s challenges. Viewfinder, a photo-sharing platform comparable to Flickr, aimed to deliver a superior experience to a global user base, potentially rivaling Snapchat or Instagram.

Providing immediate data access in diverse formats to users across the globe – from Australia to Europe and the United States – was a core requirement.

Initially, the startup utilized Amazon Web Service’s NoSQL database, DynamoDB. However, this choice presented ongoing difficulties. According to Kimball, approximately one-third of their engineering resources were dedicated to resolving database-related issues, a common occurrence in such scenarios.

Struggling to gain significant user adoption, Viewfinder was ultimately acquired by Square, a digital payments company, in 2013. Square, focused on worldwide financial transactions, sought the engineering expertise to tackle the complexities of global data management.

Square's Investment and the Open-Source Design

Square provided Kimball, Mattis, and Darnell with the opportunity to begin anew and explore innovative solutions. While at Square, they developed an open-source design document detailing the architecture of a relational database capable of supporting transactions on a global scale.

This document resonated with many individuals within the open-source community who had previously reviewed Google’s research paper on Spanner.

The paper also garnered interest from venture capitalists interested in investing in the database technology sector. Consequently, Cockroach Labs was founded in 2015.

However, an additional year of development was required before the product was prepared for a beta launch.

Designing Databases for Worldwide Operation

Inspired significantly by the Spanner paper from Google and their associated design documentation, the company’s three founders made a distinct choice regarding the technology’s licensing. In contrast to Spanner, CockroachDB utilizes an open-source model, and its code remains freely accessible on GitHub.

CockroachDB is engineered to empower developers in building databases that extend across multiple geographical areas. The technology centers around three core capabilities, which will be examined in greater detail in the subsequent section of this EC-1. Firstly, it facilitates the localization of data within specific regions. For instance, information pertaining to users in New York can be housed in an East Coast data center.

Secondly, the system guarantees data accuracy with every database query. Finally, it ensures continuous data availability, even in the event of a data center outage, through robust data replication mechanisms.

The combination of these three features positions this database as a potentially transformative force within the industry. The name “CockroachDB” is, in fact, a deliberate reflection of its inherent resilience. As Kimball explains, “It’s a testament to its durability; it’s virtually indestructible, much like a cockroach.” (The name is also quite memorable, at least within the realm of database nomenclature).

Initially, the revenue strategy involved offering support services to enterprise clients and charging licensing fees for advanced features. However, the business model has evolved over time. While enterprise licenses and support remain available, Cockroach Labs is now heavily focused on its database-as-a-service (DBaaS) offering, CockroachCloud, which will be explored further in part three.

Since its founding in 2012 by former Google employees, Cockroach Labs has experienced substantial growth. The company is headquartered in New York City, a location that defies the common perception that New York is primarily suited for fintech and advertising firms. The city provides access to the specific technical expertise the company requires.

Kimball notes that when recruiting developers in New York City compared to Silicon Valley, “The talent pool here is less focused on purely financial gain than in the Bay Area.” He also observes that the New York tech ecosystem has expanded at a rate comparable to the company’s own growth.

The company has reported nearly 300% year-over-year growth, with 2020 revenue doubling that of 2019, although specific figures were not disclosed. Currently, Cockroach Labs employs over 250 individuals and has secured over $350 million in funding from investors including Benchmark, GV, Index Ventures, and Redpoint, as reported by Crunchbase.

Significantly, the open-source codebase boasts contributions from more than 300 external developers. Founded by developers, Cockroach Labs remains committed to its origins, investing considerably in developer support and training initiatives.

Kimball, Mattis, and Darnell believe that the knowledge they acquired during their time at Google, along with their experiences at two subsequent startups, distinguishes CockroachDB from competing databases. They are not creating something entirely new, but rather refining a technology that originated as an idea at Google and has been progressively developed over the years.

Considering its history, technical expertise, the extensive developer community, and financial resources, CockroachDB – and, consequently, CockroachCloud – is poised to become a major player in the distributed database and database services landscape, both now and in the future.

However, a thorough understanding of that future necessitates a detailed examination of CockroachDB’s core product, which will be the focus of part two of this EC-1.

CockroachDB EC-1 Contents

• Introduction
• Part 1: The Company’s Origins
• Part 2: Technical Architecture
• Part 3: Developer Engagement and Business Strategy
• Part 4: Competitive Analysis and Future Outlook

Explore additional EC-1 articles on Extra Crunch.