Expensify Tech Stack: How They Scaled & Innovated

The Driving Force Behind Expensify's Success: Tenacity and Unconventional Choices

A defining characteristic of many thriving startups is a resolute spirit. The unwavering determination to realize a company’s vision, coupled with a tenacious mindset, frequently serves as the primary catalyst for achievement, and this quality is often mirrored in the individuals who comprise these organizations.

Therefore, it is reasonable to suggest that the team at Expensify possesses a considerable degree of steadfastness – a trait that has ultimately proven advantageous. This collective of individuals, initially operating with a peer-to-peer, innovative approach to building an expense management application, remained true to their instincts and established their own operational guidelines.

Challenging Conventional Wisdom

They posed inquiries that others overlooked, such as: Is a large workforce truly necessary when efficient work and substantial profitability can be achieved with a smaller team? Why limit operations to a San Francisco office when remote work, even from locations like a Caribbean sailboat, is viable thanks to the internet?

This approach aligns with a rebellious ethos; if one adopts a pirate mentality, conventional rules become less relevant. This is particularly true when the rationale behind those rules is unclear or absent.

Building a Custom Tech Stack

Following this line of thought, Expensify questioned the need to conform to industry standards and decided to develop its own bespoke technology infrastructure. Several of their technical decisions have been met with skepticism – including utilizing an open-source frontend, employing cross-platform mobile development, and leveraging contractors for AI training, as well as engaging open-source contributors – but these choices have consistently yielded positive results.

The company is now poised for an Initial Public Offering (IPO), demonstrating the effectiveness of its unconventional path.

Understanding Expensify's Roadmap

The extent of the technological advantage Expensify has gained through these decisions remains a subject of discussion. However, it is undeniable that these choices are fundamental to comprehending the company’s identity and future direction. A closer examination of their approach is warranted.

Built on Bedrock

Expensify, in its formative years, posed a key question: could their database function optimally when built upon a technology initially designed for smaller applications?

Surprisingly, Expensify operates using a custom database constructed on SQLite. This is notable because, while SQLite is extensively used, it’s typically associated with smaller, embedded systems like smartphones and browsers, rather than large-scale enterprise databases.

This unique database is named Bedrock, and its architecture is distinctly unconventional. Expensify defines it as a “relational database management system (RDBMS) optimized for self-healing replication across slower, less reliable wide area networks (WANs), enabling high availability and performance in multi-datacenter setups without a single point of failure.” An RDBMS is a type of database where data is organized into interconnected rows.

The central question arises: why did Expensify choose this path instead of utilizing readily available enterprise database solutions?

To understand this decision, we must revisit the company’s origins as a side project for founder and CEO, David Barrett. His initial concept involved a prepaid card for individuals experiencing homelessness, which necessitated a server connection to the Visa network, presenting significant requirements and obstacles. Barrett stated in a TechCrunch interview that a crucial need was “the ability to automatically replicate and failover.”

Achieving this in 2007 proved challenging, but Barrett embraced it. He noted that existing off-the-shelf technology wasn’t sufficiently advanced. Having a background as a peer-to-peer software developer with available time, he was motivated to build a system for the Visa backend.

His prior experience in peer-to-peer networking proved invaluable. Expensify’s first hires – P2P engineers from Red Swoosh and Akamai – were exceptionally well-suited for the task.

From this initial work, Bedrock emerged. This relational database is most effective when distributed across three data centers, with redundancy within each. Barrett explained on FLOSS Weekly that server failures trigger automatic failover, and recovery is streamlined due to a distributed consensus algorithm that efficiently identifies the new leader and followers.

This explains why Expensify characterizes Bedrock as being “blockchain-inspired” before the term gained widespread recognition. Some questioned the accuracy of this claim, but Expensify software engineer Tyler Karaszewski clarified to TechCrunch that it depends on the definition of “blockchain.” The company highlights its distributed general ledger and “Paxos-style algorithm,” which ensure data integrity and represent a private blockchain component.

Notably, Expensify accomplished this even when SQLite’s developers doubted its feasibility.

SQLite: Achieving Scalability Beyond Initial Design

Initially, there was some skepticism, but a story now shared with amusement involves Barrett explaining to Richard Hipp, SQLite’s creator, in 2012 how Expensify was utilizing SQLite for a database that had already grown to 40GB. Hipp attempted to dissuade him, expressing concerns about exceeding the engine’s intended capabilities. He communicated via the SQLite mailing list that SQLite wasn't created to function as a substitute for a robust, enterprise-level RDBMS, suggesting Expensify likely required such a system.

However, the fact that SQLite wasn’t originally conceived for Barrett’s specific application didn't preclude its ability to perform. It simply necessitated a dedicated effort to realize that potential. Hipp later remarked to TechCrunch that, in retrospect, it was logical, but the team was so immersed in their work that they hadn’t considered the possibility.

Hipp draws a parallel to a quote by George Bernard Shaw when discussing Barrett’s approach: “The reasonable man adapts himself to the world: The unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man.” He believes this perfectly encapsulates Barrett’s spirit, stating that individuals like David are essential for driving innovation. Barrett reciprocates this admiration, repeatedly acknowledging the SQLite team as being among the most skilled engineers globally.

Expensify has also consistently been a significant sponsor of the SQLite Consortium. Following Barrett’s advocacy, Hipp and his team have proactively provided support to the company. Karaszewski emphasized that the current iteration of Bedrock wouldn’t have been achievable without the SQLite team’s contributions to their product, enabling its integration into Expensify’s systems.

Specifically, the SQLite team implemented enhancements tailored for Bedrock, deploying them on a distinct branch due to the minimal performance impact of these additions. Hipp clarified via email that while a 1% increase in CPU and memory usage may seem insignificant, it can accumulate substantially when multiplied across billions of devices.

While Expensify’s utilization of SQLite remains unconventional, Hipp expresses hope that others will adopt a similar approach. He believes that wider adoption of Bedrock, or architectures modeled after it, would significantly reduce the risk of SQL injection attacks – a primary security concern. He further stated that the team is impressed by Barrett’s accomplishments.

Surprisingly, Expensify’s distinctive data architecture has proven resilient over time, withstanding both public scrutiny and the demands of the company’s growth. It has potentially also been a contributing factor to the company’s overall success.

A Unique Approach to Database Technology

Expensify’s 2016 decision to make Bedrock open-source was an unusual one, driven by a specific goal. Barrett, speaking to TechCrunch, clarified that the primary intention wasn’t to attract external contributors, but rather to subject their system to rigorous external review and uphold a higher standard of quality.

This strategy proved successful, as the company benefited from contributions, improved its internal benchmarks, and continued to expand its operations. Ultimately, the process strengthened Expensify’s confidence in Bedrock, which Barrett identifies as the company’s “most differentiated technology,” and a source of “many key competitive advantages,” as noted by First Round Review in 2016.

Bedrock's Advantages

While acknowledging the advancements in MySQL since Bedrock’s creation, Barrett maintains that Bedrock possesses a distinct advantage due to its streamlined design. The system now offers MySQL protocol support, enabling compatibility with tools like Sequel Pro and Mode Analytics.

According to the project’s website, Bedrock is engineered for contemporary hardware, specifically utilizing large SSD-backed RAID drives and substantial RAM file caches. This allows it to avoid the complex optimizations required by older databases to achieve high performance on outdated systems.

Hardware and Performance Considerations

Expensify’s testing led them to conclude that bare metal infrastructure provides the optimal performance. Barrett expressed skepticism regarding common perceptions of AWS, stating that “most of what people say about it is just objectively wrong.”

He pointed out that directly purchasing hardware is often more cost-effective than pre-paying for server capacity over an extended period. Furthermore, he emphasized that the hardware available through AWS differs from what one could procure independently, noting discrepancies between hyperthreading and true core counts, and the importance of customized system settings for peak performance.

Scale and Data Management

Bedrock is optimized for systems approaching “supercomputer class computers,” as Barrett describes. This optimization isn’t solely for scalability; it also stems from the fact that all of Expensify’s data resides within a single database – a surprisingly uncommon practice.

A Twitter user expressed their surprise, stating they had assumed Expensify utilized numerous smaller databases, one for each customer, rather than a single, massive database with 1TB of RAM.

Business Model Alignment

Barrett explains that Bedrock’s architecture directly supports Expensify’s business model. “By maintaining a single, unified database, we eliminate the need to segment customers into separate databases,” he stated.

This approach contrasts with the typical multi-tenant architecture employed by enterprise companies relying on a top-down sales strategy.

Facilitating Collaboration

Expensify prioritizes viral growth, enabling users to easily share expense reports with managers and invite colleagues, fostering interaction between companies. “Having everyone operate within the same system allows for a more organic collaboration between any two users,” Barrett elaborated.

The feasibility of replicating this user experience with alternative architectures remains a subject of discussion, but Bedrock undeniably enhances Expensify’s appeal to engineering talent. Jason Kruse, a former Expensify employee and current CTO at Lithic, described building their own database replication technology as “kind of crazy…and kind of amazing, frankly.”

Beyond Database Functionality

Kruse also highlighted Bedrock’s distributed general ledger and its ability to reliably record the sequence of events. This capability allows Bedrock to function as a job management and distribution platform, powering initiatives like Concierge, the customer support feature, and ExpensiWorks, an “in-house Mechanical Turk” leveraging artificial intelligence.

The Human Element in Artificial Intelligence at Expensify

Barrett openly shares his perspective on artificial intelligence, stating that a significant portion of AI development relies on the contributions of contractors. This viewpoint became prominent during a 2017 incident where Expensify inadvertently shared receipts originating from Amazon Mechanical Turk.

The situation, affecting “up to 208 receipts” during a beta phase, was identified as an error, leading the company to discontinue its use of Amazon’s platform. Currently, Expensify employs a direct human workforce to both enhance and refine its AI capabilities.

SmartScan: A Hybrid Approach

This core philosophy underpins SmartScan, Expensify’s receipt-scanning solution. AI is utilized to initially scan receipts and correct inaccuracies, but complex cases are then directed to human workers.

The company deliberately adopted this strategy early on, choosing to avoid relying solely on optical character recognition (OCR) for receipt processing. Barrett emphasizes that prioritizing complete accuracy over sheer speed was crucial. He reasoned that rapid, yet imperfect, results were ultimately unproductive, necessitating manual review and correction.

Barrett aimed to deliver a streamlined user experience, characterized as a “fire and forget” system. Users simply scan their receipts, conclude the process, and allow the system to handle the remaining steps automatically.

Following the OCR stage, human transcribers intervene when necessary. These individuals are carefully selected Expensify contractors and subcontractors, operating under non-disclosure agreements, rather than workers sourced through Mechanical Turk.

In response to the 2017 publicity, Expensify developed an internal platform, ExpensiWorks, to manage these tasks and personnel. This initiative was spearheaded by COO Anu Muralidharan.

“Our approach involved assembling existing tools rather than building from the ground up,” Muralidharan explained. “The platform was designed to integrate with Bedrock and leverage our existing expense infrastructure.”

Concierge: AI Training Through Customer Support

A similar synergy between human expertise and AI is central to Concierge, Expensify’s customer support feature. In this case, human agents play a more active role than simply intervening when AI falters.

“We consider our initial support responders as trainers, as their primary function is to educate the AI, with customer support as a valuable outcome,” Barrett stated. He highlights Concierge’s “completely transformative” impact on the company, attributing its revenue growth to this feature despite a stable headcount.

Concierge’s natural language processing capabilities include techniques like fuzzy matching, though detailed technical specifications are not publicly available. “Our focus is on delivering code and features, not publishing academic papers,” Barrett clarified.

This statement exemplifies Expensify’s pragmatic approach to technology and its real-world application of AI. Barrett noted in a 2017 interview that “Bedrock’s strength lies in its practical origins; it wasn’t developed as a theoretical solution but in response to genuine challenges.”

This grounded perspective also explains Expensify’s ability to simultaneously support both traditional flatbed scanners and innovative mobile expense reporting via smartphone cameras.

Maintaining Ubiquity: Expensify's Early Mobile Strategy

Expensify benefited significantly from its early adoption of mobile technology, though this initial success presented considerable challenges. Tom Jacobs, an early employee, remembers the difficulties of managing multiple platforms. He stated that maintaining separate codebases for Android, iPhone OS, Blackberry, and Palm – all sharing the same core business logic – proved incredibly demanding.

Weekly updates were routinely required for the Blackberry application, a situation Jacobs found unsustainable. To address this, Barrett proposed the development of an internal cross-platform JavaScript framework. This framework, named YAPL, which stood for either “Yet Another Platform Layer” or “Yet Another Platform Library,” represented a forward-thinking approach.

The Pioneering Nature of YAPL

At the time, alternative solutions like PhoneGap, Xamarin, and Electron were either nonexistent or in their nascent stages. YAPL distinguished itself by not only enabling code reuse across platforms but also by leveraging native event handling and controls for an optimal user experience.

The decision to pursue a cross-platform strategy, while difficult, proved advantageous. In 2012, smartphones possessed far less capability than their modern counterparts. A distinction existed between mobile phones and cameraphones, as not all devices featured functional cameras.

Furthermore, the absence of autofocus on iPhones until 2009 complicated predictions regarding the most important platform for Expensify’s receipt-scanning functionality. This uncertainty underscored the necessity of maintaining a presence on all platforms, despite the company’s limited resources.

Evolution to React Native

“The ability to write code once and deploy it across multiple platforms is the ultimate goal,” explained Puneet Lath, Director of R&D, in a recent discussion. While YAPL was an early innovator in this area, Expensify’s current investment in React Native represents a further advancement of this principle.

• Key takeaway: Early cross-platform development was crucial for Expensify's success.
• YAPL's role: It allowed Expensify to maintain a presence on multiple platforms with limited resources.
• Current strategy: Expensify continues to evolve its approach with React Native.

A Broad Overview

Expensify’s latest venture, Expensify.cash – a financial chat service operating on the principle that communication and payments are intrinsically linked – is remarkable for several reasons. Of particular interest is the fact that it represents a complete overhaul of the Expensify frontend, constructed using React Native and employing a unified codebase for iOS, Android, web, and desktop platforms.

A confluence of factors influenced Expensify’s commitment to React Native, notably the development of React Native for Web and, surprisingly, Airbnb’s decision to discontinue its use of the framework. Following an assessment of Airbnb’s experience, Expensify’s team determined that a gradual transition to React Native would likely yield suboptimal results. Consequently, they opted for a complete and immediate implementation.

Expensify.cash will be entirely open source, and its development is being undertaken with the assistance of freelance contractors sourced from Upwork. Addressing potential concerns regarding this outsourcing approach, the company clarified its reasoning:

A core tenet of Expensify’s organizational culture is a commitment to generalism, which facilitated the adoption of a React Native-centric strategy. Unlike many organizations, few within Expensify are likely to perceive Barrett’s assertion that “React Native for Web fundamentally alters the conventional engineering hierarchy” and associated specializations as a threat.

Another potentially unconventional aspect of Expensify.cash is the open-source nature of its frontend. Rather than being concerned about proprietary code, the company anticipates benefits such as enabling customers to personalize the frontend. This capability could address Expensify’s limited provision of customization services, a consequence of its “compounding product management” approach.

“Compounding” is a frequently referenced concept within Expensify. Barrett explained, “We operate under a compounding engineering philosophy, where every action focuses on enhancing existing work while simultaneously establishing a foundation for future development.” He further stated, “Each decision can be traced back to foundational technology and legal choices.”

This statement resonates with a degree of accuracy. Within Expensify’s account, the initial decisions made by Barrett and his colleagues serve as a consistent thread connecting their early concepts to the current operational practices of its workforce. This is evident in their problem-solving methodologies, their evaluation of opportunities, and their collaborative engagement with the open-source community on the technologies that underpin their operations, from Bedrock to Expensify.cash.

As will be explored in Part 5 of this EC-1 series, this long-term perspective occasionally brought Expensify into conflict with venture capitalists and presented numerous obstacles. However, the team’s resilience and confidence in its business model ultimately prevailed.

Expensify EC-1 Contents

This EC-1 was published in a series from early May to early June.

• Introduction
• Part 1: The Beginning
• Part 2: Company Culture
• Part 3: Growth and Remote Operations
• Part 4: Engineering and Technology
• Part 5: Business Strategy

Additional EC-1 reports are available on Extra Crunch.