For organizations managing substantial volumes of information, it’s common to regularly transfer data between applications, APIs, and databases, ultimately storing it in a data warehouse or data lake. Several established platforms facilitate this extract, load, and transform (ELT) process, including solutions like Fivetran and Stitchdata.

Airbyte is developing an open-source data pipeline platform. The availability of an open-source option to tools such as Fivetran or Stitchdata is valuable because creating and supporting data connectors demands significant investment. This can be a substantial hurdle for organizations just beginning to establish their data pipelines, particularly those without large engineering teams.

While current data pipeline platforms offer integrations with numerous widely-used sources and destinations, these typically include popular options like Salesforce, Stripe, Marketo, and SendGrid. Common destinations for this data include Redshift, Snowflake, and BigQuery.

However, a large number of smaller, specialized services often lack support on these conventional data pipeline platforms. The inability to import all of a company’s data can result in an incomplete understanding of its overall business performance.

Many companies ultimately resort to developing custom connectors for their specific applications. Airbyte believes that an open-source model can encourage a collaborative community to create and maintain a vast library of open-source connectors. This allows companies to benefit from each other’s work; a connector built by one organization can be utilized, enhanced, and shared with the wider community.

“We believe the open-source methodology is the most effective path forward,” explained Michel Tricot, co-founder and CEO. “If a user identifies an issue with a connector, they have the ability to resolve it for the benefit of everyone.”

The Airbyte team initially focused on a product designed for marketing data. They participated in the Y Combinator program and subsequently adjusted their focus due to the impact of the coronavirus pandemic. “The shift in the economic climate brought about by COVID-19 led to reductions in marketing budgets,” noted co-founder John Lafleur.

Airbyte connectors operate within Docker containers, ensuring each connector functions independently. Users can schedule automated updates, manually refresh connectors, and monitor their status individually.

Currently, Airbyte offers 46 connectors and is in the process of certifying them to guarantee they meet production standards. The platform is presently being utilized by 250 companies.