Airbyte

Description

Airbyte is an open-source data integration platform that moves data from APIs, databases, and files into warehouses, lakes, and AI applications. A large connector catalog and a framework for building custom connectors make it a flexible, self-hosted alternative to proprietary ETL/ELT services like Fivetran.

Features

• 600+ Connectors: Ships a large, community-maintained catalog of source and destination connectors.
• Connector Builder: Lets you build new connectors with a low-code UI or Python CDK.
• Incremental Sync: Syncs only new or changed records to keep pipelines efficient.
• Schema Change Detection: Flags upstream schema changes before they break downstream tables.
• Orchestration Integration: Triggers syncs from Airflow, Dagster, or Prefect via API.
• Self-Managed or Cloud: Runs fully self-hosted on Kubernetes/Docker, or as a managed cloud service.

Technology Stack

Platform built in Java and Kotlin, with connectors written in Python, Java, or low-code YAML via the Connector Development Kit. Deploys via Docker Compose or a Kubernetes Helm chart.

Requirements

Linux server with Docker or Kubernetes; at least 4 GB RAM for a small deployment; PostgreSQL for internal metadata storage; more resources needed as connector and sync volume grows.