cdp/ingestion

CDP Ingestion

Self-hosted CDP ingestion platform inspired by Jitsu. Segment-compatible HTTP API.

Services

Service	Lang	Port	Role
ingest	Go	3049	HTTP API → auth, validate, dedup, push to Kafka
rotor	Node.js	3401	JS functions runner (V8 isolates)
bulker	Go	3042	Kafka consumer → batch write ClickHouse / warehouses
console	React + Vite	3000	Management UI

Quick start

make up                    # docker-compose up infra (Postgres, Redis, Kafka, ClickHouse)
make migrate/up            # apply PostgreSQL migrations
make clickhouse/up         # apply ClickHouse DDL
make run/ingest            # start ingest on :3049
make run/bulker            # start bulker on :3042
make run/rotor             # start rotor on :3401
make run/console           # start console on :3000

Testing

make test                  # unit tests (no containers)
make test/integration      # repo-layer integration tests (testcontainers)

See CLAUDE_ingestion.md for the full design contract.