34 lines
1.1 KiB
Markdown
34 lines
1.1 KiB
Markdown
# CDP Ingestion
|
|
|
|
Self-hosted CDP ingestion platform inspired by Jitsu. Segment-compatible HTTP API.
|
|
|
|
## Services
|
|
|
|
| Service | Lang | Port | Role |
|
|
|---------|------|------|------|
|
|
| `ingest` | Go | 3049 | HTTP API → auth, validate, dedup, push to Kafka |
|
|
| `rotor` | Node.js | 3401 | JS functions runner (V8 isolates) |
|
|
| `bulker` | Go | 3042 | Kafka consumer → batch write ClickHouse / warehouses |
|
|
| `console` | React + Vite | 3000 | Management UI |
|
|
|
|
## Quick start
|
|
|
|
```bash
|
|
make up # docker-compose up infra (Postgres, Redis, Kafka, ClickHouse)
|
|
make migrate/up # apply PostgreSQL migrations
|
|
make clickhouse/up # apply ClickHouse DDL
|
|
make run/ingest # start ingest on :3049
|
|
make run/bulker # start bulker on :3042
|
|
make run/rotor # start rotor on :3401
|
|
make run/console # start console on :3000
|
|
```
|
|
|
|
## Testing
|
|
|
|
```bash
|
|
make test # unit tests (no containers)
|
|
make test/integration # repo-layer integration tests (testcontainers)
|
|
```
|
|
|
|
See [CLAUDE_ingestion.md](./CLAUDE_ingestion.md) for the full design contract.
|