init ingestion

This commit is contained in:
2026-05-24 22:59:24 +07:00
commit 4e8c11d545
80 changed files with 5639 additions and 0 deletions

33
ingestion/README.md Normal file
View File

@@ -0,0 +1,33 @@
# CDP Ingestion
Self-hosted CDP ingestion platform inspired by Jitsu. Segment-compatible HTTP API.
## Services
| Service | Lang | Port | Role |
|---------|------|------|------|
| `ingest` | Go | 3049 | HTTP API → auth, validate, dedup, push to Kafka |
| `rotor` | Node.js | 3401 | JS functions runner (V8 isolates) |
| `bulker` | Go | 3042 | Kafka consumer → batch write ClickHouse / warehouses |
| `console` | React + Vite | 3000 | Management UI |
## Quick start
```bash
make up # docker-compose up infra (Postgres, Redis, Kafka, ClickHouse)
make migrate/up # apply PostgreSQL migrations
make clickhouse/up # apply ClickHouse DDL
make run/ingest # start ingest on :3049
make run/bulker # start bulker on :3042
make run/rotor # start rotor on :3401
make run/console # start console on :3000
```
## Testing
```bash
make test # unit tests (no containers)
make test/integration # repo-layer integration tests (testcontainers)
```
See [CLAUDE_ingestion.md](./CLAUDE_ingestion.md) for the full design contract.