init ingestion
This commit is contained in:
33
ingestion/README.md
Normal file
33
ingestion/README.md
Normal file
@@ -0,0 +1,33 @@
|
||||
# CDP Ingestion
|
||||
|
||||
Self-hosted CDP ingestion platform inspired by Jitsu. Segment-compatible HTTP API.
|
||||
|
||||
## Services
|
||||
|
||||
| Service | Lang | Port | Role |
|
||||
|---------|------|------|------|
|
||||
| `ingest` | Go | 3049 | HTTP API → auth, validate, dedup, push to Kafka |
|
||||
| `rotor` | Node.js | 3401 | JS functions runner (V8 isolates) |
|
||||
| `bulker` | Go | 3042 | Kafka consumer → batch write ClickHouse / warehouses |
|
||||
| `console` | React + Vite | 3000 | Management UI |
|
||||
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
make up # docker-compose up infra (Postgres, Redis, Kafka, ClickHouse)
|
||||
make migrate/up # apply PostgreSQL migrations
|
||||
make clickhouse/up # apply ClickHouse DDL
|
||||
make run/ingest # start ingest on :3049
|
||||
make run/bulker # start bulker on :3042
|
||||
make run/rotor # start rotor on :3401
|
||||
make run/console # start console on :3000
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
make test # unit tests (no containers)
|
||||
make test/integration # repo-layer integration tests (testcontainers)
|
||||
```
|
||||
|
||||
See [CLAUDE_ingestion.md](./CLAUDE_ingestion.md) for the full design contract.
|
||||
Reference in New Issue
Block a user