commit 634c8e8ba55675b9d7d593ab720696894bbc2b01 Author: renolation Date: Wed Jul 8 14:58:58 2026 +0700 first commit diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..53c26aa --- /dev/null +++ b/.gitignore @@ -0,0 +1,5 @@ +# secret thô — KHÔNG commit (chỉ commit bản đã seal) +*-secret-raw.yaml +*.raw.yaml +.DS_Store +images/spark-cdc/cdc_crm_to_raw.py diff --git a/.woodpecker/build-spark-cdc.yaml b/.woodpecker/build-spark-cdc.yaml new file mode 100644 index 0000000..a6c2e49 --- /dev/null +++ b/.woodpecker/build-spark-cdc.yaml @@ -0,0 +1,23 @@ +# Woodpecker CI — build image spark-cdc, push lên container registry. +# Registry: đặt theo hạ tầng hiện tại; đích cuối là Gitea registry (git.dbiz.com) khi sẵn sàng. +# Secret khai trong Woodpecker UI: registry_user, registry_token. +when: + - event: push + branch: main + path: "images/spark-cdc/**" + +steps: + build-push: + image: woodpeckerci/plugin-docker-buildx + settings: + registry: ${REGISTRY_HOST} # vd git.dbiz.com (Gitea) hoặc registry hiện tại + repo: ${REGISTRY_HOST}/dbiz/spark-cdc + dockerfile: images/spark-cdc/Dockerfile + context: images/spark-cdc + tags: + - latest + - ${CI_COMMIT_SHA:0:8} + username: + from_secret: registry_user + password: + from_secret: registry_token diff --git a/README.md b/README.md new file mode 100644 index 0000000..36d9fd0 --- /dev/null +++ b/README.md @@ -0,0 +1,65 @@ +# dbiz-lake-k8s — DBIZ Lakehouse GitOps + +Quản deploy lakehouse lên K8s qua ArgoCD. Theo khuôn `argo-k8s` (App-of-Apps, Helm + YAML thuần). +**Công cụ: Helm + ApplicationSet. KHÔNG Kustomize.** + +## Git/CI — URL là biến số (không chờ hạ tầng) +- **``** trong các Application = placeholder repo GitOps. Chạy được với repo BẤT KỲ đang dùng. + Đổi repo (vd sang Gitea `git.dbiz.com` sau này) = một lệnh: `grep -rl '' | xargs sed -i 's###g'`, + rồi cập nhật ArgoCD repo creds. Không phải chờ Gitea. +- **``** trong values = registry chứa image. Đổi tương tự khi chuyển registry. +- **CI: Woodpecker** (đã có) build + push image. File: `.woodpecker/`. Secret registry khai trong Woodpecker UI. +- Đích cuối: code + registry ở Gitea `git.dbiz.com` (built-in registry) — nhưng KHÔNG chặn việc chạy hôm nay. + +## Triết lý +- **Lakehouse as a product**: DBIZ lake = "khách số 0". Mọi thứ THAM SỐ HOÁ (không hardcode) + để sau nhân bản cho khách mua gói (Vietbank/Gemadept) qua **ApplicationSet** — CHƯA làm, hoãn tới khi có khách thứ 2. +- **Stateful đứng yên, compute di**: MinIO (data lake) GIỮ NGUYÊN ngoài K8s, trỏ qua Service+Endpoints. + Kafka dựng mới (Strimzi) trên K8s — chấp nhận snapshot CDC lại. +- **dev giống prod**: cùng chart, khác values theo env (dev/prod). Helm values-driven, portable. +- **Portable cho on-prem**: chart nhận mọi dependency (S3/nguồn/kafka) qua values → chạy được cả trên + hạ tầng DBIZ lẫn on-prem tại khách (air-gapped: mirror image sang registry nội bộ khách). + +## Cấu trúc +``` +charts/lake-cdc-job/ Chart CDC job (SparkApplication) — THAM SỐ HOÁ theo customer +envs/dev/ + apps/ ArgoCD Applications (App-of-Apps) + root.yaml Root -> quét apps/ + infra.yaml External Service+Endpoints (MinIO ngoài, Postgres nguồn) + strimzi-operator.yaml Strimzi (Helm upstream) + spark-operator.yaml Spark Operator (Helm upstream) + kafka-cluster.yaml -> envs/dev/kafka (Kafka CRD) + cdc-job.yaml CDC job (chart lake-cdc-job + values, multi-source) + kafka/ Kafka CRD (Strimzi, KRaft, Longhorn PV) + infra/ external-services.yaml (IP hạ tầng ngoài) + values/ values per-app (cdc-job.yaml = khách số 0 dbiz) + sealed-secrets/ SealedSecrets (imagePullSecret, creds) — an toàn commit +images/spark-cdc/ Dockerfile (image bundled) +.woodpecker/ Woodpecker CI (build + push image lên Gitea registry) +``` + +## Điều kiện tiên quyết trên cụm (Phase 0 — dựng TRƯỚC) +1. ArgoCD (đã có). +2. **Longhorn** (StorageClass `longhorn`) — cho Kafka PV + checkpoint. +3. **Sealed Secrets controller**. +4. **cert-manager** + ingress (nếu expose UI). +5. **Gitea `git.dbiz.com`** online + repo này + **Deploy Token** (đọc repo cho ArgoCD, đọc registry cho pull image). +6. **Woodpecker** nối Gitea, khai secret `gitea_registry_user` / `gitea_registry_token`. + +## Bootstrap (khi Gitea + nền sẵn sàng) +1. Điền IP thật vào `envs/dev/infra/external-services.yaml` (MinIO, Postgres nguồn). +2. Seal imagePullSecret (Gitea registry, tên `gitea-registry`) -> `envs/dev/sealed-secrets/`. +3. Push `images/spark-cdc/**` -> Woodpecker tự build + push image lên `git.dbiz.com/dbiz/spark-cdc`. +4. `kubectl apply -f envs/dev/apps/root.yaml` -> ArgoCD sync tất cả. + +## Nhân bản cho khách mới (TƯƠNG LAI) +- Copy `envs/dev/values/cdc-job.yaml`, đổi `customer`, nguồn, bucket, namespace, s3.endpoint. +- Nhiều khách -> **ApplicationSet** (generator theo customer). +- Mức cách ly (1/2/3) = preset values + namespace/cụm riêng. Vietbank on-prem = cùng chart, cụm khác. +- S3: DBIZ dựng (ngoài/trong cụm) hoặc khách tự cấp — chỉ đổi `s3.endpoint` + secret, không sửa chart. + +## LƯU Ý +- CDC Docker cũ GIỮ CHẠY tới khi CDC K8s verify xong, rồi mới tắt (không đập cái đang chạy). +- Version Helm chart (Strimzi/Spark Operator) đang pin TẠM — kiểm bản mới nhất khi dựng thật. +- 5 VM/1 host Proxmox: HA storage là ảo giác tới khi trải 2 host. Backup (Velero/pg_dump/mc mirror) là phòng thủ thật. diff --git a/charts/lake-cdc-job/Chart.yaml b/charts/lake-cdc-job/Chart.yaml new file mode 100644 index 0000000..201e5c0 --- /dev/null +++ b/charts/lake-cdc-job/Chart.yaml @@ -0,0 +1,6 @@ +apiVersion: v2 +name: lake-cdc-job +description: DBIZ Lakehouse CDC streaming job (SparkApplication) — tham số hoá theo khách hàng +type: application +version: 0.1.0 +appVersion: "1.0" diff --git a/charts/lake-cdc-job/templates/checkpoint-pvc.yaml b/charts/lake-cdc-job/templates/checkpoint-pvc.yaml new file mode 100644 index 0000000..326a434 --- /dev/null +++ b/charts/lake-cdc-job/templates/checkpoint-pvc.yaml @@ -0,0 +1,11 @@ +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: cdc-{{ .Values.customer }}-checkpoint + namespace: {{ .Release.Namespace }} +spec: + accessModes: [ ReadWriteOnce ] + storageClassName: {{ .Values.checkpoint.storageClass }} + resources: + requests: + storage: {{ .Values.checkpoint.size }} diff --git a/charts/lake-cdc-job/templates/sparkapplication.yaml b/charts/lake-cdc-job/templates/sparkapplication.yaml new file mode 100644 index 0000000..33015c0 --- /dev/null +++ b/charts/lake-cdc-job/templates/sparkapplication.yaml @@ -0,0 +1,68 @@ +apiVersion: sparkoperator.k8s.io/v1beta2 +kind: SparkApplication +metadata: + name: cdc-{{ .Values.customer }} + namespace: {{ .Release.Namespace }} +spec: + type: Python + mode: cluster + image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" + imagePullPolicy: {{ .Values.image.pullPolicy }} + imagePullSecrets: + - {{ .Values.imagePullSecret }} + mainApplicationFile: "local:///opt/spark/work-dir/cdc_crm_to_raw.py" + sparkVersion: "{{ .Values.spark.version }}" + # streaming job chạy mãi -> restart nếu chết (thay cho việc submit tay foreground) + restartPolicy: + type: Always + onFailureRetryInterval: 30 + onSubmissionFailureRetryInterval: 30 + sparkConf: + # Iceberg catalog (Polaris REST) + MinIO + "spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions" + "spark.sql.catalog.iceberg": "org.apache.iceberg.spark.SparkCatalog" + "spark.sql.catalog.iceberg.type": "rest" + "spark.sql.catalog.iceberg.uri": "{{ .Values.catalog.uri }}" + "spark.sql.catalog.iceberg.warehouse": "{{ .Values.catalog.warehouse }}" + "spark.sql.catalog.iceberg.header.X-Iceberg-Access-Delegation": "vended-credentials" + "spark.sql.catalog.iceberg.io-impl": "org.apache.iceberg.aws.s3.S3FileIO" + "spark.sql.catalog.iceberg.s3.endpoint": "{{ .Values.s3.endpoint }}" + "spark.sql.catalog.iceberg.s3.path-style-access": "true" + "spark.sql.catalog.iceberg.s3.region": "{{ .Values.s3.region }}" + "spark.sql.defaultCatalog": "iceberg" + driver: + cores: {{ .Values.spark.driverCores }} + memory: "{{ .Values.spark.driverMemory }}" + labels: { app: cdc-{{ .Values.customer }} } + env: + - name: KAFKA_BOOTSTRAP + value: "{{ .Values.kafka.bootstrap }}" + - name: TOPIC_PATTERN + value: "{{ .Values.topicPattern }}" + - name: TARGET_TABLE + value: "{{ .Values.targetTable }}" + - name: TRIGGER_INTERVAL + value: "{{ .Values.triggerInterval }}" + - name: MAX_OFFSETS_PER_TRIGGER + value: "{{ .Values.maxOffsetsPerTrigger }}" + - name: CHECKPOINT + value: "file:///checkpoints/cdc_{{ .Values.customer }}" + volumeMounts: + - name: checkpoint + mountPath: /checkpoints + executor: + instances: {{ .Values.spark.executorInstances }} + cores: {{ .Values.spark.executorCores }} + memory: "{{ .Values.spark.executorMemory }}" + volumeMounts: + - name: checkpoint + mountPath: /checkpoints + volumes: + - name: checkpoint + persistentVolumeClaim: + claimName: cdc-{{ .Values.customer }}-checkpoint + deps: + packages: + - org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.9.1 + - org.apache.iceberg:iceberg-aws-bundle:1.9.1 + - org.apache.spark:spark-sql-kafka-0-10_2.12:3.5.6 diff --git a/charts/lake-cdc-job/values.yaml b/charts/lake-cdc-job/values.yaml new file mode 100644 index 0000000..dbdd572 --- /dev/null +++ b/charts/lake-cdc-job/values.yaml @@ -0,0 +1,36 @@ +# ====== THAM SỐ HOÁ THEO KHÁCH HÀNG (customer = "khách số 0" dbiz) ====== +customer: dbiz # định danh khách; đổi khi nhân bản (vietbank/gemadept) +image: + repository: "" # bắt buộc override: /spark-cdc + tag: "latest" + pullPolicy: Always +imagePullSecret: gitea-registry # SealedSecret imagePullSecret + +spark: + version: "3.5.6" + driverCores: 1 + driverMemory: "1g" + executorInstances: 1 + executorCores: 2 + executorMemory: "2g" + +# nguồn Kafka (Strimzi trong cụm) + đích Iceberg +kafka: + bootstrap: "lake-kafka-kafka-bootstrap:9092" # service Strimzi sinh ra +topicPattern: "crm\\.public\\..*" +targetTable: "iceberg.raw_crm.cdc_events" +triggerInterval: "30 seconds" +maxOffsetsPerTrigger: "50000" + +# checkpoint: PVC Longhorn (bền qua restart) — mỗi khách 1 PVC riêng +checkpoint: + storageClass: longhorn + size: 5Gi + +# catalog (Polaris) + MinIO endpoint — trỏ hạ tầng lake +catalog: + uri: "http://polaris:8181/api/catalog" # điều chỉnh theo Polaris trong cụm + warehouse: "dbiz_warehouse" +s3: + endpoint: "http://minio-lake:9000" # external-service trỏ MinIO ngoài + region: "us-east-1" diff --git a/envs/dev/apps/cdc-job.yaml b/envs/dev/apps/cdc-job.yaml new file mode 100644 index 0000000..2c7cef6 --- /dev/null +++ b/envs/dev/apps/cdc-job.yaml @@ -0,0 +1,22 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: lake-cdc-job-dev + namespace: argocd +spec: + project: default + sources: + - repoURL: + targetRevision: main + path: charts/lake-cdc-job + helm: + valueFiles: + - $values/envs/dev/values/cdc-job.yaml + - repoURL: + targetRevision: main + ref: values + destination: + server: https://kubernetes.default.svc + namespace: dbiz-lake-dev + syncPolicy: + automated: { prune: true, selfHeal: true } diff --git a/envs/dev/apps/infra.yaml b/envs/dev/apps/infra.yaml new file mode 100644 index 0000000..1070510 --- /dev/null +++ b/envs/dev/apps/infra.yaml @@ -0,0 +1,22 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: lake-infra-dev + namespace: argocd +spec: + project: default + source: + repoURL: + targetRevision: main + path: envs/dev/infra + directory: + recurse: true + destination: + server: https://kubernetes.default.svc + namespace: dbiz-lake-dev + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true diff --git a/envs/dev/apps/kafka-cluster.yaml b/envs/dev/apps/kafka-cluster.yaml new file mode 100644 index 0000000..9eba1f5 --- /dev/null +++ b/envs/dev/apps/kafka-cluster.yaml @@ -0,0 +1,17 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: lake-kafka-dev + namespace: argocd +spec: + project: default + source: + repoURL: + targetRevision: main + path: envs/dev/kafka + directory: { recurse: true } + destination: + server: https://kubernetes.default.svc + namespace: dbiz-lake-dev + syncPolicy: + automated: { prune: true, selfHeal: true } diff --git a/envs/dev/apps/root.yaml b/envs/dev/apps/root.yaml new file mode 100644 index 0000000..f260d87 --- /dev/null +++ b/envs/dev/apps/root.yaml @@ -0,0 +1,20 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: lake-root-dev + namespace: argocd +spec: + project: default + source: + repoURL: + targetRevision: main + path: envs/dev/apps + destination: + server: https://kubernetes.default.svc + namespace: argocd + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true diff --git a/envs/dev/apps/spark-operator.yaml b/envs/dev/apps/spark-operator.yaml new file mode 100644 index 0000000..bd5faf1 --- /dev/null +++ b/envs/dev/apps/spark-operator.yaml @@ -0,0 +1,25 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: lake-spark-operator-dev + namespace: argocd +spec: + project: default + source: + repoURL: https://kubeflow.github.io/spark-operator + chart: spark-operator + targetRevision: 2.1.1 # pin (kiểm bản mới khi dựng) + helm: + releaseName: spark-operator + values: | + spark: + jobNamespaces: + - dbiz-lake-dev + webhook: + enable: true + destination: + server: https://kubernetes.default.svc + namespace: dbiz-lake-dev + syncPolicy: + automated: { prune: true, selfHeal: true } + syncOptions: [ CreateNamespace=true ] diff --git a/envs/dev/apps/strimzi-operator.yaml b/envs/dev/apps/strimzi-operator.yaml new file mode 100644 index 0000000..6439265 --- /dev/null +++ b/envs/dev/apps/strimzi-operator.yaml @@ -0,0 +1,19 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: lake-strimzi-operator-dev + namespace: argocd +spec: + project: default + source: + repoURL: https://strimzi.io/charts/ + chart: strimzi-kafka-operator + targetRevision: 0.45.0 # pin version (kiểm bản mới nhất khi dựng thật) + helm: + releaseName: strimzi + destination: + server: https://kubernetes.default.svc + namespace: dbiz-lake-dev + syncPolicy: + automated: { prune: true, selfHeal: true } + syncOptions: [ CreateNamespace=true ] diff --git a/envs/dev/infra/external-services.yaml b/envs/dev/infra/external-services.yaml new file mode 100644 index 0000000..0e5bbd4 --- /dev/null +++ b/envs/dev/infra/external-services.yaml @@ -0,0 +1,50 @@ +# Service + Endpoints không selector -> trỏ tới hạ tầng NGOÀI K8s (theo pattern argo-k8s). +# MinIO lake: GIỮ NGUYÊN, K8s compute trỏ vào đây như S3 endpoint. +# ĐIỀN IP THẬT của máy chạy MinIO (VM lake) trước khi apply. +--- +apiVersion: v1 +kind: Service +metadata: + name: minio-lake + namespace: dbiz-lake-dev +spec: + ports: + - name: s3 + port: 9000 + targetPort: 9000 +--- +apiVersion: v1 +kind: Endpoints +metadata: + name: minio-lake + namespace: dbiz-lake-dev +subsets: + - addresses: + - ip: 192.168.110.37 # <-- ĐIỀN IP máy MinIO (VM lake) + ports: + - name: s3 + port: 9000 +--- +# Postgres nguồn CRM (172.20.109.5) — để Debezium/Strimzi trỏ tới khi CDC +apiVersion: v1 +kind: Service +metadata: + name: crm-source-pg + namespace: dbiz-lake-dev +spec: + ports: + - name: pg + port: 5432 + targetPort: 5432 +--- +apiVersion: v1 +kind: Endpoints +metadata: + name: crm-source-pg + namespace: dbiz-lake-dev +subsets: + - addresses: + - ip: 172.20.109.5 + ports: + - name: pg + port: 5432 diff --git a/envs/dev/kafka/kafka.yaml b/envs/dev/kafka/kafka.yaml new file mode 100644 index 0000000..a887e3b --- /dev/null +++ b/envs/dev/kafka/kafka.yaml @@ -0,0 +1,45 @@ +# Strimzi Kafka (KRaft mode, không Zookeeper). PV qua Longhorn. +# Dev: 1 broker; prod: 3 broker (đổi replicas + storage). +apiVersion: kafka.strimzi.io/v1beta2 +kind: KafkaNodePool +metadata: + name: dual-role + namespace: dbiz-lake-dev + labels: + strimzi.io/cluster: lake-kafka +spec: + replicas: 1 + roles: [ controller, broker ] + storage: + type: persistent-claim + size: 20Gi + class: longhorn + deleteClaim: false +--- +apiVersion: kafka.strimzi.io/v1beta2 +kind: Kafka +metadata: + name: lake-kafka + namespace: dbiz-lake-dev + annotations: + strimzi.io/node-pools: enabled + strimzi.io/kraft: enabled +spec: + kafka: + version: 3.9.0 + listeners: + - name: plain + port: 9092 + type: internal + tls: false + config: + # message to (giải bài RecordTooLarge đã gặp): 10MB + message.max.bytes: 10485760 + offsets.topic.replication.factor: 1 + transaction.state.log.replication.factor: 1 + transaction.state.log.min.isr: 1 + default.replication.factor: 1 + min.insync.replicas: 1 + entityOperator: + topicOperator: {} + userOperator: {} diff --git a/envs/dev/sealed-secrets/.gitkeep b/envs/dev/sealed-secrets/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/envs/dev/values/cdc-job.yaml b/envs/dev/values/cdc-job.yaml new file mode 100644 index 0000000..cf660a0 --- /dev/null +++ b/envs/dev/values/cdc-job.yaml @@ -0,0 +1,23 @@ +# DBIZ lake — "khách số 0". Nhân bản: copy file này, đổi customer + nguồn + bucket. +customer: dbiz +image: + repository: git.dbiz.com/dbiz/spark-cdc # <-- Gitea registry built-in (git.dbiz.com/dbiz/spark-cdc) + tag: "latest" +imagePullSecret: gitea-registry + +kafka: + bootstrap: "lake-kafka-kafka-bootstrap:9092" +topicPattern: "crm\\.public\\..*" +targetTable: "iceberg.raw_crm.cdc_events" +triggerInterval: "30 seconds" + +catalog: + uri: "http://polaris:8181/api/catalog" + warehouse: "dbiz_warehouse" +s3: + endpoint: "http://minio-lake:9000" + region: "us-east-1" + +checkpoint: + storageClass: longhorn + size: 5Gi diff --git a/envs/prod/sealed-secrets/.gitkeep b/envs/prod/sealed-secrets/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/images/spark-cdc/Dockerfile b/images/spark-cdc/Dockerfile new file mode 100644 index 0000000..1692db0 --- /dev/null +++ b/images/spark-cdc/Dockerfile @@ -0,0 +1,14 @@ +# Image CDC job: Spark + jar Iceberg/Kafka bundled + script. Reproducible, khởi động nhanh. +FROM spark:3.5.6 + +USER root +ARG IVY_PKGS="org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.9.1,org.apache.iceberg:iceberg-aws-bundle:1.9.1,org.apache.spark:spark-sql-kafka-0-10_2.12:3.5.6" + +# Tải sẵn các jar vào /opt/spark/jars (không tải runtime mỗi lần pod start) +RUN /opt/spark/bin/spark-shell --packages "${IVY_PKGS}" --conf spark.jars.ivy=/tmp/.ivy <<< 'System.exit(0)' || true \ + && find /root/.ivy2 -name '*.jar' -exec cp {} /opt/spark/jars/ \; 2>/dev/null || true + +# Script CDC (copy vào work-dir) +COPY cdc_crm_to_raw.py /opt/spark/work-dir/cdc_crm_to_raw.py + +USER 185