first commit

2026-07-08 14:58:58 +07:00
commit 634c8e8ba5
19 changed files with 471 additions and 0 deletions
--- a/.gitignore
+++ b/.gitignore
@@ -0,0 +1,5 @@
+# secret thô — KHÔNG commit (chỉ commit bản đã seal)
+*-secret-raw.yaml
+*.raw.yaml
+.DS_Store
+images/spark-cdc/cdc_crm_to_raw.py
--- a/.woodpecker/build-spark-cdc.yaml
+++ b/.woodpecker/build-spark-cdc.yaml
@@ -0,0 +1,23 @@
+# Woodpecker CI — build image spark-cdc, push lên container registry.
+# Registry: đặt theo hạ tầng hiện tại; đích cuối là Gitea registry (git.dbiz.com) khi sẵn sàng.
+# Secret khai trong Woodpecker UI: registry_user, registry_token.
+when:
+  - event: push
+    branch: main
+    path: "images/spark-cdc/**"
+
+steps:
+  build-push:
+    image: woodpeckerci/plugin-docker-buildx
+    settings:
+      registry: ${REGISTRY_HOST}          # vd git.dbiz.com (Gitea) hoặc registry hiện tại
+      repo: ${REGISTRY_HOST}/dbiz/spark-cdc
+      dockerfile: images/spark-cdc/Dockerfile
+      context: images/spark-cdc
+      tags:
+        - latest
+        - ${CI_COMMIT_SHA:0:8}
+      username:
+        from_secret: registry_user
+      password:
+        from_secret: registry_token
--- a/README.md
+++ b/README.md
@@ -0,0 +1,65 @@
+# dbiz-lake-k8s — DBIZ Lakehouse GitOps
+
+Quản deploy lakehouse lên K8s qua ArgoCD. Theo khuôn `argo-k8s` (App-of-Apps, Helm + YAML thuần).
+**Công cụ: Helm + ApplicationSet. KHÔNG Kustomize.**
+
+## Git/CI — URL là biến số (không chờ hạ tầng)
+- **`<REPO_URL>`** trong các Application = placeholder repo GitOps. Chạy được với repo BẤT KỲ đang dùng.
+  Đổi repo (vd sang Gitea `git.dbiz.com` sau này) = một lệnh: `grep -rl '<REPO_URL>' | xargs sed -i 's#<REPO_URL>#<url-thật>#g'`,
+  rồi cập nhật ArgoCD repo creds. Không phải chờ Gitea.
+- **`<IMAGE_REGISTRY>`** trong values = registry chứa image. Đổi tương tự khi chuyển registry.
+- **CI: Woodpecker** (đã có) build + push image. File: `.woodpecker/`. Secret registry khai trong Woodpecker UI.
+- Đích cuối: code + registry ở Gitea `git.dbiz.com` (built-in registry) — nhưng KHÔNG chặn việc chạy hôm nay.
+
+## Triết lý
+- **Lakehouse as a product**: DBIZ lake = "khách số 0". Mọi thứ THAM SỐ HOÁ (không hardcode)
+  để sau nhân bản cho khách mua gói (Vietbank/Gemadept) qua **ApplicationSet** — CHƯA làm, hoãn tới khi có khách thứ 2.
+- **Stateful đứng yên, compute di**: MinIO (data lake) GIỮ NGUYÊN ngoài K8s, trỏ qua Service+Endpoints.
+  Kafka dựng mới (Strimzi) trên K8s — chấp nhận snapshot CDC lại.
+- **dev giống prod**: cùng chart, khác values theo env (dev/prod). Helm values-driven, portable.
+- **Portable cho on-prem**: chart nhận mọi dependency (S3/nguồn/kafka) qua values → chạy được cả trên
+  hạ tầng DBIZ lẫn on-prem tại khách (air-gapped: mirror image sang registry nội bộ khách).
+
+## Cấu trúc
+```
+charts/lake-cdc-job/        Chart CDC job (SparkApplication) — THAM SỐ HOÁ theo customer
+envs/dev/
+  apps/                     ArgoCD Applications (App-of-Apps)
+    root.yaml               Root -> quét apps/
+    infra.yaml              External Service+Endpoints (MinIO ngoài, Postgres nguồn)
+    strimzi-operator.yaml   Strimzi (Helm upstream)
+    spark-operator.yaml     Spark Operator (Helm upstream)
+    kafka-cluster.yaml      -> envs/dev/kafka (Kafka CRD)
+    cdc-job.yaml            CDC job (chart lake-cdc-job + values, multi-source)
+  kafka/                    Kafka CRD (Strimzi, KRaft, Longhorn PV)
+  infra/                    external-services.yaml (IP hạ tầng ngoài)
+  values/                   values per-app (cdc-job.yaml = khách số 0 dbiz)
+  sealed-secrets/           SealedSecrets (imagePullSecret, creds) — an toàn commit
+images/spark-cdc/           Dockerfile (image bundled)
+.woodpecker/                Woodpecker CI (build + push image lên Gitea registry)
+```
+
+## Điều kiện tiên quyết trên cụm (Phase 0 — dựng TRƯỚC)
+1. ArgoCD (đã có).
+2. **Longhorn** (StorageClass `longhorn`) — cho Kafka PV + checkpoint.
+3. **Sealed Secrets controller**.
+4. **cert-manager** + ingress (nếu expose UI).
+5. **Gitea `git.dbiz.com`** online + repo này + **Deploy Token** (đọc repo cho ArgoCD, đọc registry cho pull image).
+6. **Woodpecker** nối Gitea, khai secret `gitea_registry_user` / `gitea_registry_token`.
+
+## Bootstrap (khi Gitea + nền sẵn sàng)
+1. Điền IP thật vào `envs/dev/infra/external-services.yaml` (MinIO, Postgres nguồn).
+2. Seal imagePullSecret (Gitea registry, tên `gitea-registry`) -> `envs/dev/sealed-secrets/`.
+3. Push `images/spark-cdc/**` -> Woodpecker tự build + push image lên `git.dbiz.com/dbiz/spark-cdc`.
+4. `kubectl apply -f envs/dev/apps/root.yaml` -> ArgoCD sync tất cả.
+
+## Nhân bản cho khách mới (TƯƠNG LAI)
+- Copy `envs/dev/values/cdc-job.yaml`, đổi `customer`, nguồn, bucket, namespace, s3.endpoint.
+- Nhiều khách -> **ApplicationSet** (generator theo customer).
+- Mức cách ly (1/2/3) = preset values + namespace/cụm riêng. Vietbank on-prem = cùng chart, cụm khác.
+- S3: DBIZ dựng (ngoài/trong cụm) hoặc khách tự cấp — chỉ đổi `s3.endpoint` + secret, không sửa chart.
+
+## LƯU Ý
+- CDC Docker cũ GIỮ CHẠY tới khi CDC K8s verify xong, rồi mới tắt (không đập cái đang chạy).
+- Version Helm chart (Strimzi/Spark Operator) đang pin TẠM — kiểm bản mới nhất khi dựng thật.
+- 5 VM/1 host Proxmox: HA storage là ảo giác tới khi trải 2 host. Backup (Velero/pg_dump/mc mirror) là phòng thủ thật.
--- a/charts/lake-cdc-job/Chart.yaml
+++ b/charts/lake-cdc-job/Chart.yaml
@@ -0,0 +1,6 @@
+apiVersion: v2
+name: lake-cdc-job
+description: DBIZ Lakehouse CDC streaming job (SparkApplication) — tham số hoá theo khách hàng
+type: application
+version: 0.1.0
+appVersion: "1.0"
--- a/charts/lake-cdc-job/templates/checkpoint-pvc.yaml
+++ b/charts/lake-cdc-job/templates/checkpoint-pvc.yaml
@@ -0,0 +1,11 @@
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: cdc-{{ .Values.customer }}-checkpoint
+  namespace: {{ .Release.Namespace }}
+spec:
+  accessModes: [ ReadWriteOnce ]
+  storageClassName: {{ .Values.checkpoint.storageClass }}
+  resources:
+    requests:
+      storage: {{ .Values.checkpoint.size }}
--- a/charts/lake-cdc-job/templates/sparkapplication.yaml
+++ b/charts/lake-cdc-job/templates/sparkapplication.yaml
@@ -0,0 +1,68 @@
+apiVersion: sparkoperator.k8s.io/v1beta2
+kind: SparkApplication
+metadata:
+  name: cdc-{{ .Values.customer }}
+  namespace: {{ .Release.Namespace }}
+spec:
+  type: Python
+  mode: cluster
+  image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
+  imagePullPolicy: {{ .Values.image.pullPolicy }}
+  imagePullSecrets:
+    - {{ .Values.imagePullSecret }}
+  mainApplicationFile: "local:///opt/spark/work-dir/cdc_crm_to_raw.py"
+  sparkVersion: "{{ .Values.spark.version }}"
+  # streaming job chạy mãi -> restart nếu chết (thay cho việc submit tay foreground)
+  restartPolicy:
+    type: Always
+    onFailureRetryInterval: 30
+    onSubmissionFailureRetryInterval: 30
+  sparkConf:
+    # Iceberg catalog (Polaris REST) + MinIO
+    "spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"
+    "spark.sql.catalog.iceberg": "org.apache.iceberg.spark.SparkCatalog"
+    "spark.sql.catalog.iceberg.type": "rest"
+    "spark.sql.catalog.iceberg.uri": "{{ .Values.catalog.uri }}"
+    "spark.sql.catalog.iceberg.warehouse": "{{ .Values.catalog.warehouse }}"
+    "spark.sql.catalog.iceberg.header.X-Iceberg-Access-Delegation": "vended-credentials"
+    "spark.sql.catalog.iceberg.io-impl": "org.apache.iceberg.aws.s3.S3FileIO"
+    "spark.sql.catalog.iceberg.s3.endpoint": "{{ .Values.s3.endpoint }}"
+    "spark.sql.catalog.iceberg.s3.path-style-access": "true"
+    "spark.sql.catalog.iceberg.s3.region": "{{ .Values.s3.region }}"
+    "spark.sql.defaultCatalog": "iceberg"
+  driver:
+    cores: {{ .Values.spark.driverCores }}
+    memory: "{{ .Values.spark.driverMemory }}"
+    labels: { app: cdc-{{ .Values.customer }} }
+    env:
+      - name: KAFKA_BOOTSTRAP
+        value: "{{ .Values.kafka.bootstrap }}"
+      - name: TOPIC_PATTERN
+        value: "{{ .Values.topicPattern }}"
+      - name: TARGET_TABLE
+        value: "{{ .Values.targetTable }}"
+      - name: TRIGGER_INTERVAL
+        value: "{{ .Values.triggerInterval }}"
+      - name: MAX_OFFSETS_PER_TRIGGER
+        value: "{{ .Values.maxOffsetsPerTrigger }}"
+      - name: CHECKPOINT
+        value: "file:///checkpoints/cdc_{{ .Values.customer }}"
+    volumeMounts:
+      - name: checkpoint
+        mountPath: /checkpoints
+  executor:
+    instances: {{ .Values.spark.executorInstances }}
+    cores: {{ .Values.spark.executorCores }}
+    memory: "{{ .Values.spark.executorMemory }}"
+    volumeMounts:
+      - name: checkpoint
+        mountPath: /checkpoints
+  volumes:
+    - name: checkpoint
+      persistentVolumeClaim:
+        claimName: cdc-{{ .Values.customer }}-checkpoint
+  deps:
+    packages:
+      - org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.9.1
+      - org.apache.iceberg:iceberg-aws-bundle:1.9.1
+      - org.apache.spark:spark-sql-kafka-0-10_2.12:3.5.6
--- a/charts/lake-cdc-job/values.yaml
+++ b/charts/lake-cdc-job/values.yaml
@@ -0,0 +1,36 @@
+# ====== THAM SỐ HOÁ THEO KHÁCH HÀNG (customer = "khách số 0" dbiz) ======
+customer: dbiz                       # định danh khách; đổi khi nhân bản (vietbank/gemadept)
+image:
+  repository: ""                     # bắt buộc override: <registry>/spark-cdc
+  tag: "latest"
+  pullPolicy: Always
+imagePullSecret: gitea-registry     # SealedSecret imagePullSecret
+
+spark:
+  version: "3.5.6"
+  driverCores: 1
+  driverMemory: "1g"
+  executorInstances: 1
+  executorCores: 2
+  executorMemory: "2g"
+
+# nguồn Kafka (Strimzi trong cụm) + đích Iceberg
+kafka:
+  bootstrap: "lake-kafka-kafka-bootstrap:9092"   # service Strimzi sinh ra
+topicPattern: "crm\\.public\\..*"
+targetTable: "iceberg.raw_crm.cdc_events"
+triggerInterval: "30 seconds"
+maxOffsetsPerTrigger: "50000"
+
+# checkpoint: PVC Longhorn (bền qua restart) — mỗi khách 1 PVC riêng
+checkpoint:
+  storageClass: longhorn
+  size: 5Gi
+
+# catalog (Polaris) + MinIO endpoint — trỏ hạ tầng lake
+catalog:
+  uri: "http://polaris:8181/api/catalog"          # điều chỉnh theo Polaris trong cụm
+  warehouse: "dbiz_warehouse"
+s3:
+  endpoint: "http://minio-lake:9000"              # external-service trỏ MinIO ngoài
+  region: "us-east-1"
--- a/envs/dev/apps/cdc-job.yaml
+++ b/envs/dev/apps/cdc-job.yaml
@@ -0,0 +1,22 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: lake-cdc-job-dev
+  namespace: argocd
+spec:
+  project: default
+  sources:
+    - repoURL: <REPO_URL>
+      targetRevision: main
+      path: charts/lake-cdc-job
+      helm:
+        valueFiles:
+          - $values/envs/dev/values/cdc-job.yaml
+    - repoURL: <REPO_URL>
+      targetRevision: main
+      ref: values
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: dbiz-lake-dev
+  syncPolicy:
+    automated: { prune: true, selfHeal: true }
--- a/envs/dev/apps/infra.yaml
+++ b/envs/dev/apps/infra.yaml
@@ -0,0 +1,22 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: lake-infra-dev
+  namespace: argocd
+spec:
+  project: default
+  source:
+    repoURL: <REPO_URL>
+    targetRevision: main
+    path: envs/dev/infra
+    directory:
+      recurse: true
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: dbiz-lake-dev
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true
--- a/envs/dev/apps/kafka-cluster.yaml
+++ b/envs/dev/apps/kafka-cluster.yaml
@@ -0,0 +1,17 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: lake-kafka-dev
+  namespace: argocd
+spec:
+  project: default
+  source:
+    repoURL: <REPO_URL>
+    targetRevision: main
+    path: envs/dev/kafka
+    directory: { recurse: true }
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: dbiz-lake-dev
+  syncPolicy:
+    automated: { prune: true, selfHeal: true }
--- a/envs/dev/apps/root.yaml
+++ b/envs/dev/apps/root.yaml
@@ -0,0 +1,20 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: lake-root-dev
+  namespace: argocd
+spec:
+  project: default
+  source:
+    repoURL: <REPO_URL>
+    targetRevision: main
+    path: envs/dev/apps
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: argocd
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true
--- a/envs/dev/apps/spark-operator.yaml
+++ b/envs/dev/apps/spark-operator.yaml
@@ -0,0 +1,25 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: lake-spark-operator-dev
+  namespace: argocd
+spec:
+  project: default
+  source:
+    repoURL: https://kubeflow.github.io/spark-operator
+    chart: spark-operator
+    targetRevision: 2.1.1            # pin (kiểm bản mới khi dựng)
+    helm:
+      releaseName: spark-operator
+      values: |
+        spark:
+          jobNamespaces:
+            - dbiz-lake-dev
+        webhook:
+          enable: true
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: dbiz-lake-dev
+  syncPolicy:
+    automated: { prune: true, selfHeal: true }
+    syncOptions: [ CreateNamespace=true ]
--- a/envs/dev/apps/strimzi-operator.yaml
+++ b/envs/dev/apps/strimzi-operator.yaml
@@ -0,0 +1,19 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: lake-strimzi-operator-dev
+  namespace: argocd
+spec:
+  project: default
+  source:
+    repoURL: https://strimzi.io/charts/
+    chart: strimzi-kafka-operator
+    targetRevision: 0.45.0          # pin version (kiểm bản mới nhất khi dựng thật)
+    helm:
+      releaseName: strimzi
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: dbiz-lake-dev
+  syncPolicy:
+    automated: { prune: true, selfHeal: true }
+    syncOptions: [ CreateNamespace=true ]
--- a/envs/dev/infra/external-services.yaml
+++ b/envs/dev/infra/external-services.yaml
@@ -0,0 +1,50 @@
+# Service + Endpoints không selector -> trỏ tới hạ tầng NGOÀI K8s (theo pattern argo-k8s).
+# MinIO lake: GIỮ NGUYÊN, K8s compute trỏ vào đây như S3 endpoint.
+# ĐIỀN IP THẬT của máy chạy MinIO (VM lake) trước khi apply.
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: minio-lake
+  namespace: dbiz-lake-dev
+spec:
+  ports:
+    - name: s3
+      port: 9000
+      targetPort: 9000
+---
+apiVersion: v1
+kind: Endpoints
+metadata:
+  name: minio-lake
+  namespace: dbiz-lake-dev
+subsets:
+  - addresses:
+      - ip: 192.168.110.37   # <-- ĐIỀN IP máy MinIO (VM lake)
+    ports:
+      - name: s3
+        port: 9000
+---
+# Postgres nguồn CRM (172.20.109.5) — để Debezium/Strimzi trỏ tới khi CDC
+apiVersion: v1
+kind: Service
+metadata:
+  name: crm-source-pg
+  namespace: dbiz-lake-dev
+spec:
+  ports:
+    - name: pg
+      port: 5432
+      targetPort: 5432
+---
+apiVersion: v1
+kind: Endpoints
+metadata:
+  name: crm-source-pg
+  namespace: dbiz-lake-dev
+subsets:
+  - addresses:
+      - ip: 172.20.109.5
+    ports:
+      - name: pg
+        port: 5432
--- a/envs/dev/kafka/kafka.yaml
+++ b/envs/dev/kafka/kafka.yaml
@@ -0,0 +1,45 @@
+# Strimzi Kafka (KRaft mode, không Zookeeper). PV qua Longhorn.
+# Dev: 1 broker; prod: 3 broker (đổi replicas + storage).
+apiVersion: kafka.strimzi.io/v1beta2
+kind: KafkaNodePool
+metadata:
+  name: dual-role
+  namespace: dbiz-lake-dev
+  labels:
+    strimzi.io/cluster: lake-kafka
+spec:
+  replicas: 1
+  roles: [ controller, broker ]
+  storage:
+    type: persistent-claim
+    size: 20Gi
+    class: longhorn
+    deleteClaim: false
+---
+apiVersion: kafka.strimzi.io/v1beta2
+kind: Kafka
+metadata:
+  name: lake-kafka
+  namespace: dbiz-lake-dev
+  annotations:
+    strimzi.io/node-pools: enabled
+    strimzi.io/kraft: enabled
+spec:
+  kafka:
+    version: 3.9.0
+    listeners:
+      - name: plain
+        port: 9092
+        type: internal
+        tls: false
+    config:
+      # message to (giải bài RecordTooLarge đã gặp): 10MB
+      message.max.bytes: 10485760
+      offsets.topic.replication.factor: 1
+      transaction.state.log.replication.factor: 1
+      transaction.state.log.min.isr: 1
+      default.replication.factor: 1
+      min.insync.replicas: 1
+  entityOperator:
+    topicOperator: {}
+    userOperator: {}
--- a/envs/dev/sealed-secrets/.gitkeep
+++ b/envs/dev/sealed-secrets/.gitkeep
--- a/envs/dev/values/cdc-job.yaml
+++ b/envs/dev/values/cdc-job.yaml
@@ -0,0 +1,23 @@
+# DBIZ lake — "khách số 0". Nhân bản: copy file này, đổi customer + nguồn + bucket.
+customer: dbiz
+image:
+  repository: git.dbiz.com/dbiz/spark-cdc   # <-- Gitea registry built-in (git.dbiz.com/dbiz/spark-cdc)
+  tag: "latest"
+imagePullSecret: gitea-registry
+
+kafka:
+  bootstrap: "lake-kafka-kafka-bootstrap:9092"
+topicPattern: "crm\\.public\\..*"
+targetTable: "iceberg.raw_crm.cdc_events"
+triggerInterval: "30 seconds"
+
+catalog:
+  uri: "http://polaris:8181/api/catalog"
+  warehouse: "dbiz_warehouse"
+s3:
+  endpoint: "http://minio-lake:9000"
+  region: "us-east-1"
+
+checkpoint:
+  storageClass: longhorn
+  size: 5Gi
--- a/envs/prod/sealed-secrets/.gitkeep
+++ b/envs/prod/sealed-secrets/.gitkeep
--- a/images/spark-cdc/Dockerfile
+++ b/images/spark-cdc/Dockerfile
@@ -0,0 +1,14 @@
+# Image CDC job: Spark + jar Iceberg/Kafka bundled + script. Reproducible, khởi động nhanh.
+FROM spark:3.5.6
+
+USER root
+ARG IVY_PKGS="org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.9.1,org.apache.iceberg:iceberg-aws-bundle:1.9.1,org.apache.spark:spark-sql-kafka-0-10_2.12:3.5.6"
+
+# Tải sẵn các jar vào /opt/spark/jars (không tải runtime mỗi lần pod start)
+RUN /opt/spark/bin/spark-shell --packages "${IVY_PKGS}" --conf spark.jars.ivy=/tmp/.ivy <<< 'System.exit(0)' || true \
+    && find /root/.ivy2 -name '*.jar' -exec cp {} /opt/spark/jars/ \; 2>/dev/null || true
+
+# Script CDC (copy vào work-dir)
+COPY cdc_crm_to_raw.py /opt/spark/work-dir/cdc_crm_to_raw.py
+
+USER 185