init

.opencode/skills/backend-development/references/backend-debugging.md

@@ -0,0 +1,904 @@
+# Backend Debugging Strategies
+
+Comprehensive debugging techniques, tools, and best practices for backend systems (2025).
+
+## Debugging Mindset
+
+### The Scientific Method for Debugging
+
+1. **Observe** - Gather symptoms and data
+2. **Hypothesize** - Form theories about the cause
+3. **Test** - Verify or disprove theories
+4. **Iterate** - Refine understanding
+5. **Fix** - Apply solution
+6. **Verify** - Confirm fix works
+
+### Golden Rules
+
+1. **Reproduce first** - Debugging without reproduction is guessing
+2. **Simplify the problem** - Isolate variables
+3. **Read the logs** - Error messages contain clues
+4. **Check assumptions** - "It should work" isn't debugging
+5. **Use scientific method** - Avoid random changes
+6. **Document findings** - Future you will thank you
+
+## Logging Best Practices
+
+### Structured Logging
+
+**Node.js (Pino - Fastest)**
+```typescript
+import pino from 'pino';
+
+const logger = pino({
+  level: process.env.LOG_LEVEL || 'info',
+  transport: {
+    target: 'pino-pretty',
+    options: { colorize: true }
+  }
+});
+
+// Structured logging with context
+logger.info({ userId: '123', action: 'login' }, 'User logged in');
+
+// Error logging with stack trace
+try {
+  await riskyOperation();
+} catch (error) {
+  logger.error({ err: error, userId: '123' }, 'Operation failed');
+}
+```
+
+**Python (Structlog)**
+```python
+import structlog
+
+logger = structlog.get_logger()
+
+# Structured context
+logger.info("user_login", user_id="123", ip="192.168.1.1")
+
+# Error with exception
+try:
+    risky_operation()
+except Exception as e:
+    logger.error("operation_failed", user_id="123", exc_info=True)
+```
+
+**Go (Zap - High Performance)**
+```go
+import "go.uber.org/zap"
+
+logger, _ := zap.NewProduction()
+defer logger.Sync()
+
+// Structured fields
+logger.Info("user logged in",
+    zap.String("user_id", "123"),
+    zap.String("ip", "192.168.1.1"),
+)
+
+// Error logging
+if err := riskyOperation(); err != nil {
+    logger.Error("operation failed",
+        zap.Error(err),
+        zap.String("user_id", "123"),
+    )
+}
+```
+
+### Log Levels
+
+| Level | Purpose | Example |
+|-------|---------|---------|
+| **TRACE** | Very detailed, dev only | Request/response bodies |
+| **DEBUG** | Detailed info for debugging | SQL queries, cache hits |
+| **INFO** | General informational | User login, API calls |
+| **WARN** | Potential issues | Deprecated API usage |
+| **ERROR** | Error conditions | Failed API calls, exceptions |
+| **FATAL** | Critical failures | Database connection lost |
+
+### What to Log
+
+**✅ DO LOG:**
+- Request/response metadata (not bodies in prod)
+- Error messages with context
+- Performance metrics (duration, size)
+- Security events (login, permission changes)
+- Business events (orders, payments)
+
+**❌ DON'T LOG:**
+- Passwords or secrets
+- Credit card numbers
+- Personal identifiable information (PII)
+- Session tokens
+- Full request bodies in production
+
+## Debugging Tools by Language
+
+### Node.js / TypeScript
+
+**1. Chrome DevTools (Built-in)**
+```bash
+# Run with inspect flag
+node --inspect-brk app.js
+
+# Open chrome://inspect in Chrome
+# Set breakpoints, step through code
+```
+
+**2. VS Code Debugger**
+```json
+// .vscode/launch.json
+{
+  "version": "0.2.0",
+  "configurations": [
+    {
+      "type": "node",
+      "request": "launch",
+      "name": "Debug Server",
+      "skipFiles": ["<node_internals>/**"],
+      "program": "${workspaceFolder}/src/index.ts",
+      "preLaunchTask": "npm: build",
+      "outFiles": ["${workspaceFolder}/dist/**/*.js"]
+    }
+  ]
+}
+```
+
+**3. Debug Module**
+```typescript
+import debug from 'debug';
+
+const log = debug('app:server');
+const error = debug('app:error');
+
+log('Starting server on port %d', 3000);
+error('Failed to connect to database');
+
+// Run with: DEBUG=app:* node app.js
+```
+
+### Python
+
+**1. PDB (Built-in Debugger)**
+```python
+import pdb
+
+def problematic_function(data):
+    # Set breakpoint
+    pdb.set_trace()
+
+    # Debugger commands:
+    # l - list code
+    # n - next line
+    # s - step into
+    # c - continue
+    # p variable - print variable
+    # q - quit
+    result = process(data)
+    return result
+```
+
+**2. IPython Debugger (Better)**
+```python
+from IPython import embed
+
+def problematic_function(data):
+    # Drop into IPython shell
+    embed()
+
+    result = process(data)
+    return result
+```
+
+**3. VS Code Debugger**
+```json
+// .vscode/launch.json
+{
+  "version": "0.2.0",
+  "configurations": [
+    {
+      "name": "Python: FastAPI",
+      "type": "python",
+      "request": "launch",
+      "module": "uvicorn",
+      "args": ["main:app", "--reload"],
+      "jinja": true
+    }
+  ]
+}
+```
+
+### Go
+
+**1. Delve (Standard Debugger)**
+```bash
+# Install
+go install github.com/go-delve/delve/cmd/dlv@latest
+
+# Debug
+dlv debug main.go
+
+# Commands:
+# b main.main - set breakpoint
+# c - continue
+# n - next line
+# s - step into
+# p variable - print variable
+# q - quit
+```
+
+**2. VS Code Debugger**
+```json
+// .vscode/launch.json
+{
+  "version": "0.2.0",
+  "configurations": [
+    {
+      "name": "Launch Package",
+      "type": "go",
+      "request": "launch",
+      "mode": "debug",
+      "program": "${workspaceFolder}"
+    }
+  ]
+}
+```
+
+### Rust
+
+**1. LLDB/GDB (Native Debuggers)**
+```bash
+# Build with debug info
+cargo build
+
+# Debug with LLDB
+rust-lldb ./target/debug/myapp
+
+# Debug with GDB
+rust-gdb ./target/debug/myapp
+```
+
+**2. VS Code Debugger (CodeLLDB)**
+```json
+// .vscode/launch.json
+{
+  "version": "0.2.0",
+  "configurations": [
+    {
+      "type": "lldb",
+      "request": "launch",
+      "name": "Debug",
+      "program": "${workspaceFolder}/target/debug/myapp",
+      "args": [],
+      "cwd": "${workspaceFolder}"
+    }
+  ]
+}
+```
+
+## Database Debugging
+
+### SQL Query Debugging (PostgreSQL)
+
+**1. EXPLAIN ANALYZE**
+```sql
+-- Show query execution plan and actual timings
+EXPLAIN ANALYZE
+SELECT u.name, COUNT(o.id) as order_count
+FROM users u
+LEFT JOIN orders o ON u.id = o.user_id
+WHERE u.created_at > '2024-01-01'
+GROUP BY u.id, u.name
+ORDER BY order_count DESC
+LIMIT 10;
+
+-- Look for:
+-- - Seq Scan on large tables (missing indexes)
+-- - High execution time
+-- - Large row estimates
+```
+
+**2. Enable Slow Query Logging**
+```sql
+-- PostgreSQL configuration
+ALTER DATABASE mydb SET log_min_duration_statement = 1000; -- Log queries >1s
+
+-- Check slow queries
+SELECT query, calls, total_exec_time, mean_exec_time
+FROM pg_stat_statements
+ORDER BY mean_exec_time DESC
+LIMIT 10;
+```
+
+**3. Active Query Monitoring**
+```sql
+-- See currently running queries
+SELECT pid, now() - query_start as duration, query, state
+FROM pg_stat_activity
+WHERE state = 'active'
+ORDER BY duration DESC;
+
+-- Kill a long-running query
+SELECT pg_terminate_backend(pid);
+```
+
+### MongoDB Debugging
+
+**1. Explain Query Performance**
+```javascript
+db.users.find({ email: 'test@example.com' }).explain('executionStats')
+
+// Look for:
+// - totalDocsExamined vs nReturned (should be close)
+// - COLLSCAN (collection scan - needs index)
+// - executionTimeMillis (should be low)
+```
+
+**2. Profile Slow Queries**
+```javascript
+// Enable profiling for queries >100ms
+db.setProfilingLevel(1, { slowms: 100 })
+
+// View slow queries
+db.system.profile.find().limit(5).sort({ ts: -1 }).pretty()
+
+// Disable profiling
+db.setProfilingLevel(0)
+```
+
+### Redis Debugging
+
+**1. Monitor Commands**
+```bash
+# See all commands in real-time
+redis-cli MONITOR
+
+# Check slow log
+redis-cli SLOWLOG GET 10
+
+# Set slow log threshold (microseconds)
+redis-cli CONFIG SET slowlog-log-slower-than 10000
+```
+
+**2. Memory Analysis**
+```bash
+# Memory usage by key pattern
+redis-cli --bigkeys
+
+# Memory usage details
+redis-cli INFO memory
+
+# Analyze specific key
+redis-cli MEMORY USAGE mykey
+```
+
+## API Debugging
+
+### HTTP Request Debugging
+
+**1. cURL Testing**
+```bash
+# Verbose output with headers
+curl -v https://api.example.com/users
+
+# Include response headers
+curl -i https://api.example.com/users
+
+# POST with JSON
+curl -X POST https://api.example.com/users \
+  -H "Content-Type: application/json" \
+  -d '{"name":"John","email":"john@example.com"}' \
+  -v
+
+# Save response to file
+curl https://api.example.com/users -o response.json
+```
+
+**2. HTTPie (User-Friendly)**
+```bash
+# Install
+pip install httpie
+
+# Simple GET
+http GET https://api.example.com/users
+
+# POST with JSON
+http POST https://api.example.com/users name=John email=john@example.com
+
+# Custom headers
+http GET https://api.example.com/users Authorization:"Bearer token123"
+```
+
+**3. Request Logging Middleware**
+
+**Express/Node.js:**
+```typescript
+import morgan from 'morgan';
+
+// Development
+app.use(morgan('dev'));
+
+// Production (JSON format)
+app.use(morgan('combined'));
+
+// Custom format
+app.use(morgan(':method :url :status :response-time ms - :res[content-length]'));
+```
+
+**FastAPI/Python:**
+```python
+from fastapi import Request
+import time
+
+@app.middleware("http")
+async def log_requests(request: Request, call_next):
+    start_time = time.time()
+    response = await call_next(request)
+    duration = time.time() - start_time
+
+    logger.info(
+        "request_processed",
+        method=request.method,
+        path=request.url.path,
+        status_code=response.status_code,
+        duration_ms=duration * 1000
+    )
+    return response
+```
+
+## Performance Debugging
+
+### CPU Profiling
+
+**Node.js (0x)**
+```bash
+# Install
+npm install -g 0x
+
+# Profile application
+0x node app.js
+
+# Open flamegraph in browser
+# Identify hot spots (red areas)
+```
+
+**Node.js (Clinic.js)**
+```bash
+# Install
+npm install -g clinic
+
+# CPU profiling
+clinic doctor -- node app.js
+
+# Heap profiling
+clinic heapprofiler -- node app.js
+
+# Event loop analysis
+clinic bubbleprof -- node app.js
+```
+
+**Python (cProfile)**
+```python
+import cProfile
+import pstats
+
+# Profile function
+profiler = cProfile.Profile()
+profiler.enable()
+
+# Your code
+result = expensive_operation()
+
+profiler.disable()
+stats = pstats.Stats(profiler)
+stats.sort_stats('cumulative')
+stats.print_stats(10)  # Top 10 functions
+```
+
+**Go (pprof)**
+```go
+import (
+    "net/http"
+    _ "net/http/pprof"
+)
+
+func main() {
+    // Enable profiling endpoint
+    go func() {
+        http.ListenAndServe("localhost:6060", nil)
+    }()
+
+    // Your application
+    startServer()
+}
+
+// Profile CPU
+// go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30
+
+// Profile heap
+// go tool pprof http://localhost:6060/debug/pprof/heap
+```
+
+### Memory Debugging
+
+**Node.js (Heap Snapshots)**
+```typescript
+// Take heap snapshot programmatically
+import { writeHeapSnapshot } from 'v8';
+
+app.get('/debug/heap', (req, res) => {
+    const filename = writeHeapSnapshot();
+    res.send(`Heap snapshot written to ${filename}`);
+});
+
+// Analyze in Chrome DevTools
+// 1. Load heap snapshot
+// 2. Compare snapshots to find memory leaks
+// 3. Look for detached DOM nodes, large arrays
+```
+
+**Python (Memory Profiler)**
+```python
+from memory_profiler import profile
+
+@profile
+def memory_intensive_function():
+    large_list = [i for i in range(1000000)]
+    return sum(large_list)
+
+# Run with: python -m memory_profiler script.py
+# Shows line-by-line memory usage
+```
+
+## Production Debugging
+
+### Application Performance Monitoring (APM)
+
+**New Relic**
+```typescript
+// newrelic.js
+export const config = {
+  app_name: ['My Backend API'],
+  license_key: process.env.NEW_RELIC_LICENSE_KEY,
+  logging: { level: 'info' },
+  distributed_tracing: { enabled: true },
+};
+
+// Import at app entry
+import 'newrelic';
+```
+
+**DataDog**
+```typescript
+import tracer from 'dd-trace';
+
+tracer.init({
+  service: 'backend-api',
+  env: process.env.NODE_ENV,
+  version: '1.0.0',
+  logInjection: true
+});
+```
+
+**Sentry (Error Tracking)**
+```typescript
+import * as Sentry from '@sentry/node';
+
+Sentry.init({
+  dsn: process.env.SENTRY_DSN,
+  environment: process.env.NODE_ENV,
+  tracesSampleRate: 1.0,
+});
+
+// Capture errors
+try {
+  await riskyOperation();
+} catch (error) {
+  Sentry.captureException(error, {
+    user: { id: userId },
+    tags: { operation: 'payment' },
+  });
+}
+```
+
+### Distributed Tracing
+
+**OpenTelemetry (Vendor-Agnostic)**
+```typescript
+import { NodeSDK } from '@opentelemetry/sdk-node';
+import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
+import { JaegerExporter } from '@opentelemetry/exporter-jaeger';
+
+const sdk = new NodeSDK({
+  traceExporter: new JaegerExporter({
+    endpoint: 'http://localhost:14268/api/traces',
+  }),
+  instrumentations: [getNodeAutoInstrumentations()],
+});
+
+sdk.start();
+
+// Traces HTTP, database, Redis automatically
+```
+
+### Log Aggregation
+
+**ELK Stack (Elasticsearch, Logstash, Kibana)**
+```yaml
+# docker-compose.yml
+version: '3'
+services:
+  elasticsearch:
+    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
+    environment:
+      - discovery.type=single-node
+    ports:
+      - 9200:9200
+
+  logstash:
+    image: docker.elastic.co/logstash/logstash:8.11.0
+    volumes:
+      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
+
+  kibana:
+    image: docker.elastic.co/kibana/kibana:8.11.0
+    ports:
+      - 5601:5601
+```
+
+**Loki + Grafana (Lightweight)**
+```yaml
+# promtail config for log shipping
+server:
+  http_listen_port: 9080
+
+positions:
+  filename: /tmp/positions.yaml
+
+clients:
+  - url: http://loki:3100/loki/api/v1/push
+
+scrape_configs:
+  - job_name: system
+    static_configs:
+      - targets:
+          - localhost
+        labels:
+          job: backend-api
+          __path__: /var/log/app/*.log
+```
+
+## Common Debugging Scenarios
+
+### 1. High CPU Usage
+
+**Steps:**
+1. Profile CPU (flamegraph)
+2. Identify hot functions
+3. Check for:
+   - Infinite loops
+   - Heavy regex operations
+   - Inefficient algorithms (O(n²))
+   - Blocking operations in event loop (Node.js)
+
+**Node.js Example:**
+```typescript
+// ❌ Bad: Blocking event loop
+function fibonacci(n) {
+  if (n <= 1) return n;
+  return fibonacci(n - 1) + fibonacci(n - 2); // Exponential time
+}
+
+// ✅ Good: Memoized or iterative
+const memo = new Map();
+function fibonacciMemo(n) {
+  if (n <= 1) return n;
+  if (memo.has(n)) return memo.get(n);
+  const result = fibonacciMemo(n - 1) + fibonacciMemo(n - 2);
+  memo.set(n, result);
+  return result;
+}
+```
+
+### 2. Memory Leaks
+
+**Symptoms:**
+- Memory usage grows over time
+- Eventually crashes (OOM)
+- Performance degradation
+
+**Common Causes:**
+```typescript
+// ❌ Memory leak: Event listeners not removed
+class DataService {
+  constructor(eventBus) {
+    eventBus.on('data', (data) => this.processData(data));
+    // Listener never removed, holds reference to DataService
+  }
+}
+
+// ✅ Fix: Remove listeners
+class DataService {
+  constructor(eventBus) {
+    this.eventBus = eventBus;
+    this.handler = (data) => this.processData(data);
+    eventBus.on('data', this.handler);
+  }
+
+  destroy() {
+    this.eventBus.off('data', this.handler);
+  }
+}
+
+// ❌ Memory leak: Global cache without limits
+const cache = new Map();
+function getCachedData(key) {
+  if (!cache.has(key)) {
+    cache.set(key, expensiveOperation(key)); // Grows forever
+  }
+  return cache.get(key);
+}
+
+// ✅ Fix: LRU cache with size limit
+import LRU from 'lru-cache';
+const cache = new LRU({ max: 1000, ttl: 1000 * 60 * 60 });
+```
+
+**Detection:**
+```bash
+# Node.js: Check heap size over time
+node --expose-gc --max-old-space-size=4096 app.js
+
+# Take periodic heap snapshots
+# Compare snapshots in Chrome DevTools
+```
+
+### 3. Slow Database Queries
+
+**Steps:**
+1. Enable slow query log
+2. Analyze with EXPLAIN
+3. Add indexes
+4. Optimize query
+
+**PostgreSQL Example:**
+```sql
+-- Before: Slow full table scan
+SELECT * FROM orders
+WHERE user_id = 123
+ORDER BY created_at DESC
+LIMIT 10;
+
+-- EXPLAIN shows: Seq Scan on orders
+
+-- Fix: Add index
+CREATE INDEX idx_orders_user_id_created_at
+ON orders(user_id, created_at DESC);
+
+-- After: Index Scan using idx_orders_user_id_created_at
+-- 100x faster
+```
+
+### 4. Connection Pool Exhaustion
+
+**Symptoms:**
+- "Connection pool exhausted" errors
+- Requests hang indefinitely
+- Database connections at max
+
+**Causes & Fixes:**
+```typescript
+// ❌ Bad: Connection leak
+async function getUser(id) {
+  const client = await pool.connect();
+  const result = await client.query('SELECT * FROM users WHERE id = $1', [id]);
+  return result.rows[0];
+  // Connection never released!
+}
+
+// ✅ Good: Always release
+async function getUser(id) {
+  const client = await pool.connect();
+  try {
+    const result = await client.query('SELECT * FROM users WHERE id = $1', [id]);
+    return result.rows[0];
+  } finally {
+    client.release(); // Always release
+  }
+}
+
+// ✅ Better: Use pool directly
+async function getUser(id) {
+  const result = await pool.query('SELECT * FROM users WHERE id = $1', [id]);
+  return result.rows[0];
+  // Automatically releases
+}
+```
+
+### 5. Race Conditions
+
+**Example:**
+```typescript
+// ❌ Bad: Race condition
+let counter = 0;
+
+async function incrementCounter() {
+  const current = counter; // Thread 1 reads 0
+  await doSomethingAsync(); // Thread 2 reads 0
+  counter = current + 1; // Thread 1 writes 1, Thread 2 writes 1
+  // Expected: 2, Actual: 1
+}
+
+// ✅ Fix: Atomic operations (Redis)
+async function incrementCounter() {
+  return await redis.incr('counter');
+  // Atomic, thread-safe
+}
+
+// ✅ Fix: Database transactions
+async function incrementCounter(userId) {
+  await db.transaction(async (trx) => {
+    const user = await trx('users')
+      .where({ id: userId })
+      .forUpdate() // Row-level lock
+      .first();
+
+    await trx('users')
+      .where({ id: userId })
+      .update({ counter: user.counter + 1 });
+  });
+}
+```
+
+## Debugging Checklist
+
+**Before Diving Into Code:**
+- [ ] Read error message completely
+- [ ] Check logs for context
+- [ ] Reproduce the issue reliably
+- [ ] Isolate the problem (binary search)
+- [ ] Verify assumptions
+
+**Investigation:**
+- [ ] Enable debug logging
+- [ ] Add strategic log points
+- [ ] Use debugger breakpoints
+- [ ] Profile performance if slow
+- [ ] Check database queries
+- [ ] Monitor system resources
+
+**Production Issues:**
+- [ ] Check APM dashboards
+- [ ] Review distributed traces
+- [ ] Analyze error rates
+- [ ] Compare with previous baseline
+- [ ] Check for recent deployments
+- [ ] Review infrastructure changes
+
+**After Fix:**
+- [ ] Verify fix in development
+- [ ] Add regression test
+- [ ] Document the issue
+- [ ] Deploy with monitoring
+- [ ] Confirm fix in production
+
+## Debugging Resources
+
+**Tools:**
+- Node.js: https://nodejs.org/en/docs/guides/debugging-getting-started/
+- Chrome DevTools: https://developer.chrome.com/docs/devtools/
+- Clinic.js: https://clinicjs.org/
+- Sentry: https://docs.sentry.io/
+- DataDog: https://docs.datadoghq.com/
+- New Relic: https://docs.newrelic.com/
+
+**Best Practices:**
+- 12 Factor App Logs: https://12factor.net/logs
+- Google SRE Book: https://sre.google/sre-book/table-of-contents/
+- OpenTelemetry: https://opentelemetry.io/docs/
+
+**Database:**
+- PostgreSQL EXPLAIN: https://www.postgresql.org/docs/current/using-explain.html
+- MongoDB Performance: https://www.mongodb.com/docs/manual/administration/analyzing-mongodb-performance/