LoadBalancer-KisumAI-Worker (Cloudflare Worker LB)

Related documentation: Kisum System · Infrastructure Tasks · Chat-Kisum-MCP-Node

What it is

modules/System-KisumAI-LoadBalancer is a free Cloudflare Worker load balancer. It is one codebase (worker.js) deployed several times — once per system — via a separate wrangler-*.jsonc config each. The SERVICES variable in a config decides how that deployment behaves.

Two modes (chosen by the shape of `SERVICES`)

`SERVICES` shape	Mode	Behavior
Array of origins	Single-service (passthrough)	No path routing; the full path is forwarded to one origin pool. Synthetic service name `default`.
Object `{ service: [...] }`	Multi-service (path routing)	Routes by the first path segment; each key is a service with its own origins/health/failover. Unknown paths → 404.

Deployment targets

Config file	Worker name	Mode	Purpose
`wrangler-kisum.jsonc`	`loadbalancer-kisum`	multi-service	Kisum platform behind `api-v2.kisum.dev`
`wrangler-ai.jsonc`	`loadbalancer-kisumai`	single-service	AI chat backend (`chat-a` / `chat-b`)
`wrangler-stage.jsonc`	`loadbalancer-thestage`	single-service	TheStage API

Deploy a target with -c (npm scripts deploy:kisum / deploy:ai / deploy:stage). There is no default wrangler.jsonc. For Git-connected Cloudflare Workers Builds, each target’s deploy command must also pass -c wrangler-<target>.jsonc (e.g. npx wrangler deploy -c wrangler-ai.jsonc, non-prod npx wrangler versions upload -c wrangler-ai.jsonc), build path /. Full command table: modules/System-KisumAI-LoadBalancer/README.md.

Kisum public entry point: api-v2.kisum.dev. In multi-service mode requests are routed by the first path segment:

Public path	Service	Example origins
`/auth/*`	Auth	`api-A-auth.kisum.dev`, `api-B-auth.kisum.dev`, …
`/core/*`	Core	`api-A-core.kisum.dev`, …
`/artists/*`	Artists	`api-A-artists.kisum.dev`, …
`/venues/*`	Venues	`api-A-venues.kisum.dev`, …
`/promoters/*`	Promoters	`api-A-promoters.kisum.dev`, …
`/musicdata/*`	MusicData	`api-A-musicdata.kisum.dev`, …
`/chat/*`	Chat (+ Centrifugo live-feed WebSocket under `/chat/live-feed/`)	`api-a-chat.kisum.dev`, `api-b-chat.kisum.dev`, …
`/ai/*`	AI (chat + sessions)	`api-a-ai.kisum.dev`, `api-b-ai.kisum.dev`, …

Example: api.kisum.io/ai/chat → one of api-a-ai.kisum.dev/chat, … The /ai prefix is stripped before forwarding (/ai/sessions → /sessions). In single-service (array) mode there is no prefix logic — the full path is forwarded unchanged.

Deliberately not Cloudflare’s paid Load Balancing product — a free, good-enough balancer is enough here.

Behavior contract

Routing: single-service mode forwards every path to one pool (no 404). Multi-service mode routes by the first path segment; unknown paths return 404.
Per-service isolation: each service has its own origins, weights, and health cache. Failover never crosses services — /auth/* only fails over between Auth origins, never to Core/Artists/etc.
Weighted random split per request (statistical, not exact) among a service’s origins.
Failover: connection errors → retry the next origin of the same service (all methods). 5xx responses → retry for GET/HEAD/OPTIONS on any 5xx; for writes (POST/PUT/PATCH/DELETE) only on 502/503/504 (a 500 on a write is returned as-is to avoid double-processing). Clients should send an Idempotency-Key header on important writes (payments, orders, offers, AI tasks); the Worker forwards it untouched so retries are safe.
Failover ≠ marking a server dead (2026-08-06): the two are separate decisions. A request fails over per the rule above, but the origin is recorded unhealthy only on a connection error or 502/503/504. A plain 500 is one broken route on a live server — that request still retries elsewhere, but the origin keeps its traffic, so an application bug cannot silently become a routing change until the next cron heals it.
Shared health memory (KV + cron): cron runs live GET /health on every origin (limited concurrency, so a pile of dead A hosts does not starve healthy B checks) and writes alive/dead to Workers KV (HEALTH_KV, key health). Once a snapshot exists, requests prefer alive origins (weighted among the alive set). If KV says zero alive for a service (stale all-dead — e.g. site A offline and cron falsely marked B dead), the Worker falls back to try-all + failover and heals that service’s KV in the background — it does not hard-502 while a live origin can still answer. Live request failures patch KV immediately. Before the first cron run, try-all + per-request failover applies. Healthy = GET /health returns 2xx with JSON {"status":"ok"}.
KV write budget (2026-08-06): all health state is one KV key and KV allows roughly 1 write per second per key. A patch whose flags already match what is stored is skipped, so an outage under load cannot turn every failing request into a KV write. Write failures are swallowed — routing keeps working and /__lb/status still answers, with kvUpdated: false. KV is also eventually consistent across regions (up to ~60s), so one location can briefly route on slightly stale health; per-request failover covers that gap.
Worker-owned CORS: JSON errors from the Worker (404, 502, config errors) echo the request Origin so browsers do not show fake CORS when an origin is down.
Headers: adds X-Forwarded-Host, X-Forwarded-Proto, X-Load-Balancer upstream; X-Origin-Service and X-Origin-Server on normal HTTP responses.
WebSocket (2026-08-06): e.g. wss://api.kisum.io/chat/live-feed/connection/websocket → service chat, path /live-feed/... on the origin. Origin 101 upgrade responses are returned as-is (Workers cannot wrap status: 101 in new Response without a webSocket handle). Successful upgrades therefore omit X-Origin-* response headers.
Status: GET /__lb/status guarded by LB_STATUS_SECRET (Wrangler secret), runs live checks, writes those results into shared KV immediately (so routing updates without waiting for cron), and returns health grouped by service, plus kvCheckedAt, kvUpdated, and per-origin kvHealthy. Optional &server=a|b filters to that site only (server is echoed in the JSON; services with no matching origin are omitted):
```
{
  "status": "ok",
  "server": "b",
  "kvUpdated": true,
  "kvCheckedAt": "2026-07-09T07:00:00.000Z",
  "services": {
    "auth": [
      { "name": "auth-b", "healthy": true, "kvHealthy": true }
    ],
    "core": [{ "name": "core-b", "healthy": true, "kvHealthy": true }]
  }
}
```
Configuration: backends live in the SERVICES env var. Array → single-service; object (serviceName → array of origins, or { "origins": [...], "stripPrefix": false }) → multi-service. Each origin needs baseUrl; optional name, weight; healthUrl = {baseUrl}/health. baseUrl may carry a path prefix (https://host/api) — it is kept on real traffic as well as on the health probe. Origin names are the KV health keys, so they must be unique within a service; two origins on the same host (different ports) would both default to the hostname, and the Worker now fails config parse with duplicate origin name rather than letting them share one health flag. Set per target in wrangler-<target>.jsonc vars (real JSON) or in the Cloudflare dashboard (string); local dev via .dev.vars (string, shared across targets). Any number of origins per service. (Replaces the earlier single ORIGINS list as of 2026-07-07 — breaking.)

DNS requirement (breaks without this)

A Worker cannot fetch() a proxied (orange-cloud) hostname of its own Cloudflare zone (error 1042). Therefore:

Origin hostnames on a Cloudflare zone must be DNS only (grey cloud) and serve a publicly trusted TLS cert (Let’s Encrypt; Cloudflare Origin CA certs are not publicly trusted).
The Worker gets api-v2.kisum.dev via Custom Domain — never on an origin hostname.
Keep origins private: origins should accept traffic only from Cloudflare (IP allowlist / Authenticated Origin Pulls / shared secret header).
Orange-cloud alternative: global_fetch_strictly_public compatibility flag (commented in each wrangler-*.jsonc); grey cloud is the recommended default.

Full operator detail: modules/System-KisumAI-LoadBalancer/README.md.

AI chat origins — shared Postgres (Stage 2)

The AI deployment (wrangler-ai.jsonc, chat-a / chat-b) uses alive-only routing (Stage 1). For threaded chats (sessionId), both origins must use the same POSTGRES_URI pointing at one external Postgres (e.g. DigitalOcean Managed PostgreSQL) — not per-host bundled ai-memory-postgres. Otherwise the LoadBalancer can route the same session to different servers and lose chat history.

Health: when POSTGRES_URI is set, ai-agent-chat /health runs SELECT 1; Postgres down → 503 (origin marked unhealthy).
Local dev: docker compose --profile local-memory up keeps bundled Postgres.
Runbook: modules/AI-Systems/Chat-Kisum-MCP-Node/README.md § Production: shared chat memory; platform page Chat-Kisum-MCP-Node.