# LKG: Last Known Good Most outages are caused by a change. LKG puts a service back on its last known-good config within seconds, with no model in that path. Then it asks a model what broke and ships a fixed config that keeps the change the team actually wanted. **User:** the on-call engineer on a small team with no 24/7 coverage. A bad config deploy has taken the service down. They need it back up, and they need to know why it broke. **Result in the recorded cloud run:** the shop was back online **6.6 s** after its first failed health check. The intended change (a Black Friday banner) was live again, without the bad setting, at **13.2 s**. The incident was recorded in PostgreSQL. No human was involved.  Video of the full run (30 s, 1920×1080): [`docs/lkg-demo-cloud.mp4`](docs/lkg-demo-cloud.mp4) ## How it works ``` laptop ── deploy/lkg.sh (SSH, LKG's own key) ──┐ ▼ Agent VM · Seattle App VM · Seattle (Cloud Firewall, inbound DROP) ┌──────────────────────┐ GET /health :80 ┌────────────────────────────────────────────┐ │ agent.py the healer│ ────────────────────▶ │ app.py the shop; re-reads app.toml │ │ dashboard.py screen │ │ on every request │ │ (127.0.0.1:8700) │ /deploy /rollback │ sidecar.py the only writer of app.toml; │ │ │ /state :9000 ─────▶ │ :9000 open to the agent VM only│ └──┬────────────┬──────┘ + X-LKG-Token └──────────────────┬─────────────────────────┘ │ │ │ put / list / get ▼ ▼ ▼ Akamai Managed PostgreSQL · Seattle Object Storage · Chicago Inference incidents table snapshots/<UTC time>.toml qwen3-8-27b (key limited to this one bucket) ``` ### One incident, step by step | # | Step | What happens | Model? | |---|---|---|---| | 1 | **Deploy** | Someone pushes a config through the sidecar's `/deploy`. The sidecar checks that it parses as TOML, then atomically swaps `app.toml` (temp file + `os.replace`). The app picks it up on its next request. No restart. | no | | 2 | **Detect** | `agent.py` checks `GET /health` every 2 s. Three failures in a row (about 4 s) open an incident. The agent reads the bad config and the last 50 log lines from `/state`, then inserts an `open` incident row. | no | | 3 | **Roll back** | `/rollback` takes the newest object under `snapshots/` in the bucket and swaps it in. The agent waits for 3 consecutive healthy checks, 1 s apart. Time to recovery is measured from the first failed check to that point. | no | | 4 | **Triage** | One `qwen3-8-27b` call (plus one retry on error) with the logs, the bad config and the restored good config. It returns JSON `{cause, summary, confidence, patched_config}`. | **yes** | | 5 | **Patch** | If the cause is known and confidence ≥ 0.5, the patch is checked: it must parse, have the same keys as the good config, and keep each value's type. It is then deployed through the same `/deploy` a human uses. | no | | 6 | **Verify** | 3 consecutive healthy checks within 20 s gives `patched`. Otherwise the agent rolls back once more and stops (`patch_failed`). An unknown cause or low confidence leaves it rolled back for a human (`rolled_back`). | no | ### Why the newest snapshot is safe to restore The sidecar saves a config to the bucket only after it has been healthy for **30 s** in a row (`SNAPSHOT_AFTER_S`), and only if `app.toml` hasn't changed in that time. A bad deploy logs `snapshot skipped` and never reaches the bucket. So a rollback can take the newest object without checking anything. A patch that stays healthy for 30 s becomes the next known-good snapshot. ### Design choices - **Rollback never depends on the model.** Detect → roll back → recover uses only HTTP and Object Storage. The model runs after the service is already healthy. Inference, database and event-file errors are caught and logged; they never stop a rollback. - **The healer doesn't run on the box it heals.** The agent is on its own VM. The sidecar on the App VM only carries out deploys, rollbacks and snapshots. - **Least privilege.** The App VM holds only the bucket key, limited to that one bucket. The Agent VM holds the Inference key, the PostgreSQL password and the sidecar token. - **Network lock-down.** Both firewalls drop inbound traffic by default. Port 9000 (sidecar) accepts only the Agent VM. Port 80 accepts your IP and the Agent VM (or everyone, with `-var app_open_to_world=true`, so judges can open the shop). SSH accepts only your IP. PostgreSQL allows only your IP and the two VMs. - **It never loops.** Each incident gets at most one patch, followed by at most one more rollback. ## Akamai services used | Service | Role | |---|---| | 2 × Linode Nanode (Seattle) + 2 Cloud Firewalls | App VM (shop + sidecar), Agent VM (agent + dashboard) | | Object Storage (Chicago) | Known-good snapshots, kept off both VMs | | Managed PostgreSQL 17 (Seattle) | One row per incident: health reason, logs, bad config, snapshot, time to recovery, cause, summary, confidence, patch, outcome (`open` / `rolled_back` / `patched` / `patch_failed` / `failed`) | | Akamai Inference, `qwen3-8-27b` | `temperature=0`, JSON-schema response format, `max_tokens=500` | ## Run the demo (Akamai Cloud) Prerequisites: [uv](https://docs.astral.sh/uv/), OpenTofu, `curl`, `ssh`. `nix develop` (or direnv) provides all of them. Copy `.env.example` to `.env` and fill in `LINODE_TOKEN` and the `AKAMAI_*` values. Tofu generates everything else. ```sh export LINODE_TOKEN=$(grep ^LINODE_TOKEN= .env | cut -d= -f2-) tofu -chdir=infra init tofu -chdir=infra apply -var name=<you> -var "my_ip=$(curl -s https://ipv4.icanhazip.com)" # 2 VMs, 2 firewalls, PostgreSQL, bucket, scoped key, SSH key, # and the gitignored .env.app / .env.agent / .env.cloud deploy/lkg.sh setup # copy code, start app + sidecar (systemd), deploy good.toml, # wait for its first snapshot (~35 s) deploy/lkg.sh agent # terminal 1: agent.py in tmux "lkg" on the agent VM (detach: Ctrl-b d) deploy/lkg.sh dashboard # terminal 2: dashboard in tmux "dash" on the agent VM, # tunneled to http://localhost:8700 ``` Open http://localhost:8700 and press **Break: db pool = 0** (or `b`). The screen shows the incident live: - the shop as a shopper sees it; - a "back online" clock and a "change shipped" clock; - a map of the Akamai services that lights up with each call the agent makes. Every deploy, rollback and patch enters the App VM through the firewall gate on :9000, reaches the sidecar, and ends with the sidecar writing `app.toml` and the shop reading it; - a timeline with the no-model rollback bracketed; - a bad / known-good / patched diff of what the model changed; - the agent's log. | Dashboard control | Effect | |---|---| | **Break: db pool = 0** (`b`) | Deploys `config/bad-pool.toml`: the Black Friday banner plus `db_pool_size = 0` | | **Break: timeout = 10 ms** (`t`) | Deploys `config/bad-timeout.toml`: `request_timeout_ms = 10` | | **Reset to good** (`g`) | Deploys `config/good.toml`. Wait 30 s for its snapshot before the next break. | The agent's terminal shows the same run: ```text WATCH failure 3/3: db pool exhausted: size=0 INCIDENT #17 db pool exhausted: size=0 ROLLBACK restored snapshots/20261120T221824Z.toml ROLLBACK healthy again after 6.6 s TRIAGE cause=db_pool_size confidence=1.00 The deploy set db_pool_size to 0, causing the database connection pool to be exhausted and resulting in errors. Reverting this value to 16 resolved the issue. PATCH deployed, checking health PATCH healthy with the patched config ``` Other commands: ```sh deploy/lkg.sh push config/bad-pool.toml # break it without the dashboard (sent through the agent VM) deploy/lkg.sh history # last 5 incidents from PostgreSQL deploy/lkg.sh snapshots # known-good configs in the bucket deploy/lkg.sh recording # copy the newest dashboard recording to recordings/ tofu -chdir=infra destroy # delete every cloud resource ``` ```text $ deploy/lkg.sh history #17 2026-11-20 22:28:04 ttr=6.6s patched cause=db_pool_size db pool exhausted: size=0 #16 2026-11-20 22:17:20 ttr=6.5s patched cause=request_timeout_ms upstream timeout after 10ms ``` ### Offline backup The dashboard records every live session as JSON lines. A replay plays the last incident in the file on the same screen, with the original timing. It needs no network and no `.env`: ```sh uv run dashboard.py --replay docs/lkg-demo-cloud.jsonl # then press space (or p) to play again ``` Flags: `--speed` (replay speed), `--port` (default 8700), `--no-record`. ## Run it locally The sidecar needs a real Object Storage bucket (the `OBJ_*` values). PostgreSQL is optional: without `PGHOST`, the agent skips the database and incident numbers show as `#-`. ```sh cp .env.example .env # fill in; SNAPSHOT_AFTER_S=5 makes iteration faster uv run app.py # terminal 1: shop on :8080 uv run sidecar.py # terminal 2: sidecar on :9000 uv run agent.py # terminal 3 uv run dashboard.py # terminal 4, optional: http://localhost:8700 # seed through the sidecar (a copied app.toml is never snapshotted), wait SNAPSHOT_AFTER_S, then break it: curl -s -X POST -H "X-LKG-Token: $LKG_TOKEN" --data-binary @config/good.toml http://localhost:9000/deploy curl -s -X POST -H "X-LKG-Token: $LKG_TOKEN" --data-binary @config/bad-pool.toml http://localhost:9000/deploy ``` ## Evaluation `uv run eval.py` runs the agent's triage prompt (copied from `agent.py`) against 4 frozen incidents in `eval/cases.json`. One of them is an incident the config doesn't explain. A case passes when the cause is right and the patch reverts only that key, keeping every other change. For the unexplained case, it passes when the model hands it to a human (cause `unknown` or confidence < 0.5). ```text | case | expected | got | confidence | patch keeps other changes | PASS/FAIL | | ----------- | ------------------ | ------------------ | ---------- | ------------------------- | --------- | | pool | db_pool_size | db_pool_size | 1.00 | yes | PASS | | timeout | request_timeout_ms | request_timeout_ms | 1.00 | yes | PASS | | two-changes | db_pool_size | db_pool_size | 1.00 | yes | PASS | | unknown | unknown | unknown | 0.10 | n/a | PASS | 4/4 passed with qwen3-8-27b ``` Don't run the eval right before a demo: it uses the shared per-minute token budget. ## Repository layout | Path | What it is | |---|---| | `app.py` | The shop. Re-reads `app.toml` on every request. `/health` returns 503 with the reason (`db_pool_size ≤ 0` or `request_timeout_ms < 50`). | | `sidecar.py` | App VM. `/deploy`, `/rollback`, `/state`, all token-protected. Uploads known-good snapshots to the bucket. | | `agent.py` | Agent VM. Watches, rolls back, triages, patches, records. `--history` prints the last 5 incidents. | | `dashboard.py`, `dashboard.html` | The incident screen, live or replayed. Read-only, except for its Break/Reset buttons. | | `eval.py`, `eval/cases.json` | Frozen-incident eval for the triage prompt. | | `config/` | `good.toml` and the two bad deploys (`bad-pool.toml`, `bad-timeout.toml`). | | `infra/main.tf` | OpenTofu: every cloud resource, plus the gitignored `.env.*` files and SSH key. | | `deploy/` | `lkg.sh` (setup, push, agent, dashboard, recording, history, snapshots), cloud-init that installs uv, systemd units for app and sidecar. `agent.service` is installed but not enabled, because the demo runs the agent in tmux so you can watch it. | | `docs/` | Demo video, final screenshot, and the dashboard recording used for `--replay`. | | `LKG-PLAN.md` | The original design plan, kept for history. The code and this README are the current reference. | | `flake.nix`, `.envrc` | Nix dev shell: uv, Python, OpenTofu, linode-cli, psql, s3cmd, tmux. | Each `*.py` is a single-file uv script that declares its own dependencies (Python ≥ 3.11). ## Known limits - **Config-only.** It handles config changes only, and the patch can only revert keys to their known-good values. It covers one app. - **Plain HTTP between the VMs.** The sidecar token is sent unencrypted; the firewall is what keeps others out. - **"Known-good" means healthy for 30 s.** A failure that only shows up later would still be snapshotted. - **Empty bucket.** Until the first snapshot exists, `/rollback` returns 409 and the agent pages a human.
Built at [Seattle] Build with Akamai Cloud at GitHub Bellevue
LKG: Last Known Good
# LKG: Last Known Good Most outages are caused by a change. LKG puts a service back on its last known-good config within seconds, with no model in that path. Then it asks a model what broke and ships a fixed config that keeps the change the team actually wanted. **User:** the on-call engineer on a small team with no 24/7 coverage. A bad config deploy has taken the service down. They need it back up, and they need to know why it broke. **Result in the recorded cloud run:** the shop was back online **6.6 s** after its first failed health check. The intended change (a Black Friday banner) was live again, without the bad setting, at **13.2 s**. The incident was recorded in PostgreSQL. No human was involved.  Video of the full run (30 s, 1920×1080): [`docs/lkg-demo-cloud.mp4`](docs/lkg-demo-cloud.mp4) ## How it works ``` laptop ── deploy/lkg.sh (SSH, LKG's own key) ──┐ ▼ Agent VM · Seattle App VM · Seattle (Cloud Firewall, inbound DROP) ┌──────────────────────┐ GET /health :80 ┌────────────────────────────────────────────┐ │ agent.py the healer│ ────────────────────▶ │ app.py the shop; re-reads app.toml │ │ dashboard.py screen │ │ on every request │ │ (127.0.0.1:8700) │ /deploy /rollback │ sidecar.py the only writer of app.toml; │ │ │ /state :9000 ─────▶ │ :9000 open to the agent VM only│ └──┬────────────┬──────┘ + X-LKG-Token └──────────────────┬─────────────────────────┘ │ │ │ put / list / get ▼ ▼ ▼ Akamai Managed PostgreSQL · Seattle Object Storage · Chicago Inference incidents table snapshots/<UTC time>.toml qwen3-8-27b (key limited to this one bucket) ``` ### One incident, step by step | # | Step | What happens | Model? | |---|---|---|---| | 1 | **Deploy** | Someone pushes a config through the sidecar's `/deploy`. The sidecar checks that it parses as TOML, then atomically swaps `app.toml` (temp file + `os.replace`). The app picks it up on its next request. No restart. | no | | 2 | **Detect** | `agent.py` checks `GET /health` every 2 s. Three failures in a row (about 4 s) open an incident. The agent reads the bad config and the last 50 log lines from `/state`, then inserts an `open` incident row. | no | | 3 | **Roll back** | `/rollback` takes the newest object under `snapshots/` in the bucket and swaps it in. The agent waits for 3 consecutive healthy checks, 1 s apart. Time to recovery is measured from the first failed check to that point. | no | | 4 | **Triage** | One `qwen3-8-27b` call (plus one retry on error) with the logs, the bad config and the restored good config. It returns JSON `{cause, summary, confidence, patched_config}`. | **yes** | | 5 | **Patch** | If the cause is known and confidence ≥ 0.5, the patch is checked: it must parse, have the same keys as the good config, and keep each value's type. It is then deployed through the same `/deploy` a human uses. | no | | 6 | **Verify** | 3 consecutive healthy checks within 20 s gives `patched`. Otherwise the agent rolls back once more and stops (`patch_failed`). An unknown cause or low confidence leaves it rolled back for a human (`rolled_back`). | no | ### Why the newest snapshot is safe to restore The sidecar saves a config to the bucket only after it has been healthy for **30 s** in a row (`SNAPSHOT_AFTER_S`), and only if `app.toml` hasn't changed in that time. A bad deploy logs `snapshot skipped` and never reaches the bucket. So a rollback can take the newest object without checking anything. A patch that stays healthy for 30 s becomes the next known-good snapshot. ### Design choices - **Rollback never depends on the model.** Detect → roll back → recover uses only HTTP and Object Storage. The model runs after the service is already healthy. Inference, database and event-file errors are caught and logged; they never stop a rollback. - **The healer doesn't run on the box it heals.** The agent is on its own VM. The sidecar on the App VM only carries out deploys, rollbacks and snapshots. - **Least privilege.** The App VM holds only the bucket key, limited to that one bucket. The Agent VM holds the Inference key, the PostgreSQL password and the sidecar token. - **Network lock-down.** Both firewalls drop inbound traffic by default. Port 9000 (sidecar) accepts only the Agent VM. Port 80 accepts your IP and the Agent VM (or everyone, with `-var app_open_to_world=true`, so judges can open the shop). SSH accepts only your IP. PostgreSQL allows only your IP and the two VMs. - **It never loops.** Each incident gets at most one patch, followed by at most one more rollback. ## Akamai services used | Service | Role | |---|---| | 2 × Linode Nanode (Seattle) + 2 Cloud Firewalls | App VM (shop + sidecar), Agent VM (agent + dashboard) | | Object Storage (Chicago) | Known-good snapshots, kept off both VMs | | Managed PostgreSQL 17 (Seattle) | One row per incident: health reason, logs, bad config, snapshot, time to recovery, cause, summary, confidence, patch, outcome (`open` / `rolled_back` / `patched` / `patch_failed` / `failed`) | | Akamai Inference, `qwen3-8-27b` | `temperature=0`, JSON-schema response format, `max_tokens=500` | ## Run the demo (Akamai Cloud) Prerequisites: [uv](https://docs.astral.sh/uv/), OpenTofu, `curl`, `ssh`. `nix develop` (or direnv) provides all of them. Copy `.env.example` to `.env` and fill in `LINODE_TOKEN` and the `AKAMAI_*` values. Tofu generates everything else. ```sh export LINODE_TOKEN=$(grep ^LINODE_TOKEN= .env | cut -d= -f2-) tofu -chdir=infra init tofu -chdir=infra apply -var name=<you> -var "my_ip=$(curl -s https://ipv4.icanhazip.com)" # 2 VMs, 2 firewalls, PostgreSQL, bucket, scoped key, SSH key, # and the gitignored .env.app / .env.agent / .env.cloud deploy/lkg.sh setup # copy code, start app + sidecar (systemd), deploy good.toml, # wait for its first snapshot (~35 s) deploy/lkg.sh agent # terminal 1: agent.py in tmux "lkg" on the agent VM (detach: Ctrl-b d) deploy/lkg.sh dashboard # terminal 2: dashboard in tmux "dash" on the agent VM, # tunneled to http://localhost:8700 ``` Open http://localhost:8700 and press **Break: db pool = 0** (or `b`). The screen shows the incident live: - the shop as a shopper sees it; - a "back online" clock and a "change shipped" clock; - a map of the Akamai services that lights up with each call the agent makes. Every deploy, rollback and patch enters the App VM through the firewall gate on :9000, reaches the sidecar, and ends with the sidecar writing `app.toml` and the shop reading it; - a timeline with the no-model rollback bracketed; - a bad / known-good / patched diff of what the model changed; - the agent's log. | Dashboard control | Effect | |---|---| | **Break: db pool = 0** (`b`) | Deploys `config/bad-pool.toml`: the Black Friday banner plus `db_pool_size = 0` | | **Break: timeout = 10 ms** (`t`) | Deploys `config/bad-timeout.toml`: `request_timeout_ms = 10` | | **Reset to good** (`g`) | Deploys `config/good.toml`. Wait 30 s for its snapshot before the next break. | The agent's terminal shows the same run: ```text WATCH failure 3/3: db pool exhausted: size=0 INCIDENT #17 db pool exhausted: size=0 ROLLBACK restored snapshots/20261120T221824Z.toml ROLLBACK healthy again after 6.6 s TRIAGE cause=db_pool_size confidence=1.00 The deploy set db_pool_size to 0, causing the database connection pool to be exhausted and resulting in errors. Reverting this value to 16 resolved the issue. PATCH deployed, checking health PATCH healthy with the patched config ``` Other commands: ```sh deploy/lkg.sh push config/bad-pool.toml # break it without the dashboard (sent through the agent VM) deploy/lkg.sh history # last 5 incidents from PostgreSQL deploy/lkg.sh snapshots # known-good configs in the bucket deploy/lkg.sh recording # copy the newest dashboard recording to recordings/ tofu -chdir=infra destroy # delete every cloud resource ``` ```text $ deploy/lkg.sh history #17 2026-11-20 22:28:04 ttr=6.6s patched cause=db_pool_size db pool exhausted: size=0 #16 2026-11-20 22:17:20 ttr=6.5s patched cause=request_timeout_ms upstream timeout after 10ms ``` ### Offline backup The dashboard records every live session as JSON lines. A replay plays the last incident in the file on the same screen, with the original timing. It needs no network and no `.env`: ```sh uv run dashboard.py --replay docs/lkg-demo-cloud.jsonl # then press space (or p) to play again ``` Flags: `--speed` (replay speed), `--port` (default 8700), `--no-record`. ## Run it locally The sidecar needs a real Object Storage bucket (the `OBJ_*` values). PostgreSQL is optional: without `PGHOST`, the agent skips the database and incident numbers show as `#-`. ```sh cp .env.example .env # fill in; SNAPSHOT_AFTER_S=5 makes iteration faster uv run app.py # terminal 1: shop on :8080 uv run sidecar.py # terminal 2: sidecar on :9000 uv run agent.py # terminal 3 uv run dashboard.py # terminal 4, optional: http://localhost:8700 # seed through the sidecar (a copied app.toml is never snapshotted), wait SNAPSHOT_AFTER_S, then break it: curl -s -X POST -H "X-LKG-Token: $LKG_TOKEN" --data-binary @config/good.toml http://localhost:9000/deploy curl -s -X POST -H "X-LKG-Token: $LKG_TOKEN" --data-binary @config/bad-pool.toml http://localhost:9000/deploy ``` ## Evaluation `uv run eval.py` runs the agent's triage prompt (copied from `agent.py`) against 4 frozen incidents in `eval/cases.json`. One of them is an incident the config doesn't explain. A case passes when the cause is right and the patch reverts only that key, keeping every other change. For the unexplained case, it passes when the model hands it to a human (cause `unknown` or confidence < 0.5). ```text | case | expected | got | confidence | patch keeps other changes | PASS/FAIL | | ----------- | ------------------ | ------------------ | ---------- | ------------------------- | --------- | | pool | db_pool_size | db_pool_size | 1.00 | yes | PASS | | timeout | request_timeout_ms | request_timeout_ms | 1.00 | yes | PASS | | two-changes | db_pool_size | db_pool_size | 1.00 | yes | PASS | | unknown | unknown | unknown | 0.10 | n/a | PASS | 4/4 passed with qwen3-8-27b ``` Don't run the eval right before a demo: it uses the shared per-minute token budget. ## Repository layout | Path | What it is | |---|---| | `app.py` | The shop. Re-reads `app.toml` on every request. `/health` returns 503 with the reason (`db_pool_size ≤ 0` or `request_timeout_ms < 50`). | | `sidecar.py` | App VM. `/deploy`, `/rollback`, `/state`, all token-protected. Uploads known-good snapshots to the bucket. | | `agent.py` | Agent VM. Watches, rolls back, triages, patches, records. `--history` prints the last 5 incidents. | | `dashboard.py`, `dashboard.html` | The incident screen, live or replayed. Read-only, except for its Break/Reset buttons. | | `eval.py`, `eval/cases.json` | Frozen-incident eval for the triage prompt. | | `config/` | `good.toml` and the two bad deploys (`bad-pool.toml`, `bad-timeout.toml`). | | `infra/main.tf` | OpenTofu: every cloud resource, plus the gitignored `.env.*` files and SSH key. | | `deploy/` | `lkg.sh` (setup, push, agent, dashboard, recording, history, snapshots), cloud-init that installs uv, systemd units for app and sidecar. `agent.service` is installed but not enabled, because the demo runs the agent in tmux so you can watch it. | | `docs/` | Demo video, final screenshot, and the dashboard recording used for `--replay`. | | `LKG-PLAN.md` | The original design plan, kept for history. The code and this README are the current reference. | | `flake.nix`, `.envrc` | Nix dev shell: uv, Python, OpenTofu, linode-cli, psql, s3cmd, tmux. | Each `*.py` is a single-file uv script that declares its own dependencies (Python ≥ 3.11). ## Known limits - **Config-only.** It handles config changes only, and the patch can only revert keys to their known-good values. It covers one app. - **Plain HTTP between the VMs.** The sidecar token is sent unencrypted; the firewall is what keeps others out. - **"Known-good" means healthy for 30 s.** A failure that only shows up later would still be snapshotted. - **Empty bucket.** Until the first snapshot exists, `/rollback` returns 409 and the agent pages a human.
Keep exploring what builders shipped.
Bainbridge
hedgefinder
kalshi hedge
Sahil Bhatt's team
Retry Storm
# CascadeIQ Virtual resilience audit: **AI interprets** an incident → **deterministic simulation** → **AI mitigation** → **verified rerun**. Hackathon prototype (Akamai Seattle AI Hackathon, Nov 2026). **Repo:** https://github.com/Sahil9477/CascadeIQ **Demo:** http://127.0.0.1:8000/ (`uv run serve.py`) **Active branch:** `main` ## Judging ```text User and task: CS / cloud students (and judges) learning why retry storms break checkout paths; they run Lab 01, see the outage, pick a mitigation, and verify it. Benefit: Turns an abstract SRE failure into a hands-on lab: AI proposes, deterministic sim proves, shareable /runs/{uuid} for grading. Services: Akamai Inference (Qwen interpret + recommend; Gemma chart review); Managed Postgres (persist audit runs / share links); Linode Types API (published compute pricing beside outage loss). App can run on a Linode VM (deploy/) — laptop demo uses serve.py. Inference: qwen3-8-27b for scenario JSON + allowlisted mitigation; gemma-4-26b-a4b-it reads the before chart as a hint (metrics win). Checked by same-seed Python verify and fixture vs live flags (inference_used). Technical choice: Simulator owns truth — models never invent metrics; allowlist + re-sim with same seed so AI can’t fake a win. Our idea: Scenario-first teaching lab (Build → Cost → Break → Fix → Learn), two-model diagnose with Python as the judge. Demo: Start Lab → untick fixtures → Run audit → Ask Qwen → Verify → Save → open /runs/{uuid}. Known limits: No public Linode app URL until Nanode is provisioned; shared minute token budget (429); retry_cap_reduction may show clearer wins than backoff alone. ``` ## Team | Owner | Owns | |---|---| | **Sahil** | Deterministic simulator, Qwen interpret/recommend, FastAPI `/api/interpret\|audit\|recommend\|verify`, contracts/fixtures, Lab 01 + catalog/cost panel | | **Danny** | Interactive dashboard (`static/` instrument), Postgres `/api/runs`, `serve.py`, Linode `deploy/` | ## Quick start ```bash cd ~/Desktop/CascadeIQ git checkout main cp .env.example .env # paste AKAMAI_API_KEY + PG* from workshop / Cloud Manager (never commit .env) uv sync --extra dev uv run serve.py # Lab UI: http://127.0.0.1:8000/ # OpenAPI: http://127.0.0.1:8000/docs # Health: http://127.0.0.1:8000/api/health # Engine-only (no Postgres overlay): # uv run uvicorn cascadeiq.api.main:app --reload --port 8010 uv run pytest -q uv run python scripts/demo_flow.py # live Qwen + sim uv run python scripts/demo_flow.py --fixture uv run python scripts/smoke_runs_db.py # POST/GET /api/runs when PG* set ``` ### Lab 01: The Retry Storm Open `/` after `serve.py` starts. **Start Lab** loads the Gateway→Orders→Payments demo. Leave **Use fixtures** unticked for live Qwen/Gemma; tick it for offline rehearsal (`inference_used` is always shown honestly). Flow: Start Lab → **Run resilience audit** → **Ask Qwen for a mitigation** (Gemma chart review → informed recommend) → **Verify** → **Save** → open `/runs/{uuid}`. Play streams the audit timeline only after a baseline exists. Cost panel uses published Linode types (`GET /api/catalog/compute?region=us-sea`) beside simulated outage loss — never invents prices. ## API See [`docs/cascadeiq-api-contracts.md`](docs/cascadeiq-api-contracts.md) and [`contracts/`](contracts/). - `POST /api/interpret` — Qwen → scenario JSON - `POST /api/audit` — baseline sim time series + metrics - `GET /api/catalog/compute?region=us-sea` — published Linode types (`source: live|unavailable`) - `POST /api/chart/review` — Gemma reads the chart (advisory) - `POST /api/recommend` / `POST /api/recommend/informed` — allowlisted mitigation (optional visual note) - `POST /api/verify` — same seed + mitigation deltas - `POST /api/runs` · `GET /api/runs/{uuid}` — Postgres persist + share (via `serve.py`) Mitigations allowlist only: `bounded_exponential_backoff_seeded_jitter` | `retry_cap_reduction`. ## Layout ``` cascadeiq/engine/ # simulator + schemas cascadeiq/inference/ # Akamai Qwen client cascadeiq/api/ # FastAPI intelligence endpoints checks/ # Gemma chart review + informed recommend (Danny) cloud/ # Postgres /api/runs (Danny) serve.py # production entry: app + runs + static static/ # Lab UI + instrument deploy/ # Linode service + deploy.sh contracts/ fixtures/ ``` Do not commit `.env` or API keys. See [`deploy/PROVISIONING.md`](deploy/PROVISIONING.md) for Linode + Managed Postgres checklist.
Heyo
CloudHand
# CloudHand CloudHand gives an AI agent a computer it can use from a browser. Run jobs on an Akamai cloud computer that keeps working after you close the tab, or use your Mac to open local apps and run tasks with your installed Codex skills. ## Try the demo 1. Open the [public judge workspace](https://cloudhand-judges.66-175-217-106.sslip.io/). No sign-in is required. 2. Ask **Calculate 27 × 14 in Calculator** and choose **Confirm action**. 3. CloudHand opens Calculator on Akamai and reports the result, **378**, in the chat. You can also find the completed run under **Akamai computer → Runs**. [Watch the 35-second walkthrough](CloudHand-Demo.mp4) ## Use your Mac Open the [local Mac workspace](http://127.0.0.1:4317/) while CloudHand is running on your Mac. Try **Open Notes on my Mac** or **Open Downloads in Finder**. Your Mac needs to be awake and have the permissions required for the action. To use an installed Codex skill, sign in to the Codex CLI on your Mac, choose **Model → Codex on this Mac**, then select a skill under **+ → Plugins & skills**. ## Run from source From the CloudHand repository, use Node.js 24, npm, and Python 3.10 or later. ```sh npm ci npm run build npm start ``` Open [http://127.0.0.1:4317/](http://127.0.0.1:4317/). Use `npm test` to run the automated checks.
