I'm Panth Patel. I build a lot of side projects, and Firebase has been my go-to for them. It has everything, and the free tier is generous.

The problem shows up late

The trouble starts when a side project stops being one. Once I wanted to run something for a company or at scale, I found no overnight way off. The app is written against Firebase's SDK, and the schema exists only in the data.

So I wanted a layer I could adopt on day one, while Firebase is still the right choice, that keeps the exit open.

Same calls, one more file

The function names and call shapes are Firebase's. The imports change:

import { collection, getDocs, orderBy, query, where } from '@panth977/icebase/firestore';
import { db } from './lib/icebase';
import { products } from '../icebase/schema';

const q = query(collection(db, products), where('category', '==', 'shoes'), orderBy('priceCents'));
const snap = await getDocs(q);   // snap.docs[0].data() is typed from the schema

The one difference you'll feel is key paths. Nested fields are arrays, where(['address', 'city'], ...), because 'rating.avg' is one key.

icebase.config.ts is the new file. It holds the schema, rules, functions, hosting and, per environment, the mode of each service:

environments: {
  local: { mode: 'emulator' },
  prod: { mode: 'firebase' },
}

The schema is the real feature

Firebase accepts anything, and that is why it is fast to start. AI makes this sharper. Building with AI is fine, but a good developer still looks at the interfaces and the schema to find bugs before the build. A loose database gives you nothing to look at.

IceBase asks for the exact shape of every collection, Realtime Database path and storage path, and for every field you filter on. Writes are type-checked and validated at runtime in every mode. A query whose index isn't declared fails at compile time:

MissingIndex<"category" | "stock">

It is also refused at runtime, in every mode, and icebase indexes check lists every such query for CI.

The later payoff is the migration. On the day you move to Postgres, you don't need to run an AI over your data to discover what the schema was. It has been in your repo the whole time.

If the project already exists, icebase init --import DIR --project ID --sample N drafts a schema from sampled documents, and learning mode reports undeclared collections and missing indexes instead of refusing them.

Three drivers

• firebase: the real Firebase SDKs, with the type, schema and index checks on top.
• emulator: my own IceBase server on SQLite. I found Firebase's emulator slow and wanted one good enough for production-like testing. It enforces your rules in Firebase's own syntax, and those same files deploy unchanged to Firebase.
• vm: the same server on Postgres, self-hosted.

Modes mix per service, so one service can move while the rest stay put. Going between the two server stores is one command, in either direction:

icebase migrate --from sqlite:.icebase --to env:DATABASE_URL

IceBase also has its own console for every mode, so you don't have to open Firebase's.

To keep me honest, the conformance suites run against Firebase's emulators, the IceBase server and a live Firebase project, and every IceBase-server suite also passes on Postgres. My design editor, Digital Canvas, runs on IceBase in firebase mode.

What is not built

• S3 for stored files. In vm mode they stay on disk today, and icebase migrate does not copy them.
• A Docker image, login and deploy to a VM, and backups.
• Import, export and link. These are what will make the move from a live Firebase project an overnight job, and they are next on the roadmap.

So the overnight move is the goal, and the pieces under it are in place. It is not a claim about today.

Have you moved a real app off Firebase? What was the part you did not see coming?

I'm Panth Patel. I build backends and developer tools, and this is a project of my own.

Why build this at all

As a graphics editor, Canva is simple. That is a compliment, and it is also why I think the editor should be free. The templates are great and they may never be free, because someone made them. The framework, the animations and the kit are different. Those are software, and software can be opened.

So I split the problem in two: an open editor now, and a template catalogue that grows over time. I set one rule on the content side. Nothing in the library is generated. Templates, stickers and palettes come from open catalogues, with credit, and the 62 mockups are real photographs.

The file is the product

A design is one .dcanvas file. It is a zip:

mimetype         application/vnd.digital-canvas+zip
document.json    { format: "digital-canvas", version: 1, width, height, pages: [...], assets: {...} }
assets/<hash>    image bytes, named by content hash
thumbnail.png    preview of page 1

document.json is plain, documented JSON, so a script can read and write a design without the editor. You choose where the files live: this browser, a folder on your computer, or your own Google Drive. Drive access uses the drive.file scope only, so the app sees what it created and what you picked.

The backend stores a user, a login count and a Drive refresh token. Your designs never pass through it. That backend runs on IceBase, which is another thing I built and a separate post.

One renderer everywhere

Drawing goes through CanvasKit, which is Skia compiled to WebAssembly. The UI around it is Svelte 5. packages/render has no DOM in it, so the same code draws in the browser and in Node, and the unit tests render real PNGs.

That choice pays off in export. Live playback, the web MP4 export and the agent's exporter all call the same frame function, so what you see while scrubbing is what lands in the file. PNG, JPG, SVG, PDF (raster or vector) and MP4 all come out of this one engine.

Today that covers 2,000+ Google Fonts, 270+ size presets, 17 adjustments, 35 filters, tables, charts, QR codes and a video timeline. The background remover (BiRefNet) and Magic Eraser (LaMa) run on your device. The AI features use your own key, and that key goes only to its provider.

Give it to your agent

This is the part I care about most. You spend hours learning an editor. An agent should be able to use that same editor for you, and leave you a design you can open and finish by hand.

packages/mcp is an MCP server with 42 tools. Any client with remote MCP and OAuth can connect with one URL:

claude mcp add --transport http digital-canvas https://digital-canvas-p977.web.app/mcp

The agent searches your uploads and free stock, composes the page, calls render_page to look at the result, and calls check_design, which measures overflow, contrast on the real pixels, cut-off text and tiny fonts. Then it fixes what failed and exports. The tools run on your own device, and the server is only a relay.

That loop is why it has eyes: the agent sees what it made and gets a measured critique before it hands anything back. The result is an ordinary .dcanvas file, not a flattened image.

What is not there yet

It is web only today. The next phase is a version you run locally from any folder, with no login and no Drive setup. After that come the tools people expect from a modern editor: Instagram-style edit animations, hooks, more mockups and many more templates.

The repo is Apache-2.0. If you try it, I'd like to know one thing: what is the first thing you tried to do that it could not?

OPM is the open-source tool I built to run on-premise Docker deployments: a cloud console, and one Go agent per server with its own GUI. Three things changed.

1. The browser tunnel is end-to-end encrypted

Some sites have no internet at all. For those, the agent borrows the browser of the person using its GUI: the tab fetches from the cloud on the agent's behalf.

Until this change, every request and response passed through that tab in clear. That included commits with secret values, and on a LAN the agent GUI may be plain http.

Now the agent seals each request to an X25519 key on the cloud, and that key has to be certified by a release key pinned in the agent's config. The cloud answers with an AES-256-GCM stream. The tab carries ciphertext.

There is no fallback. If the key is missing or the signature is wrong, the call fails and the reason shows on the Internet page. An agent with no pinned key can be told to run in clear, and that is logged every time a tunnel starts.

The security review changed the design once. A replayed request gets a second answer under the same key, and with the plain design that reuses key and nonces, which opens the first answer. So each response now starts with a random 32-byte salt that derives its own key.

The tab can still see that a call happened, its size and its timing, and it can drop calls. Relays between agents on the mesh are signed, not encrypted. Agents older than this stay in clear until they are updated and have the release key, which the install script adds and opm-agent upgrade does not.

2. The agent GUI on the cloud's own domain

Each site's agent GUI now opens from the cloud console, at a path under OPM's own domain:

<cloud>/app/deployments/{deploymentId}/remote/{siteSlug}/_opm/…

You log in to the cloud once. There is no agent key to find and no extra port for the client to open, because the site asks the cloud for work over outbound HTTPS and the cloud never calls a site. Every action is recorded on the site under the person's cloud email, and the GUI shows "through the cloud".

This one is on by default. The client switches it off in the agent GUI on the site, and open remote sessions end at once. A remote session can't flip that switch; the call answers 403.

3. Port tunnels

A container port, the site's edge or a TCP port on the server can be forwarded to a port on the cloud's VM. It carries raw TCP, so a website, an API or a database client all work, and a site with no cloud route of its own can be served through a peer.

You add a tunnel in the console, and it does nothing there. It stays off until someone switches it on in that site's agent GUI, under Agent › Tunnels. Each site keeps its own switches.

Two limits, because the switch is only as good as what sits around it:

• The tunnel switch also works from the agent GUI opened through the cloud. So a client who wants the decision to stay on their side should turn the remote GUI off as well.
• The public port has no login of its own. Whoever reaches it reaches the target, so the target's own auth is what stands there.

Why the switches sit on the client's side

Local deployment is why these clients buy Envizom in the first place. They keep the data, and now they also decide how much of a path to it exists. If a client isn't comfortable with any of this, the same agent GUI turns it off and the deployment keeps running without it.

The code is at github.com/Panth977/on-premise-manager, Apache-2.0. The sealing is in SPEC-final §9.10, the two tunnels in §21 #118 and #125.

That is Orch2, the development pipeline we run on Envizom at Oizom, where I'm a Tech Lead. One change usually touches a Go backend, a Svelte frontend, a UI library and 3 driver repos. Orch2 turns a kanban board into the control panel: a plain script runs the pipeline, Claude agents only do the coding, and every ticket gets its own sandboxed copy of the whole product. The Envizom code is proprietary, so this is the design, not the source.

Three things stay separate. The tracker is a product. Orch2 is a client of it. The agent is what Orch2 runs.

A script, not an agent

The orchestrator is ordinary TypeScript: predictable, cheap and easy to debug. An LLM starts only when there is code to write or a conflict to resolve. The board is the whole UI. There are no dashboards and no CLI for users, just stages, the thread and question cards, and orch replies in the thread with links, PRs and preview URLs. A human types one field, Src Branch. The ticket branch is the ticket key in every repo, and everything else is derived.

Every agent turn posts a short receipt and adds its token use to a board aggregate, so cost shows per ticket on the card and per day in analytics. The aggregate is stored in millions because 798,400 on a card was unreadable. It now says 0.8.

Each developer runs their own orch, with its own board agent and token, so assigning that agent to a ticket hands the ticket to that machine. The same codebase has a preview mode that runs once, on the dev server, for production previews.

A copy of the product per ticket

A workspace is 6 repo clones, its own data and its own stack on its own block of ports. Git uses a bare mirror of each repo and clones with --reference --dissociate, so a checkout takes seconds and owns its objects. GitHub auth goes per command as an HTTP header, so no credential is written to disk in a workspace.

One front router on a single port routes by hostname: <ticket>.localhost for the app and code-<ticket>.localhost for code-shell, the browser IDE. Browsers resolve *.localhost themselves, so a new ticket needs no DNS record and no proxy reload.

Data is real and safely copied. SQLite comes through a token-gated read-only file server as a VACUUM INTO snapshot, because copying a live WAL database gives a corrupt file. Downloads are checked by magic bytes, so an HTML error page can never be saved as a database.

Five layers around the agent

A ticket agent can run commands, edit, build and test. It can't push, reach secrets or touch anything outside its ticket. A gap in one layer is caught by the next:

1. Scrubbed environment. Every secret removed, for the agent, the workspace stack and the IDE.
2. OS sandbox on every shell command. Seatbelt on macOS, bubblewrap on Linux. Writes only in the workspace and build caches, network limited to localhost, package registries and GitHub.
3. A guard on every tool call. The same policy twice, in the SDK permission callback and a PreToolUse hook. It blocks git push, branch switches, reset --hard, ssh and piping downloads into a shell, orch-managed files are read-only, and every denial returns the reason.
4. A narrow tool surface. Only orch's own MCP server: ticket and thread, plan, stack control, endWork. No connectors, and no web fetch because it could leak ticket text.
5. Gates. endWork is refused until the plan is finished, the stack is healthy and the merge is clean. Orch commits and pushes only after a human moves the card to Git Push.

What production taught us

The first row decided a design rule: infra failures go to a person, never an agent, because an agent can't fix a firewall and will burn tokens trying.

What in your pipeline still lets an agent spend tokens on a problem it can't solve?

I'm Panth Patel, Tech Lead on Envizom at Oizom. This post is about process, not code.

What the job turned out to be

I started as a backend developer. I could debug anything in the stack, and I was comfortable helping with product design and the roadmap. None of that prepared me for a product that sells to clients.

Some features were wanted right now or the client wouldn't buy. Others were insistent suggestions that could sit in the backlog for months before anyone asked for an update. There was no cap on how many came in. As a dev you think everything is doable, until you realise you are no longer a proof of concept and people expect professional experience.

And I kept running into one question. If you expect no bugs and a stable release, how can you commit to a client on a 1-week timeline?

We were selling almost everything, about 80-90% of what clients asked for, and the team was praised for it many times. But every deadline slipped, and every release took new features at the last moment. Clients expected a stable system while we still acted like a startup. To scale, Envizom had to become a separate product, with client customisation in a proper framework instead of production code changes.

So I spent 2 months reading how professionals run this: how they scale, how they set KRAs and KPIs, how big tech manages a codebase with frequent releases. Some of it was too much for us and some too little. What we ended up with is good enough, and it works like a machine: each part has one job.

Shift 1: the document changes first

Every feature now starts as a 5-10 minute group discussion, without waiting for a standup. It defines the exact change to the product document before anyone implements anything. Product development happens there, and execution doesn't. If the feature needs research, a separate R&D task starts and blocks it.

Then the feature doc goes to three people at once:

The last row is the one I care about most. QA/QC reads only the latest feature doc, so a bug gets caught the way a client would hit it. Before this, the product truth always ended up being the code.

Each task also attaches two short videos: one explaining the product, one explaining the development interfaces and contract. The product owner and dev lead confirm the original clarifications were met, and only then does it go to QA/QC.

Shift 2: features have no deadlines

Nobody asks whether a feature is "immediate" any more, and nothing is "4 days" or "1 month". A feature goes into a release, and no work happens until a release is defined. A feature can join a release only when no R&D is left, just implementation. Only the release has a deadline and an estimated dev time.

The calendar has rules:

• At least 2 days of code freeze per release.
• 1 extra QA/QC day after the release.
• Releases happen on working days, with at least 2 working days ahead and no consecutive holidays.

Git follows the release too. Each release has a dev branch and a prod branch. Every feature and bug-fix branch comes off the release's dev branch and merges back to it. The prod branch has two environments: demo, recently cloned from the prod DB and running the upcoming release, and prod itself, updated on release date.

The comms come first as well. Change logs, product updates and press releases are defined at the start of release development and go out to client-facing colleagues, devs, QA/QC and leads.

Where in your team does a feature get its deadline: when someone asks for it, or when it joins a release?

I'm Panth, and I lead the software team at Oizom. I dropped out before the AI wave, because I figured the engineering was in solving problems in the real world, and my college made me choose between attendance and real-world applications. This post is about how the orchestrator works and why it is shaped the way it is.

Why tickets and not a terminal

I wanted to give my VM work from my phone, while travelling or waiting in a queue, with no laptop. The first version was a terminal in an app. It grew into a room full of agents, and three things went wrong:

1. I got attached to one long-running agent, and starting fresh ones was a chore.
2. The app was a door into the VM. If someone broke in, I would spend a long time cleaning up.
3. When the VM lost internet, I couldn't even queue a prompt.

Tickets fixed all three. Every ticket gets its own session, so there is no single agent to get attached to. The orchestrator runs on the machine and pulls work from the tracker, so nothing has to connect into that machine. The tracker is hosted apart from it, so I can raise tickets while the VM is offline, and my VM and my PC can take work from the same board.

The loop

An agent is a board member: a profile, a system prompt and a token that acts as that agent on one board. Assign a ticket to it and move the ticket to To do. From there:

To do  ──orch claims──▶ In progress ──agent reports "review"──▶ Review ──owner──▶ Done
                            │  ▲
           agent asks a     │  │ the answer (form or a comment) resumes
           question ────────▼  │ the same session
                          waiting

The orchestrator claims the ticket and starts claude -p in the repo the workspace points at, with the tracker's MCP tools. The agent reads the ticket and its thread, then publishes its plan as a task list on the ticket and marks each item as it goes.

When it needs a decision, it calls ask_question, which puts a form card in the thread, and ends its turn with status: "waiting". It does not poll. The answer, or any comment in the thread, starts the next turn with --resume on the same session, so the agent picks up with its whole history. Moving the ticket back to To do resumes it as rework. Unassigning it, or moving it anywhere else while the agent runs, stops the agent.

The brief every agent gets says it in two lines:

You run headless in [the repo] and nobody is watching your terminal. The owner only sees the ticket, so everything you want them to see goes on the ticket.

The orchestrator owns the ticket's state

The agent gets the tracker's MCP tools, minus the ones that decide where the ticket stands:

// What only the orch may do: the stage, the heartbeat, the inbox, assignment.
const DISALLOWED = ['move_ticket', 'update_ticket', 'assign_ticket', 'create_ticket', 'heartbeat', 'get_events', 'ack_events']
  .map((t) => `mcp__tm__${t}`).join(',');

The agent does the work and reports review, waiting or blocked. The orchestrator moves the ticket, beats, reads the event inbox and posts the receipt. So a ticket only reaches Review after the orchestrator has seen the agent report it done.

The receipt is one message per turn with what the turn cost. Claude Code's total_cost_usd is cumulative across --resume, so the orchestrator stores the last total and posts the difference. Because the agent pushes the cost of each turn, you can see where the money actually went — which ticket, which turn — instead of one number at the end of the month. One ticket of mine reached $402 over nine runs before the app showed any of it; the full story of that fix is in $402 on one ticket.

Health from a heartbeat

The orchestrator beats once a minute while an agent runs, and every fifth beat it re-reads the task list, so the ticket's dot shows progress like 3/12 · Running tests. The board never stores "this agent is dead". It derives health from the last status and the time since the last beat, in one function, recomputed every 15 seconds:

75 seconds is one missed beat plus a little slack. Only "working" can go stale: idle, done and error are statements that stay true until the next beat. After five minutes of silence the agent's owner gets one notification per silence.

One command layer

The app, the REST API (/v1, OpenAPI 3.1) and the MCP server (/mcp) all run the same commands. Nothing an agent can do is different from what a person can do, and every change is attributed, for example "Owner via Claude (MCP)". Tokens are scoped to one board and act either as you or as one of your agents.

What running it takes

Everything runs on one Firebase project: Firestore, Functions, Storage, Realtime Database, Auth and Hosting. "Self-hosted" here means your own Firebase project on the Blaze plan, not a Docker image. Locally, pnpm dev starts the emulators, seeds demo data and serves the app (Node 22, pnpm 9, Java 21 and the Firebase CLI).

Firebase, because the tracker had to live apart from the machines doing the work: if the VM is offline or the agents are down, I can still raise tickets. From there the app does live reads straight from Firestore, Auth and push come with it, and the orchestrator wakes on a Realtime Database revision stream, so when an agent comes back it picks up where it left off.

A workspace is a folder with workspace.json (repo, board, agent, how many agents at once), a .env with a Worker token for that agent, and brief.md with the project's rules:

node workspaces/ws.mjs new myproject --repo ~/src/myproject --board MYP --agent "Claude MyProject"
# mint a Worker token for that agent in the app (Account › Tokens), put it in workspaces/myproject/.env
node workspaces/ws.mjs up
node workspaces/ws.mjs status

Agents run unattended in your working tree, with permissionMode from workspace.json: acceptEdits in the template, bypassPermissions only if you trust the brief and the repo. Read workspaces/lib/brief.md before you enable one.

What is not in it

At work I've written about per-ticket stacks: a branch in every repo, its own services, its own copy of the dev database and a preview URL per ticket. None of that is in this repo. That code is specific to Envizom and proprietary. Here the agent works in the repo you point it at.

The repo is github.com/Panth977/ticket-tracker; workspaces/README.md has the loop.

If you run Claude Code headless, where does its question go when it needs a decision, and who answers it?

I'm Panth, and I lead the software team at Oizom, where we build Envizom, an air quality platform that many clients run on their own servers. Every one of those deployments needs a bit of custom work: pull data from somewhere, or expose an API in a schema the client gives us for a public display. That work can't go into our production code. This post is about where it lives now: codeshell, a small browser workspace I built and open-sourced under Apache-2.0.

Node-RED first

Node-RED was the default for these integrations before I joined. It is fragile for this kind of work:

• An async for loop is hard to write.
• Five API calls with something equivalent to Promise.all is close to rocket science.
• Debugging is the worst part.
• If there is a JavaScript library for the job but no Node-RED node in the marketplace, you spend five hours figuring out how to set the library up.

These are short programs. Plain code is the right tool for them.

Then one repo per deployment

The next idea: each deployment gets its own codebase, built into one Docker container that holds all of that server's customisations.

It works until the client disconnects the server from the internet. Then the integration still has to be written, so you open a shell in the container, edit the code with vi, and hope nobody pulls the latest image before you have copied the same code back into the repo.

What I actually needed

A VS Code-like editor in the browser, on the server, with the code directory mounted from outside so it survives a restart. If the code lives in one volume, and the volume is backed up, the integration doesn't need a repo at all.

VS Code in the browser is heavy for this, because the language servers and compilers run on the server. I looked for a light editor and didn't find one, so I wrote codeshell.

What codeshell is

Running one integration is one container:

docker run -d --name myapp \
  -p 8080:8080 -p 3000:3000 \
  -e CODESHELL_PASSWORD=change-me \
  -v myapp-code:/code -v myapp-state:/var/lib/codeshell \
  ghcr.io/panth977/code-shell:latest

Port 8080 is the editor. Port 3000 is whatever the integration listens on.

start.sh is the deploy

Terminals are temporary on purpose. Anything you install or start from a terminal is gone after a restart. What has to come back goes in start.sh:

#!/usr/bin/env bash
set -e
command -v node >/dev/null || (apt-get update && apt-get install -y nodejs npm)
npm ci
exec node index.js

The image is Debian slim, so start.sh can apt-get install whatever runtime the integration needs. On the first run it installs Node and the modules. On later runs node_modules is already in the volume, and mounting /root and /opt as volumes too keeps package caches and runtimes, so restarts get faster.

codeshell supervises the script. Its output shows as a start.sh entry in the terminal panel, next to your own terminals, with Restart and Stop buttons, and its state is in the top bar: running, stopped, exited, waiting or missing. The restart policy is on-failure by default, and restarts back off from 1 second, doubling, to a cap of 60 seconds. On a fresh volume it writes a commented start.sh template with a Node and a Python example.

Terminals keep their scrollback on the server, so a page reload or a dropped connection doesn't lose them.

Where it sits

The other half of our on-premise work is On-Prem Manager, which deploys and updates the containers on each client server and snapshots their volumes. With codeshell, a client's integration is one container and one volume. On-Prem Manager can persist that volume's content to its cloud, where it is versioned and reviewed, so the integration needs no repo of its own.

What it doesn't try to be

• Single user. One login, an HMAC-signed session cookie, and repeated failed logins from one IP are locked out for longer each time.
• Anyone who signs in has a root shell in the container. Put it behind HTTPS and use a strong password. The password and the session secret are removed from the environment of terminals and start.sh.
• No IntelliSense. It's an editor with syntax highlighting, quick open, split groups, find and replace, and a terminal.

The repo is github.com/Panth977/code-shell. Images are published for linux/amd64 and linux/arm64.

Where do your per-client integrations live today, and what happens to them when the client's server goes offline?

I'm Panth, and I lead the software team at Oizom, where Envizom is our environmental monitoring platform. Two days ago I wrote the short version: the eight steps that run after a developer drags a ticket to Git Push. This is the long one. It follows one ticket through every column of the board and goes through each job we took off the agent, with the reason, then what it costs and what broke.

The first version

Before agents, our problems were ordinary ones. Envizom is several repos: hardware drivers, a simulator, a backend and a frontend. A new developer needed days to get a working local setup. Releases meant keeping track of which dev database, prod database, backend and frontend were pointed at each other, and as time went on I was the only person who knew every combination.

Coding agents made my developers faster. With an agent, one developer can keep three tasks in flight. The mistakes then moved out of the code and into the branches around it. A change landed on the wrong branch, or a fix only worked on one machine. I wrote about that in One stack per ticket.

In the first version of the pipeline, the agent did more of the operational steps. It did the engineering, and it also ran git and the deploy. Compared with what we run now, it was slower and more expensive, and it was worse at the tasks that need to be boring.

The failures taught me the most. A script that fails the same way twice is a bug you can fix: you read the log, find the line and change it. An agent that merges a little differently on each ticket gives you nothing to fix. Each failure is a new story, and the next run will tell another one.

The split we ended up with follows one line. If a step is the same on every ticket, it is a script. The agent keeps the engineering.

One ticket, end to end

The board has seven columns. One of them has the agent in it.

Each developer has their own devbox on our dev VM: VS Code in the browser, with our orchestrator, orch, inside it. Every developer is paired with their own agent, and their orch only picks up tickets assigned to that agent. The board is shared. The machines and the agents are not.

Here is a real ticket from the walkthrough video. I asked for a camera layout change on a device page. The agent built it in the ticket's stack and posted the preview. I replied in the thread: "also handle more than 4 cameras with a horizontal scroll, and set up 8 test cameras on this device". The ticket went back to the agent, which picked up with the context it already had, did both, and created the 8 test cameras in the ticket's own copy of the database. I tested the new preview and dragged the ticket to Git Push. After that, two notifications, and the change was live on dev.

The person in that story touched the ticket three times: to write it, to reply to a preview, and to drag it. The agent had one column. Scripts ran the other four.

The ten jobs scripts own

This is the list no agent touches: watching the board, provisioning the stack, copying the database, issuing certificates, driving git, opening pull requests, waiting on CI, deploying, health checks and rollback. They fall into two groups, before the agent starts and after a person says yes.

Before the agent starts

Watching the board. Orch watches for tickets dragged to Approved and assigned to its agent. Nothing here needs any thought. A ticket is either in the column or it isn't, and a model would add cost and nothing else.

Provisioning the stack. Every approved ticket gets a fresh branch in every repo, named after the ticket, plus its own backend, drivers, simulator and frontend. This is the step everything else depends on, so it has to be the same every time. If the stack came out a little different per ticket, nothing built on top of it could be trusted.

Copying the database. Each ticket gets a copy of the dev database, so it can change data freely. This is the safety decision in the whole design. An agent that runs a bad migration or deletes the wrong rows does it to the ticket's copy. Nobody else's work is in that box, so nobody else's work is touched. A step that sets up the safety net shouldn't be done by the thing the net is for.

In Envizom V3, which we are rebuilding from scratch, the copy is cheap. Every store has a dev driver. Postgres, S3, the time-series database and Redis all become SQLite in dev, and MQTT becomes a service built into the Go backend. SQLite runs inside the Go process, so copying a database means copying a few files. That change made our sandboxes about 10x cheaper and faster than sandboxes with full databases.

Issuing certificates. Every ticket has a private preview URL with an HTTPS certificate issued on demand. That URL is what the developer tests in QAQC, so it has to exist and work before the agent posts it.

After a person says yes

When the developer drags a ticket to Git Push, no agent is involved from that moment on. Orch runs eight steps:

Git Push
  1  commit the work on the ticket branches
  2  merge the current dev branch into them
  3  open a pull request in each affected repo
  4  merge the pull requests
  5  post the PR and commit links into the ticket thread
VM Update
  6  wait for CI
  7  roll the new images onto the dev VM
  8  run health checks against the VM
       unhealthy -> roll back to the previous images, revert the merges,
                    tell the thread what happened
       healthy   -> Completed, notify the developer, tear the stack down

Driving git and opening pull requests. One of our goals from the start was that using agents would not weaken the normal process. Every feature still gets its own branch, commits, pull request and review. That only holds if those steps happen the same way on every ticket. An agent deciding how to merge is not a feature. A script that always merges the same way is.

Posting the links. Step 5 looks small, but it is how the thread becomes the audit trail. The PR and commit links land next to the ask, the previews and the feedback, so one ticket holds the whole story of one change.

Waiting on CI. Waiting is the clearest case of a job a model shouldn't hold. Nothing gets decided while it waits, and an agent session kept alive just to wait is still a session.

Deploying, health checks and rollback. These are the steps where sameness matters most. If the VM is unhealthy after the deploy, orch rolls it back to the previous images, reverts the merges and tells the thread, without anyone asking. A rollback that improvises is the last thing you want on the day something breaks. A scripted one runs the way it ran on every ticket before.

All of it stops at the dev environment. Auto-merge, the VM update and the rollback apply to dev only. Production goes through a separate path, where people make the call on a slower cadence. That is on purpose. We don't want automation deciding what reaches production, and we don't want a fast feature list to become the goal.

Taking away "I'm done"

The same split shows up in ticket-tracker, the board I open-sourced, where a small orchestrator turns each ticket into a headless Claude Code session. (The per-ticket stack scripts are not in it. They are Envizom-specific code.) It takes a few more jobs off the agent, ones that are easy to miss because they don't look like jobs.

Deciding it is finished. The agent never moves its own ticket. Its last message is structured output, enforced with --json-schema:

const OUTCOME_SCHEMA = {
  type: 'object',
  properties: {
    status: { type: 'string', enum: ['review', 'waiting', 'blocked'] },
    summary: { type: 'string' },
    question_id: { type: 'string' },
  },
  required: ['status', 'summary'],
};

The orchestrator reads it. review moves the ticket to Review. waiting means a person is needed, so nothing runs until someone answers. blocked usually means the turn ran out of room, so the orchestrator resumes it by itself, up to 3 times, and then says on the ticket that it has stopped. The agent reports. The script decides what that means for the board.

Saying it is alive. The heartbeat comes from the orchestrator, not the agent: one beat every 60 seconds while the agent works. A crashed process can't report that it crashed, so health is worked out from the age of the last beat. The agent's part is its task list, which the orchestrator reads for the status message. More in Agent health from a 60-second heartbeat.

Working out what a turn cost. When a run ends, the orchestrator reads Claude Code's result line and posts the turn's receipt on the ticket. The app's comment box never posts one. Each receipt goes out with an idempotency key made from the ticket and the run number, so a retried request doesn't count a turn twice.

Waiting for an answer. When the agent needs a decision, it asks in the ticket and ends its turn. It is told not to poll. My answer starts the same session again with --resume. Headless Claude Code asks, exits, then resumes on the answer has the details.

Any of these could be left to the agent. Each one is the same on every ticket, so none of them is.

What the agent still does

The engineering. It reads the ticket, works in the ticket's own repos and services, changes the code, and creates the test data it needs in the ticket's database copy, as in the 8 test cameras. When the developer replies with feedback, it picks up with the same context it had before, without re-reading the repo to find out where it was.

It also asks. A real decision that belongs to the owner, such as scope, design, or anything destructive or outward-facing, becomes a question in the ticket rather than a guess.

That is the 10%. It is the only part of the pipeline where the right answer changes from ticket to ticket, so it is the only part where a model is worth what it costs.

What stays with people

Writing the ticket. The thread is the only way to talk to the agent, so every prompt is stored with the work it produced. I read how each person on the team asks for things and coach them on it. That has done more than any prompting guide. See Our coding agent prompts live in the ticket.

QA. The agent posts a preview and moves the ticket to QAQC, and that is as far as it goes. The developer who asked tests the preview, because they are the only one who knows what they meant. An agent grading its own work would remove the one review that matters most. Every step after QA is built to be boring and repeatable, and none of them asks whether the change was the right change. QA is where that question gets asked, so it is the one place we didn't automate. More in 1 human check, 8 scripted steps.

The drag to Git Push. Shipping to dev is a person's action in the thread. Shipping to production is a separate, slower decision.

What it costs

The last row is the plain reason scripts are cheaper. Every agent turn posts its cost into the ticket thread, right under the answer. A scripted merge, deploy or health check posts no receipt, because it doesn't call a model. An operational step done by the agent is done inside a billed turn.

The receipt is a design choice too. The person who asked sees what the ask cost, where they asked it, and it changes how they write the next one.

What broke on the scripted side

Scripts break as well. The difference is how they break.

The cost number was wrong. On ticket-tracker, one ticket reached $402 over nine agent runs and nothing in the app said so. The orchestrator already wrote the cost of every run to a local log, but nothing showed it where the work was. Putting the number on the ticket had a trap: Claude Code's total_cost_usd is cumulative across --resume. Run 8 made one API call with zero output tokens and reported $392.83, the same as run 7. The fix is a few lines in the orchestrator. It keeps the last total per session and posts the difference. The full fix is in $402 on one ticket.

Polling was most of the bill. Two orchestrator loops asked the tracker's API for the full ticket and event lists every 30 seconds or so. Over three days that was 8,822 + 8,815 calls, 88% of every request the API served. The fix was again in the script. Agents now hold one stream on a small revision node and fetch only the delta when it moves. That one is in Polling was 88% of my API calls.

The heartbeat was on the bill twice. The first version kept each beat stream as a Firestore document: 1,440 writes per agent per day, plus a sweep that ran every minute to notice silence, 4,261 runs in three days. Beats moved to the Realtime Database, about 100 bytes each, and silence is now worked out at read time from the age of the last beat.

None of these needed an agent to diagnose. Each showed up in a log or a bill, and each was fixed in one place in the code. That is the trade I wanted from the split. The agent's mistakes stay inside a ticket's stack, where a bad change hit a copy. The scripts' mistakes show up in a log, the same way twice, so they can be fixed.

Why the boring 90% makes the 10% useful

Most of what makes the pipeline worth using comes from the scripted part. Per-ticket isolation is what lets the agent work freely, and lets a ticket sit in QAQC for a week while three others ship. The thread is the audit trail because scripts post into it at every step: the preview, the cost of each turn, the PR and commit links, and the rollback if there is one. Stored prompts are reviewable because the thread is the only channel. Human QA and dev-only scope are the two places we chose not to automate, and both are easier to hold when everything around them is predictable.

The order I'd build it in

If you're setting up something like this:

1. Isolation first. If every task can't run in its own branch, with its own services and its own data, nothing after it is safe.
2. A handoff surface the team already uses. For us that was the ticket.
3. Script every step that is the same on every ticket: provisioning, git, CI, deploy, health, rollback, and the receipts.
4. Only then, the agent inside the box, with the smallest job you can give it.

The ticket is the interface has the walkthrough video and the first write-up of the pipeline.

If your team runs coding agents, which step is the agent doing today that a script would do the same way every time?

I lead the software team at Oizom, where Envizom turns data from air-quality and environmental monitors in 80+ countries into something cities and industries can act on. I dropped out before the AI wave, because I figured the engineering was in solving problems in the real world, and my college made me choose between attendance and real-world applications. Find me on LinkedIn.

I'm Panth, and I lead the software team at Oizom, where we build Envizom, an air quality monitoring platform. This post is about the deployment problem that came with selling it, and On-Prem Manager (OPM), the tool I built and open-sourced under Apache-2.0 to fix it.

Why on-premise

Many organisations don't want to hand their data to a cloud. Governments most of all, and air quality is first a government responsibility. So when a government picks software like Envizom, it also needs that software on its own servers, for its own security reasons.

The more we sold, the clearer it got: local deployment is the one real selling point for an air quality platform, and per-client customisation is what wins the client.

What a deployment actually looks like

The ideal case: the client gives you exactly what you asked for, and if something breaks on their side, a firewall rule or an IP config, they own it.

That is not what happens. One client wants a DC/DR setup. Another wants high availability. Another wants a single server. Another wants DC/DR eventually, but for now on one server, with a second one later and two more after that.

Then there are the integrations: pull data from one of their systems, or expose an API in a schema they hand us for a public display. None of that belongs in our production code.

What we had

An end-to-end setup on Docker Compose, with Portainer as the GUI.

After enough deployments it was clear Compose wasn't enough. Every client is different, so every install meant a lot of configuration and setup afterwards. A simple issue took hours to close, because debugging a remote server through that stack is slow.

That setup is fine at five deployments a year with one engineer for every two. It is the bottleneck the moment you want more.

Where the idea came from

Oizom started as an IoT company. It ships devices with software inside, and every device sits at a remote location where nobody can do anything by hand. For that, Oizom uses a device manager, Aikaan.

So: what if our deployment team had the same thing for on-premise servers?

What the team needs to run a deployment is a short list. Automated volume backups. A way to make a new version live. Environment and config per site. Variables, with separate secrets for each deployment. A GUI to see the containers. I wrote down every problem we hit in operations and built a tool around the list.

What OPM is

Two parts.

The README puts it as: like Portainer, plus a git-style commit history for every server's config.

Install is one line

The cloud's Setup page gives a command per server:

curl -fsSL -H "Authorization: Bearer opi_…" <apiBase>/v1/install/script | sudo sh

It runs preflight checks, uses the Docker that is already there (20.10 or newer) or asks before installing a pinned version, downloads the agent and checks its sha256 and Ed25519 signature, writes the config, sets up a systemd service, and waits up to 60 seconds for a health check, rolling itself back if that fails. Everything it installs is pinned when the script is generated.

Images without a registry

You register images in the cloud's catalog, and the cloud keeps its own copy of their layers. The agent loads images from those blobs with docker load and never connects to a Docker registry, so it never sees a registry password. Sites in the same deployment copy layers from each other first, and only one site with internet fetches from the cloud.

Commits, not proposals

Templates, deployments and each site are branches of commits. The latest commit on a deployment's branch is what its sites should run. You can compare any two branches and cherry-pick between them.

Nothing changes on a site until a person on that site merges it, in the agent GUI. A merge runs one fixed sequence:

LOCK -> SNAPSHOT -> APPLY -> VERIFY -> COMMIT

VERIFY waits for each container to pass its health check, or to run 10 seconds without a restart, within 300 seconds by default. Any failure rolls back, and a failed merge leaves no commit. The failed container is kept for 24 hours under a -failed-<time> name, with its last 200 log lines, so the post-mortem doesn't depend on someone grabbing logs in time.

The dangerous act isn't the merge, it's restoring data. So rollbacks and volume restores get a 60-second countdown and a typed ACCEPT. The spec's reasoning: "the commit message + plan review are the deliberate act… restoring data is still the dangerous act".

Backups by default

Before every update and rollback, the agent snapshots config, image references and the volumes marked for backup, as .tar.zst archives. It keeps the last 10. A scheduled snapshot runs daily at 02:00 site time and keeps the last 7. A snapshot refuses to start if free disk is under 1.2 times its estimate. Snapshots download as signed .opmsnap files.

Docker stays the source of truth

People still change things in Docker directly. The agent scans every 60 seconds and on Docker events. A hand edit to an image, env, port, mount or file shows up as uncommitted drift, and merges are refused until it is committed or discarded. A site can also auto-commit drift once it has been stable for 30 seconds, authored as "changed in Docker".

Standalone first

Every agent GUI can do everything locally with no cloud at all: create containers, change images, edit files, variables and secrets. The cloud is how you run many sites, not something a site needs to keep running.

When the client cuts the internet

Some clients take the server off the internet for security. OPM has three routes to the cloud, in order:

1. The server's own internet.
2. A peer site in the same deployment that has internet, relaying.
3. The browser of the person using the agent GUI. The GUI tab long-polls the agent and runs the cloud requests from the browser, so an engineer's laptop on the client network becomes the tunnel.

The agent only ever talks HTTPS on 443 to one API domain, which is one line in a client's firewall request.

And for a client that allows neither the agent nor the cloud, the cloud can mirror every branch to a GitHub repository. A machine checks out its site's branch, fills in site.env and runs ./deploy.sh: plain docker compose, Caddy as the router, Portainer as the GUI, no OPM code on the box. A real-Docker test compares that output with the agent's and reports 0 differences. You lose drift detection, snapshots and rollback, which is why it's the last resort.

The size of it

The full specification, SPEC-final.md, is 894 KB, and its decision log has 116 numbered entries, each with the reason. The code is about 97,000 lines of Go, 96,000 of TypeScript and 60,000 of Svelte, with 105 Go test files and 157 TypeScript test files. There is an end-to-end harness that runs real agents against real Docker.

Integrations got their own tool too. They run in codeshell, a small browser editor in a container whose code is one volume. OPM's "Persist to cloud" versions a volume's content in the cloud, the docs name "client integration code" as the use, so that code is reviewed and kept there instead of in a git repository. codeshell gets its own post.

What I'd ask you

If you ship software that customers run on their own servers: what does one deployment cost you in engineer-hours, and which part of it did you automate first?

We didn't start there. The first version gave the agent more of the operational steps, and this post is about why we took them back.

I lead the software team at Oizom, where Envizom is our environmental monitoring platform. The pipeline below is our dev pipeline for Envizom.

The shape of it

A developer writes what they want in a ticket, in plain English, and drags it to Approved. Our orchestrator picks it up and gives the ticket its own stack: a branch in every repo named after the ticket, its own backend, drivers, simulator and frontend, a copy of the dev database, and a private preview URL with an HTTPS certificate issued on demand.

The agent works inside that stack and talks to the developer only through the ticket thread. When it's done it posts the preview link and moves the ticket to QAQC. The developer tests the preview, replies in the thread, and either sends it back or drags it on.

Approved ──► [scripts] provision stack ──► [agent] engineering work ──► QAQC
                                                    ▲                    │
                                                    └── feedback ────────┤
                                                                         ▼
Completed ◄── [scripts] commit, merge, PR, CI, deploy, health ◄── Git Push
     │                         │
     │                         └── unhealthy: roll back images, revert merges
     ▼
 stack torn down

One box in that diagram has the agent in it. Everything else is a script.

What the scripts do

Here is the list of jobs no agent touches:

• watching the board
• provisioning the per-ticket stack
• copying the database
• issuing certificates
• driving git
• opening pull requests
• waiting on CI
• deploying
• health checks
• rollback

The clearest example is what happens when a developer drags a ticket to Git Push. From that moment there is no agent involved. The orchestrator runs eight steps:

If the VM isn't healthy afterwards, the orchestrator rolls it back to the previous images and reverts the merges, on its own, and then tells the thread what happened. If it is healthy, the ticket moves to Completed and the developer gets a notification. The ticket's stack is torn down after that.

Why we took the steps back from the agent

We tried it the other way first, with the agent doing more of the operational steps. It was slower, it was more expensive, and it was worse at exactly the tasks that need to be boring.

That last one is the reason that matters. Merging, deploying and rolling back should happen the same way on every ticket, and that sameness is the whole value. An agent deciding how to merge is not a feature. A script that always merges the same way is.

The scripted part is deterministic and tested. When it breaks, I read a log and fix it.

Where the agent earns its place

The agent's job is the engineering work itself. In the walkthrough video I asked for a camera layout change on a device page. The agent shipped it and posted the preview. I replied in the thread asking it to also handle more than 4 cameras with a horizontal scroll, and to set up 8 test cameras on that device. It did both, including creating the test data in the ticket's own database copy, and it picked up with the same context it had before.

Isolation is what lets it do that safely. Because every ticket has its own stack, an agent that does something silly to the database did it to a copy. A developer can have several tickets in flight at once, and one ticket can sit in QAQC for a week while three others ship.

Two more lines we drew on purpose:

• QA is human. The developer tests the preview and decides whether the ticket moves forward. An agent grading its own work would remove the one review that matters most.
• This pipeline stops at dev. Auto-merge, rollback and VM updates all apply to the dev environment only. Production goes through a separate path where people make the call, on a slower cadence.

The order I'd build it in

If you're setting up something similar, this is the order we ended up with:

1. Isolation first. If every task can't run in its own branch, its own services and its own data, nothing after it is safe.
2. A handoff surface the team already uses. For us that was the ticket.
3. Only then, the agent inside the box.

And keep the box scripted. The agent should be the smallest part of the system.

The longer write-up, with the video walkthrough, is The ticket is the interface.

If your team runs coding agents, which step did you hand to the agent that a script should own?

The constraint that forced it: Firestore cannot OR across fields, and every combination of filters you want to query needs its own composite index. A tracker with user-defined views over custom fields would need thousands of them.

I'm Panth, and I lead the software team at Oizom. This is from ticket-tracker, my open-source board. Its decisions page lists the choices that are expensive to reverse, each with the alternative that was turned down. This is D5.

The decision

A saved view is a few lines of data. A filter is a tree of and / or groups over leaves:

export type FilterGroup = { op: 'and' | 'or'; children: FilterNode[] };
export type FilterLeaf = { field: string; cmp: Cmp; value?: unknown };
export type FilterNode = FilterGroup | FilterLeaf;

A leaf can compare any built-in field (stage, priority, assignee, tag, due date, state, creator, text, linked) or any custom field, with twelve comparisons from is to between. "Mine and high priority, or anything of Ann's" is this, straight from the tests:

const f: FilterNode = {
  op: 'or',
  children: [
    {
      op: 'and',
      children: [
        { field: 'assignee', cmp: 'is', value: 'me' },
        { field: 'priority', cmp: 'is', value: 'hi' },
      ],
    },
    { field: 'assignee', cmp: 'is', value: 'ann' },
  ],
};

As a Firestore query, that OR spans two fields, and the tree can nest as deep as people like. As a function over an array, it's a recursive evalNode.

One function draws every view

The whole engine is one call:

applyView(tickets, view, board, ctx) → groups[]

It does four things, in this order:

1. Keep the ticket states the view asks for. The default is active only, so archived tickets leave every default view.
2. Run the filter tree.
3. Sort on several keys, ties broken by rank and then id, so every client gets the same order.
4. Group, then group again inside each group for swimlanes.

Kanban, table, calendar and timeline all draw from what it returns. A ticket with two assignees or three tags appears in each of their groups. Swimlanes are computed over all visible tickets, so every column has the same rows.

me, today, thisWeek and overdue are tokens in the saved view, resolved when the function runs, against the viewer's id and time zone. A shared view called "Mine, due this week" means the same thing to everyone who opens it, each in their own week.

Because the function is pure, it's tested as one: the test file is 453 lines against 665 lines of engine, with a fixed clock (Tuesday 2026-09-22 in Kolkata) and a board with six custom field types.

The one query, and what it costs

The query that feeds the engine is about as simple as Firestore gets: the board's tickets where state == 'active', one listener per board. It needs no composite index.

Loading the whole board on every open would pay one read per card. So the board store is incremental. The first open runs the full query. After that, the device starts from Firestore's own IndexedDB cache of the last snapshot and asks only for tickets whose updatedAt is newer than the newest one it has seen. An unchanged board answers that with no documents, which Firestore bills as one read, whatever the board's size.

The delta query has no state filter, on purpose. A ticket that leaves the active set has to arrive so it can be dropped from the board, and a query for active tickets would never send it. The browser decides what to keep. One inequality on one field also means the delta needs no composite index.

A hard delete doesn't show up in any updatedAt > query, because the document is gone. So the watermark is trusted for 6 hours from when it was written. After that, the next open pays for one full query and agrees with the server again.

Where it stops

The decision writes down its own ceiling: about 5,000 active tickets, or about 5 MB, per board. Past that, the plan is to flag the board as large and run its views through Typesense, which can filter on every field, with paging. Archived tickets are never loaded by default, so the number that matters is open work, not history.

For a personal tool, where a board is the unit people share, 5,000 open tickets is far more than a board should hold. The trade is the right one for this product. It wouldn't be for a tracker that hosts a whole company on one board.

What I'd take from this for any Firestore app with user-built filters: if the data a screen needs fits on the device, query it once, keep it live, and do the rest in code you can unit test. Indexes are for the data that doesn't fit.

Where do your filters run: in the database, or over data you've already loaded?

The split stayed. A key like ENG-42 gets pasted into WhatsApp messages, emails, GitHub PR titles and other tickets, and every one of those has to keep pointing at the right ticket for as long as the link exists, including after the ticket is gone.

I'm Panth, and I lead the software team at Oizom. This is from ticket-tracker, my open-source board, written to replace a tracker called TeamTrack. Its decisions page lists the choices that are expensive to reverse. This is D4.

The decision

So a ticket lives at boards/{boardId}/tickets/{ticketId}, and ENG-42 is a field on it. The key also gets a document of its own at the top level:

// keys/{ticketKey}: '#ENG-42' → the ticket it names
export const KeyIndexSchema = z.object({
  ticketId: TicketIdSchema,
  boardId: BoardIdSchema,
  /** The key itself, equal to the doc id. */
  current: TicketKeySchema,
  deleted: z.boolean().optional(),
});

current is the redirect: an old key's row can name the key the ticket has now. Exact-key search follows one hop of it.

Then the move was dropped

Phase 1 built moving a ticket to another board: copy the ticket, its thread, activity and files to the other board, give it a new key, and redirect the old one.

Decision D13 took it out. Moving is machinery (option mapping per field, assignees who aren't on the new board, a queued copy job, the key redirects) for something that happens rarely in a personal tool, and it was the most dangerous write in the system. Now a ticket stays on the board it was created on. To connect work across boards, you link the tickets or #reference them. A ticket filed on the wrong board gets closed and filed again, and the new one can reference the old.

So the reason written next to D4 no longer happens. Nothing writes a redirect today. The id and the key stayed separate anyway, because the same split does three other jobs.

1. References store the id, not the key

When you type #ENG-42 in a comment, the editor stores a ticketRef node with the ticket's id. The key in that node is only a hint for rendering:

case 'ticketRef': {
  const id = String(n.attrs?.ticketId ?? '');
  if (id && !refs.includes(id)) refs.push(id);
  const hint = typeof n.attrs?.key === 'string' && n.attrs.key ? n.attrs.key : 'ticket';
  line += `#${opts.keyOf?.(id) ?? hint}`;
  return;
}

The server derives each message's refs as a list of ticket ids. Mentions of people work the same way: a mention stores a uid and shows the current display name, so a rename never breaks it. TeamTrack stored mentions as [@Name](mailto:…) links in Markdown and keyed assignees by email, so a change of address was a data migration.

2. A key is issued once

Keys come from a counter on the board document, inside the same transaction that creates the ticket. Not from a query for the highest number:

const start = board.nextNumber;
tx.update(ref, { nextNumber: start + ticketIds.length });
return ticketIds.map((ticketId, i) => {
  const number = start + i;
  const key = ticketKey(board.key, number);
  tx.create(keyRef(key), { ticketId, boardId, current: key });
  return { key, number };
});

tx.create fails if the document already exists. A key that was ever handed out, whether it now points at a live ticket, a deleted one or (once) a moved one, can't be overwritten. The transaction fails instead.

The price is in the comment above that code: the board document takes about one sustained write a second, so one new ticket per second per board. Bulk creation takes a range with a single counter bump.

3. Delete leaves a tombstone

Deleting a ticket removes it from the links and references of other tickets, takes it off watchers' lists, deletes the ticket document and its thread, and keeps the key:

// Tombstone: kept forever, never reissued.
const tomb: KeyIndex = k
  ? { ...k, deleted: true }
  : { ticketId, boardId, current: ticket.key, deleted: true };
tx.set(keyRefs[i]!, tomb);

After that, #ENG-42 in an old comment says the ticket was deleted. With the key as the document id, deleting the ticket deletes the only record that the key was ever used. If that key were ever issued again, every old link would open the new ticket as if nothing had happened. The comment in the schema says it more directly: the row is kept "so an old link says 'this ticket is gone' instead of landing on somebody else's work".

Deleting a whole board does the same for every ticket on it, and its board key stays claimed too. Board keys like ENG are claimed across the whole app, first come first served, like a username. That is what lets /t/ENG-42 and #ENG-42 mean one ticket without saying whose board it is.

Who reads the index

Everything that starts from a key goes through keys/, not through a board:

• the REST and MCP doors, when an agent or a script asks for ENG-42
• the GitHub hook, which finds keys in branch names, PR titles, commit messages and comments, and posts a line in each ticket's thread
• Markdown coming in through REST or MCP, where #ENG-42 is turned into a ticket id before it is stored
• the search box, for an exact key

A tombstoned key resolves to nothing for all of them. Over REST and MCP it is a 404, and so is a key on a board the caller can't read.

What it cost

One extra document per ticket, and one extra read whenever something arrives by key instead of by id: a pasted key, a webhook, a search. Links between tickets store only the id, so finding one looks on the same board first, which is almost always right, and falls back to the keys index, whose rows carry the board.

What I'd keep from this for any system with human-readable ids: let the name be a field, give it its own index, issue it once, and never delete the index row. The move that justified D4 is gone, and the split still earns its place on delete alone.

Have you ever reused an id after a delete, and found out from a link that opened the wrong thing?

The constraint: an orchestrator should be able to import the client straight from a URL and get types, and nothing I host may be a second copy of the code that drifts from the first.

I'm Panth, and I lead the software team at Oizom. This is from ticket-tracker, my open-source board, where AI agents work tickets alongside people. Writing a client by hand for every orchestrator is the thing the app exists to avoid, so it ships one: @tm/sdk, hosted next to the API it talks to.

What gets hosted

/lib/v1/sdk.js              ESM bundle, no dependencies   (browsers, Deno, Bun)
/lib/v1/sdk.min.js          the same, minified
/lib/v1/sdk.ts              the whole SDK as ONE TypeScript file
/lib/v1/sdk.d.ts            one flat declaration file, no imports
/lib/v1/tm-sdk-1.6.0.tgz    and tm-sdk.tgz, for npm install
/lib/v1/openapi.json        OpenAPI 3.1 for /v1
/lib/v1/sdk.json            version, byte sizes, SHA-256 of every file
/lib/latest/…               the same files, never cached

The v1 is the API's wire version, not the package's. A breaking change to the wire format would get /lib/v2/, and /lib/v1/ keeps working. A new SDK build for the same wire version replaces the files in place under a new tarball name.

Concatenate first

sdk/build.mjs is 237 lines. The order of the steps is the whole idea:

sdk/src/*.ts  ──concat, imports stripped──▶  dist/sdk.ts
                                               │
                    tsc --emitDeclarationOnly  │        esbuild
                                               ▼            ▼
                                         dist/sdk.d.ts   dist/sdk.js + sdk.min.js
                                                            │  npm pack
                                                            ▼
                                                  dist/tm-sdk-<version>.tgz

Step one glues the 10 modules in sdk/src/ into one file, in a fixed order, and drops every import … from './x.js' line. Anything that imports a package rather than a sibling is a bug, so the script throws instead of dropping it:

const from = stmt.match(/from\s+'([^']+)'/);
if (from && !from[1].startsWith('.'))
  throw new Error(`${file}: the SDK must have no dependencies, but it imports '${from[1]}'`);

Everything else is derived from that one file. That's what keeps the promise cheap.

• sdk.d.ts is emitted by tsc from a file that has no imports. Inside the repo the SDK's types come from the same shared zod schemas the server validates with, so a wrong field is a compile error rather than a 400. But a URL import can't resolve zod or the shared package. Emitting from the concatenated file means the declaration file can't come out mentioning them. It is also a full strict type-check on the way.
• sdk.ts is the exact file that was type-checked and bundled. Deno and Bun import it by URL and type it natively. It isn't a second copy.
• sdk.js and sdk.min.js come from esbuild, and both start with /// <reference types="./sdk.d.ts" />, which Deno and Bun read.
• The tarball carries a package.json with types first in its exports map, so tsc and editors resolve the declaration file. Node can't import over https, so this is the supported path there.

At the end, the build imports each bundle, checks the exports and the version stamp, creates a client with a fake fetch, and asks the MCP tool factory for a tool. It also fails if sdk.d.ts has a single import line. A broken build never reaches the deploy.

The header that turned .ts into video

Hosting is Firebase, and the headers come from firebase.json. Three rules matter.

The last one is real. Left alone, Hosting serves a .ts file as an MPEG transport stream, and Deno refuses it. .d.ts also ends in .ts, so one rule covers both. Next to it, /lib/v1/sdk*.js gets X-TypeScript-Types: /lib/v1/sdk.d.ts, which is what makes import … from 'https://…/sdk.js' typed in Deno.

Header rules match the request path, not the file. /lib/index.html doesn't match a request for /lib/, so the install page's rule is spelled three ways. This looks right in review and is wrong in production, which is why the headers are asserted rather than argued: a check starts the real Hosting emulator on the real config, fetches all eighteen files and asserts the content type, cache rule, CORS header and X-TypeScript-Types on each.

Proving the types arrive

"It compiles" proves little if the types never loaded, because untyped code compiles too. sdk/fixtures/check.mjs runs one check per runtime against a local stand-in that sends the same headers as firebase.json:

• Node: npm i the real tarball into a project of its own, then tsc --strict.
• Deno: deno check a script that imports sdk.ts, another that imports sdk.js, and a third that gets only the X-TypeScript-Types header.

Every fixture ends in @ts-expect-error lines. They fail if the mistake compiles, and they also fail if the types never resolved, since an expected error that never happens is itself an error. So "no types" can't pass as "types fine". And there is a negative control: the same bundle with both type hints removed must fail deno check. It does. If it ever passed, the checks above it would prove nothing, and the script says so.

The deploy won't ship a stale bundle

The SDK is built by pnpm sdk:build, not by the deploy. So the deploy checks what it's about to upload: every artefact exists, carries the version sdk/package.json claims, starts with the triple-slash reference, has a flat declaration file, and is newer than every file under sdk/src. If not, it stops and prints the command to run.

The failure it guards against is quiet: yesterday's bundle going out next to today's API. Nothing would error. A field would just be missing from the types until someone hit it.

What it costs

The rule is "derive, don't copy", and the price is mostly the checks. The build script is 237 lines, and the fixture and header checks add their own. In return there is one place to change the client, and the four runtimes see the same code with the same types.

How do you ship a typed client to users who import it over a URL rather than from npm?

The constraint: an agent should need one page, not a tour of the docs site, and that page can't be a second copy of the truth that rots the week after it's written.

I'm Panth, and I lead the software team at Oizom. This is from ticket-tracker, my open-source board, which my own agents work from. Section O of its agent plan is titled "One page an orchestrator can be pointed at". This post is how that page is built.

Four files, one source

The app serves these beside itself:

/llms.txt          the short index: what this is, the URLs, how to authenticate
/llms-full.txt     the one page: the whole operating context, Markdown
/integrate         the same content as a readable web page, for people
/integrate.json    a small manifest: api, mcp, sdk, openapi and llms URLs + version

/llms.txt is 85 lines and mostly says "read /llms-full.txt". The full page has nine chapters: what the software is, how an agent gets in, the three doors (SDK, REST, MCP), the loop as a complete runnable orchestrator, the reference, the rules of the house, recipes, artifacts, and the version and changelog.

All four come out of one script, scripts/gen-integration.mjs. It also builds the Claude Code plugin, whose skill ships llms-full.txt verbatim as its reference file, so the plugin can't hold a second copy either.

What is written by hand, and what isn't

The prose lives in scripts/integration-source/, one Markdown file per chapter. That is the only hand-written part. Where the prose needs a table that the code already knows, it holds a marker instead:

## 5. Reference
...
{{gen:contracts}}
...
{{gen:rest}}
...
{{gen:mcp}}
...
{{gen:sdk}}

The generator replaces each marker with a block built from the same sources the software runs on:

The reference chapter alone is 3,218 lines: 72 REST endpoints and 36 MCP tools. Nobody types those. When a route is added, it shows up in the page on the next build because the route's schema is the page's source.

The expansion is strict. A marker the generator doesn't know stops it:

.replace(/\{\{gen:([a-z]+)\}\}/g, (_, name) => {
  if (!(name in blocks)) fail(`unknown generated block {{gen:${name}}} in the prose`);
  return blocks[name];
})

Some tables need a sentence of meaning next to each generated name, like what an agent with the commenter role may do. Those sentences are written in the script, keyed by name. If the shared package grows a new role and nobody wrote what it means, the build fails instead of printing a blank row.

The guard

node scripts/gen-integration.mjs --check generates everything fresh and compares it with the committed copy. The deploy script runs it before firebase deploy, so nothing goes out on a stale page. If I change a schema and forget pnpm integrate:gen, the deploy stops.

That only works if the same inputs give the same bytes. So every list is emitted in a fixed order and nothing reads the clock. The page's "updated" date is the date of the newest changelog entry, which an author bumps when the contract moves. A new Date() stamp would make the guard fail every midnight.

The guard is tested too. The end-to-end suite generates into a temp directory, appends a line to each file in turn and checks that exactly that file is reported stale. The plugin zip is compared as bytes, so the test flips one byte in the middle of it and expects the same.

Served as text, uncached

The page exists to answer "has anything changed?", so a cached copy defeats it, and a model handed HTML where it expected Markdown is worse off than with no page. Hosting serves /llms*.txt as text/plain; charset=utf-8 with no-store and access-control-allow-origin: *. A test reads firebase.json and asserts those headers, and a separate check proves them against the hosting emulator.

What an agent gets from it

A person creates an agent, adds it to a board and gives it a token. The agent gets that token and one URL. From the page it learns how to wait for work without polling, publish a task list, heartbeat every minute, ask a question when it is blocked and post its result with files, with every endpoint and tool it needs in the same file.

The changelog at the end goes back to version 1.0.0 on 2026-09-26, so an agent can tell whether what it learned last week still holds.

Where do your agents learn your API from, and what stops that from going stale?

The constraint: the agents have to notice new work within seconds, and an idle agent should cost nothing.

I'm Panth, and I lead the software team at Oizom. This is from ticket-tracker, my open-source board, which my agents work from. Its plan has a section on what the app costs, and this post is the part of it about the agents: they used to poll, and now they wake.

What the polling cost

The numbers are from 23 to 26 September 2026.

Two orchestrator loops ran every 30 seconds or so. Each one asked for the full list every time: tickets with limit=200 and no updated_since, events with unacked=1 and no cursor. The answer was the same almost every time, and it was paid for every time.

The API runs on Cloud Run, which bills CPU for the whole request, including the time it spends waiting on I/O. Each call did an API-key lookup, a board read, an agent read and two rate-limit transactions before the actual work. An idle agent was the largest line on the bill.

The shape that costs nothing while idle

The API's data is in Firestore, which bills per operation. The Realtime Database bills for bandwidth. So the signal lives in the Realtime Database, and the data stays where it is.

Two tiny nodes carry the signal:

rev/{boardId}            'something on this board changed'
agents/{agentId}/wake    'something is in your inbox'

Every write goes through one command layer. After a command succeeds, it overwrites the board's rev node with about 40 bytes:

export const bumpBoardRev = (boardId: string, by: string | null, at: number): Promise<void> =>
  liveSafe(`rev ${boardId}`, () =>
    rtdbAdmin()
      .ref(live.rev(boardId))
      .set({ at, ...(by ? { by } : {}) }),
  );

Nothing about the change travels through it. The node only says "ask again". Two commands don't move it: ping, which changes nothing, and the agent heartbeat, which has its own node. Bumping on a heartbeat would wake every watcher of the board once a minute per agent, which is the polling all over again.

The agent's side is one long-lived stream on those nodes, then one REST call that asks only for what changed: updated_since for tickets, a cursor for events. One Realtime Database write per command, one small query per real change, and nothing at all while the board is quiet.

Streaming without the Firebase SDK

The Realtime Database's REST API streams a node as plain Server-Sent Events. Ask for {database_url}/{path}.json with accept: text/event-stream and you get put and patch frames as the node changes. There is no Firebase SDK in the SDK, so the same code runs in Node, Deno and Bun. An idle stream is an open socket and close to zero bytes.

The stream needs a credential, and an agent holds an API token, not a Firebase login. GET /v1/live bridges that. It mints a Firebase custom token for the token's principal, with a board claim for an agent's board, exchanges it server-side for an ID token, and returns the database URL, the token, its lifetime and the exact paths it may stream. The database rules then let an agent read its own wake node and its board's rev, and nothing else. The watcher re-mints when the credential expires.

Two details that keep it from becoming a poll again

A reconnect is not a change. The Realtime Database replays the node's current value as a put every time a stream connects. If every frame woke the agent, a flapping connection would be a polling loop with extra steps. So a frame only counts when its at timestamp has moved past the last one seen, and the first frame after a connect only records where things stand.

Ten bumps are one fetch. The watcher keeps at most one wake queued. If ten commands land while the agent is busy, it wakes once and makes one delta fetch, because the delta covers all ten. A real change replaces a waiting backstop wake, never the other way round.

When the stream can't open

Some setups can't hold an SSE stream: a proxy that eats it, a runtime without streaming bodies, an older server without /v1/live. Then the watcher polls, and says why. The poll starts at 2 seconds while work is arriving and backs off to 60 seconds when nothing is found. The caller tells it which, with the number of rows the last delta returned:

const w = tm.watch();
for await (const signal of w) {
  const page = await tm.events.list({ cursor, unacked: true });
  cursor = page.next_cursor ?? cursor;
  for (const ev of page.data) handle(ev);
  w.found(page.data.length);        // 0 ⇒ back off, >0 ⇒ stay eager
}

Even while streaming, the SDK fetches once every 15 minutes in case a bump was lost. That backstop is the only REST traffic an idle streaming watcher makes.

What it was for

The target in the plan is 50 boards with 10 live tickets each, always on, for 50 rupees a month or less. That only works if the steady state sits inside Firestore's free daily quota and the API functions stay off the read path. Agents that poll every 30 seconds can't fit in that. Agents that wake on a 40-byte write can.

If your workers poll an API on a timer, what share of your requests is them asking whether anything changed?

The constraint: an MCP client that a person uses every day should not hold a long-lived secret the person had to handle by hand, and the person should be able to see and cut off what it can reach.

I'm Panth, and I lead the software team at Oizom. This is from ticket-tracker, my open-source board. Its decisions page lists the choices that are expensive to reverse, each with the alternative that was turned down. This is D9.

The decision

API keys did not go away. They still work on /mcp for headless setups, where nobody is there to click Allow. Claude Code takes a token in a header, and so do the orchestrators that run my agents. OAuth is for the case where a person is sitting in front of the client.

What the client does, with nobody pasting anything

The whole flow is discovery. The client starts knowing one URL, and every other address comes from a document the server publishes.

1. POST /mcp, no token
   ← 401  WWW-Authenticate: Bearer resource_metadata=".../.well-known/oauth-protected-resource/mcp"
2. GET  /.well-known/oauth-protected-resource/mcp        (RFC 9728)
   ← { authorization_servers: [base], scopes_supported: [...] }
3. GET  /.well-known/oauth-authorization-server          (RFC 8414)
   ← authorize, token, register and revoke endpoints; PKCE S256 only
4. POST /oauth/register                                  (RFC 7591)
   ← client_id
5. GET  /oauth/authorize?...&code_challenge=...  → our consent screen
6. person picks boards, presses Allow  → redirect with ?code=
7. POST /oauth/token  code + code_verifier  ← access + refresh tokens

Step 1 carries the design. The 401 is not "your token is bad". It is "go and sign in, and here is where to start". A client that has never heard of this server can follow the header all the way to a working token.

Some clients probe OIDC discovery first, so /.well-known/openid-configuration answers with the same document.

The consent screen is where scope gets decided

The screen names the app asking, lists the permissions it wants in plain words ("Read tickets", "Post, edit, pin and react in threads"), and asks which boards it may act on: every board, or the ones you tick. Nothing is granted until you press Allow.

The server checks the ticked boards against the boards the person can actually read. A board you are not on is an error, not a silent drop.

What the grant gives back shapes the MCP server itself. A tool the grant's scopes do not allow is not registered at all, so it is missing from tools/list, and every handler checks again anyway. The model never sees a tool it would only get a 403 from.

Token lifetimes

Tokens are stored only as their SHA-256. The grant is a document under the person (users/{uid}/oauthGrants/{grantId}), and it is the revocation switch: revoking deletes it and every token issued under it. Revoking in Connected apps stops the app on its next call, including sessions already open.

A refresh may narrow scopes, never widen them. Asking for a scope the grant does not hold fails with invalid_scope.

The rule that catches a stolen refresh token

Rotation alone does not help if nobody notices the old token coming back. So a used refresh token is kept, marked with when it was used, until it expires. If it is presented again, that is two parties holding the same token: the real client and someone else. The server revokes the whole grant.

// A client that refreshes twice at once, or retries after losing the response,
// presents the same token seconds apart: that is not theft.
const retry = row.usedAt !== undefined && opts.now - row.usedAt <= REFRESH_RETRY_GRACE_MS;
if (row.usedAt !== undefined && !retry) {
  // THEFT DETECTION: an already-rotated refresh token came back.
  reused = { uid: row.uid, grantId: row.grantId };
  return null;
}

The grace window is 60 seconds. Without it, a client that fires two refreshes in parallel, or loses the response and retries, would lock its own user out. With it, a retry gets a fresh pair and a replay an hour later kills the grant with "Refresh token reuse detected — the app must be connected again".

The small checks that keep it from becoming an open door

Dynamic registration means anyone can register a client, so the edges matter:

• Registration is rate-limited per IP address: 20 a minute, 500 a day.
• Redirect URIs must be https, or http on localhost, 127.0.0.1 or [::1] only, or a private-use scheme such as cursor:// or vscode://. javascript:, data:, file:, vbscript: and blob: are refused, and so is any URI with a fragment.
• A bad client or redirect URI is shown as an error page, never redirected to. Redirecting there would make the server an open redirector. Any other error goes back to the client as ?error= the way RFC 6749 says.
• Only two grant types: authorization_code and refresh_token.

What it cost

One more surface to own. An API key is one table and one header check. This is an authorization server with eight endpoints, a consent screen in the app, and a Connected apps page. My own review notes had a row for it: the MCP flow pointed at a consent page that did not exist yet. That was fixed by building the page at /oauth/consent, after which the grant shows in Connected apps.

For an MCP server people add to Claude by hand, I would pay it again. The person never touches a secret, sees exactly which boards they handed over, and has one Revoke button.

If you run an MCP server, which do your users hold today: an OAuth grant or a key they pasted?

I'm Panth, and I lead the software team at Oizom, where Envizom is the platform that turns data from air-quality and environmental monitors into something cities and industries can act on. This post is about one design goal of our agent pipeline: the prompt is kept with the work, and it can be reviewed.

The only channel is the thread

A developer writes what they want in a ticket, in plain English, and drags it to Approved. There is no IDE open, no terminal and no prompt window. That ticket text is the whole spec.

Our orchestrator, orch, gives the ticket its own stack: a branch in every repo, its own backend, drivers, simulator and frontend, a copy of the dev database and a private preview URL. A coding agent does the work inside that stack. When it is done, it posts the preview link into the thread and moves the ticket to QAQC.

The developer opens the preview, tests it, replies in the same thread and moves the ticket back. The same agent, with the same context, picks up where it left off. Every follow-up is a message in the thread, so every follow-up is stored too.

When we set out to build this, it was one of five things we wanted from the start:

Every prompt should be stored with the work it produced, so we can review and coach how people ask for things.

What one thread holds

By the time a ticket is done, its thread holds the whole exchange, turn by turn:

Here is the shape of a real one. In the walkthrough video I asked for a camera layout change on a device page:

ticket   camera layout change on the device page          -> Approved
agent    builds it in the ticket's stack, posts the preview -> QAQC
me       "also handle more than 4 cameras with a horizontal scroll,
          and set up 8 test cameras on this device"         -> back to the agent
agent    does both, the 8 test cameras in the ticket's own
          database copy, posts the preview                  -> QAQC
me       drags it to Git Push
orch     commits, opens and merges the PRs, posts the links,
          waits for CI, deploys to dev, checks health        -> Completed

Every line of that is in the thread. Anyone can open the ticket and read what was asked, in the words it was asked in, next to what came out of it.

Reading prompts like pull requests

Because the prompts are stored, they can be reviewed. I read how each person on the team asks for things and coach them on it. That has been more useful than any prompting guide.

A prompting guide tells you what good asks look like in general. A ticket thread shows you how one person on your team asked for one change, what the agent built from it, and how many replies it took to get to something they were happy with. You can point at a specific message and talk about it, the way you would point at a line in a pull request.

The cost line matters here as well. Each agent turn posts its cost into the thread, right where the ask was made. The person asking sees what the ask cost, and it changes how they write the next one. That is on purpose too.

Why it needs one stack per ticket

A stored prompt is only useful if it sits next to the work it produced, and nothing else. That is where per-ticket isolation comes in.

Each ticket has its own branch in every repo, its own services and its own copy of the dev database. So a ticket's thread is a self-contained record: what was asked, what was built in that stack, what was tested on that preview. A ticket can sit in QA for a week while three others ship, and its history is still in one place, not mixed into anyone else's.

The thread is also where the rest of the process shows up. After Git Push, no agent is involved: orch commits, merges, opens and merges the pull requests, waits for CI, deploys to the dev VM and checks its health, and posts the PR and commit links back into the same thread. The prompt, the result and the review trail end up in one ticket.

What we get from it

• Every prompt the team writes is stored with the ticket it belongs to.
• I can review how people ask for things using their own tickets, not examples from a guide.
• The cost of each ask is visible where it was asked.
• The ticket is the audit trail: what was asked, what was built, what was tested and what feedback was given.

If your team uses coding agents, where do the prompts go once the work ships?

I lead the software team at Oizom. The full pipeline, including where we drew the line (dev only, human QA, visible cost), is in The ticket is the interface.

I'm Panth, and I lead the software team at Oizom, where Envizom is the platform. This post is about that one human step in our agent pipeline: where it sits, what it checks, and why we kept it on purpose.

Where QA sits

A developer writes what they want in a ticket, in plain English, and drags it to Approved. Our orchestrator, orch, gives the ticket its own stack: a branch in every repo, its own services, a copy of the dev database and a private preview URL. A coding agent does the work inside that stack.

When the agent is done, it posts the preview link in the ticket thread and moves the ticket to QAQC. That is as far as it goes. The agent never marks its own work as done.

Approved ──▶ agent works in the ticket's own stack
                │
                ▼
             QAQC  ◀──── feedback in the thread, ticket moved back
                │            (same agent, same context)
                │ a person tests the preview
                ▼
            Git Push ──▶ 8 scripted steps, no agent ──▶ dev VM
                                                       (rollback if unhealthy)

What the person does

The developer who asked opens the preview and tests it. If something is off, they reply in the same thread and move the ticket back. The same agent picks it up with the same context and carries on.

In our demo video I asked for a camera layout change on a device page. The agent shipped it and posted the preview. My QA reply was: "also handle more than 4 cameras with a horizontal scroll, and set up 8 test cameras on this device". The agent did both, and it created the test data in the ticket's own copy of the database, so nothing it added touched anyone else's data.

That reply is a QA note and a new requirement in one message, and it went to the agent the same way the original ask did: as words in the ticket thread.

Why it stays human

The QA step is where someone checks that the thing that was built is the thing that was asked for.

If the agent graded its own work, the one review that matters most would disappear. Every step after it is built to be boring and repeatable, and none of them asks whether the change was the right change. QA is the only place in the pipeline where that question gets asked, so it is the one place we did not automate.

It also keeps the record honest. What was asked, what was built, which preview was tested and what feedback went back all sit in one ticket thread, and the decision to ship is a person's action in that thread, not the agent's.

Everything after QA is a script

When the developer is happy, they drag the ticket to Git Push. From there orch:

If the VM is not healthy afterwards, orch rolls it back to the previous images, reverts the merges, and tells the thread what happened. When it is healthy, the ticket moves to Completed and the developer gets a notification.

None of this is an agent. It is deterministic, it is tested, and when it breaks I read a log and fix it. That is what lets me leave the merge and the deploy to a machine: the only judgement call in the chain has already been made by a person.

It stops at dev

All of that automation, the auto-merge, the VM update, the rollback, stops at the dev environment. Production goes through a separate path where people make the call, on a slower cadence.

This is deliberate too. We do not want automation deciding what reaches production, and we do not want a fast feature list to become the goal. A human QA step on dev and a human decision for production are two different checks, and we kept both.

My own review moved earlier

QA is the developer's step. Mine happens before anything is built. I read the architecture docs, the plans and the prompts, and they tell me about 90% of what is coming. The other 10% is reviewing the output and writing QAQC cases.

So the agent sits between two kinds of human attention: the plan and the prompt going in, and the preview test coming out. It does the engineering in the middle, inside a stack that exists only for its ticket.

If you are setting this up

Give the agent a place to stop that is not "done". For us that is a column called QAQC, a preview link and a thread where the reply goes. Make the person who asked for the change the one who tests it, because they are the only one who knows what they meant. Then script everything after that decision, and keep it out of production.

If your team runs coding agents, who decides that a task is done: the agent, or the person who asked for it?

I'm Panth. I lead the software team at Oizom, and Octupuz is my own project, built in about a month. This post is about the parts of it you can't see in a screenshot: why the backend knows nothing about puzzles, why a board has to come out the same on every device, and how the server checks a solve without trusting the phone.

What Octupuz is

A social feed that looks and moves like Instagram, except every post is a puzzle you play right there. 41 kinds are live: Queens, Nonogram, Pipes, Water Sort, Suguru, Skyscrapers, a whodunit called Case File, and more. New ones every day. When you solve one, your time lands on the post.

It runs on the web and installs as a PWA at octupuz.in. The Play Store is next. The stack: Postgres on Supabase with row-level security, a Svelte client, Capacitor for the Android shell, and games in plain JavaScript.

The spec came first, and the summary failed

The spec is a set of JSON files. screens.json holds the 72 screens: per screen its route, what it reads, what it calls, and every transition with its gesture and motion. decisions.json holds every settled decision with the reasoning behind it. motion.json is 157 KB of animations researched from Instagram's own app.

The build prompt for the second attempt opens with one instruction, in bold: read the specification first, all of it, before writing any code. The sections after it are, in its own words, "a map to the requirements", not the requirements. Where the prompt and the spec disagree, the spec wins.

The same prompt lists two holes the first attempt had, both invisible when you read the SQL and both found only by sending the request an attacker would send to the live API:

• revoke execute ... from anon does nothing, because Postgres grants EXECUTE to PUBLIC and anon inherits it.
• A table-level revoke select ... from authenticated silently disables every RLS policy on that table.

So the working rule became: verify from the outside, with only the public key. Reading a policy is not the same as testing it.

The backend is Instagram, the frontend is the game

This is the decision everything else hangs on. The backend stores social data: players, follows, likes, comments, messages, plays. For each game it stores one string, bundle_path. It does not know what Queens is.

A post is a row with a puzzle column the backend never looks inside:

create table if not exists public.posts (
  post_id       uuid primary key default gen_random_uuid(),
  game_id       uuid not null references public.games(game_id) on delete cascade,
  puzzle        jsonb not null,            -- opaque here; only the game reads it
  post_sequence int  not null,             -- this game's nth puzzle
  ...
);

Today puzzle is usually a seed. The board it draws does not exist until a phone imports the game's bundle and asks for it.

A game is written once, in games/src/<game>/index.js, and each difficulty is a two-line entry file:

const diff = 'hard'
export default (await import('../../src/queens/index.js')).make(diff)

The database stores the relative path (/games/queens/hard.js) and the app loads it with a dynamic import() when a post needs it. Adding a game adds a row. No app update, no store review.

On Android, the JavaScript updates itself over the air: a signed manifest, a staged rollout by percentage, and never mid-session. A new bundle applies the next time the app goes to the background, not while someone is playing. That created one edge case. Posts come from the database the moment a game goes live, but a phone can still be on last week's bundle. So the app tries its own copy of the game first, and if it has none, fetches the same file from the server.

A board has to come out the same everywhere

If a post is a seed, two friends only play the same puzzle if the seed draws the same board on an iPhone, an Android phone, a laptop and the server. The shared random package opens with a list of what is banned in anything that decides a board:

• Math.random, Date.now, performance.now. Time goes into the move log and nowhere else.
• Math.sin, cos, exp, log, pow. Their precision is implementation-defined and differs between V8, JavaScriptCore and SpiderMonkey. Integers, Math.imul, |0 and >>>0 are exact everywhere.
• sort(() => rng() - 0.5). It is not a real shuffle, and it is not stable across engines.
• localeCompare, Intl, toLocaleString in game logic.

Randomness comes from one place: the seed string is hashed with cyrb128 into four 32-bit integers, which seed sfc32. Both use integer operations only.

A golden file keeps it honest. It records the boards a fixed set of seeds draw, and it is committed. If someone changes a generator, the test fails, because every past puzzle of that game would now redraw differently.

Queens is the exception. A Queens board has to be solvable without guessing, which means generating candidates and rejecting the ones a person could only guess at. The rejection test is a backtracking solution count, run thousands of times. That takes about a second: fine once on a server, not fine in a feed scrolling past thirty previews. So Queens boards are generated once, offline, and the post carries the finished grid, with its seed alongside so any board can be traced back to what made it. A few other games are published the same way.

A time on a post means the board was solved

Here is the whole path:

posts row: game_id + puzzle (seed or baked board)
   |
   +--> phone:    import(bundle_path) -> build(puzzle) -> play -> move log
   |                                                                |
   +--> verifier: same rules.js -> replay(puzzle, moves) <----------+
                        |
                        +--> server's own stats -> time on the post

Every tap goes into a move log with a timestamp. For Queens it looks like this:

q34@2100;d05@2600;D4043@3100

A crown at row 3, column 4 at 2.1 s, a dot at row 0, column 5, then one painted run from (4,0) to (4,3). A painted run is one entry, not four. Written as four dots sharing a millisecond, an honest player would look like a script to the timing check.

When you finish, the phone sends the post id and the move log. The verifier loads the puzzle, picks the game's rules module, and replays the log from an empty board:

const run = game.rules.replay(ctx.puzzle, moves, game.difficulty);

if (run.illegal || !run.completed) {
  return json({ error: 'the moves do not replay to an end', illegal: !!run.illegal }, 422);
}

// the client's numbers are ignored
const elapsed = Date.now() - Date.parse(ctx.started_at);
const plausible =
     run.stats.time <= elapsed + 2000
  && run.minMoveGapMs >= game.rules.humanFloorMs
  && (ctx.attempts < 50 || !ctx.p10_ms || run.stats.time >= ctx.p10_ms * 0.25);

The rules.js it runs is the same file the game imports in the browser. No DOM, no clock, no Math.random. One copy of the rules, run by both sides.

A few details make it hold:

• The phone's numbers are thrown away. Time, moves, mistakes and hints all come from the replay. A modified app can lie about its time, and the lie never reaches the board.
• The server's clock starts the game. The play row is created by the first move, with the server's timestamp. A claimed time can't be longer than the time that has actually passed, plus 2 seconds.
• Nobody taps forty times a second. Each game sets a floor for the gap between moves. For Queens it is 90 ms.
• Implausible is kept, not shown. A finish that fails those checks is recorded with flagged = true. The player sees their own result. Nobody else sees a number the server did not believe: no story, no "beat you" notification, no place on the post.
• The server is the only way to finish. The plays table's update policy refuses is_completed, and the function that records a finish is granted to the service role alone. A browser can save where it got to. It cannot say it arrived.

The verifier runs as a Cloudflare Pages Function and talks to Postgres through Hyperdrive, so the database credential never leaves Cloudflare. A Worker can't evaluate code it downloads, so the rules for all 41 games are imported statically and picked by bundle_path, the same path the phone used to choose its copy.

That static list failed once. Nineteen games were carried over from an older project of mine and nobody added them to the verifier's list. The app played those puzzles all the way to the last move, then the server answered 501 and refused to record the finish. The only test that noticed was the one that plays a real board to its end. Now a test reads the verifier's imports and fails the build when any folder under games/src is missing.

There was one more bug, and it was in the database write. Passing ${JSON.stringify(stats)}::jsonb to the Postgres driver looks right. The driver already serialises a string as a JSON string when the target is jsonb, so Postgres stored a string that contained JSON, not an object. ->> 'time' on that is null, and every recorded finish had no time. The fix is two casts, ::text::jsonb, which makes Postgres parse the text instead of wrapping it.

What replay buys is simple to say. A time on a post means a solved board. And nobody ever needs to see another person's board to trust it, which matters, because the board and the move log are the answer.

Decisions that are product, not tech

Some rules in the spec look technical and aren't:

• No winning and no losing. A puzzle is completed or it is not. A wrong move counts a mistake and the board carries on. The UI never says "won", "lost", "score" or "game over".
• Play is a button. Tapping a board in the feed does nothing. Play opens a full-screen player. Scrolling past a puzzle should never start one.
• Nobody sees another person's board, during play or after.
• Sign-in is required. No anonymous access, so every request is attributable to an account that can be suspended. Email sign-in is a code, not a magic link, because a magic link signs in whichever browser opened it.
• Two tables, never one. players are people. games are publishers run by the back office. A game cannot like, play, follow, be messaged or have a story. There is no user_id column anywhere, only player_id and game_id.

Each of those is written in decisions.json with its reason, so the build didn't have to decide them again.

How it was built

The first commit is dated 2026-09-18. I aimed at three things: the best UI and UX I could get, the best board generation, and an architecture that ships a new game without an app update. The last one is where most of this post came from. A game is a bundle and a row, the phone draws the board from the post, and the server checks the result with the same code.

If you've built something where the client reports a result, what do you replay or re-check on the server, and what do you simply trust?

The constraint: by default the orchestrator runs one agent at a time, and a run is capped at 240 minutes. A session sitting idle until I read my notifications would spend both.

I'm Panth, and I lead the software team at Oizom. This is from ticket-tracker, my open-source board where headless Claude Code sessions work tickets next to people. The orchestrator is the workspaces/ folder: one process per board, one Claude Code session per ticket.

The loop

The README draws it like this:

To do  ──orch claims──▶ In progress ──agent reports "review"──▶ Review ──owner──▶ Done
                            │  ▲
           agent asks a     │  │ the answer (form or a comment) resumes
           question ────────▼  │ the same session
                          waiting

"waiting" is not a stage on the board. The ticket stays In progress. It is a phase the orchestrator keeps in its own state file, next to the session id.

What the agent is told

Every agent gets the same brief. Step 3 of it:

If a real decision is the owner's to make (scope, design, anything destructive or outward-facing), call ask_question with blocking: true and real options. Then stop working and finish your turn with status: "waiting" and the question_id. Do not poll for the answer. The orch resumes this same session with the answer when it arrives.

ask_question puts a form card in the ticket thread: a title, optional context, and 1 to 10 fields (single choice, multiple choice, text, number, yes/no, date). A blocking question puts a "Waiting for you" badge on the ticket card and notifies whoever it is addressed to. When I submit it, the card locks and shows my answer.

The agent never decides on its own that it is finished. Its last message is structured output, enforced with --json-schema:

const OUTCOME_SCHEMA = {
  type: 'object',
  properties: {
    status: { type: 'string', enum: ['review', 'waiting', 'blocked'] },
    summary: { type: 'string' },
    question_id: { type: 'string' },
  },
  required: ['status', 'summary'],
};

Three ways to end a turn, and each one means something different to the orchestrator.

The split between the last two matters. waiting means a person is needed. blocked on a long ticket almost always means the turn ran out of room, so the orchestrator resumes it without asking anyone, a bounded number of times.

Why the session doesn't wait

Each run is one claude -p process. When it exits, the orchestrator posts a receipt with what the turn cost and marks the slot free, so the next ticket in To do can start. With maxAgents: 1, a session waiting for me would hold the only slot. Every other ticket on the board would wait behind my inbox.

There is a second reason in the brief. The session is headless, so when a turn ends the process exits and takes its background tasks with it. A turn either finishes its work or ends cleanly. There is no half-alive process in between for me to find later.

And nothing polls. The agent is told not to. The orchestrator doesn't sit on a timer either: it wakes on the board's revision, and the answer reaches it as an event.

How the answer gets back in

When I submit the form, the tracker puts a question_answered event in the agent's inbox. The orchestrator drains its inbox, sees the event for a ticket in waiting, and resumes the session with a prompt built from my answer:

answered: (t, q) => `The owner answered your question "${q.title}" on ${t.key}.

Values: ${JSON.stringify(q.answer?.values ?? q.values ?? {})}
Comment: …

Carry on with the work.`,

The resume itself is one flag. If the session file for that id still exists, the orchestrator passes --resume <id>; if not, it starts a fresh session and resets its cost to zero.

...(resume ? ['--resume', sessionId] : ['--session-id', sessionId]),

The agent picks up with its whole context: the files it read, its plan, the question it asked. It doesn't re-read the repo to work out where it was.

A few edges are handled on the way:

• A comment instead of the form. If I reply in the thread rather than filling in the card, that also resumes the session, with my message in the prompt.
• No slot free. If another ticket is running, the ticket's status line says "Got your reply · queued until an agent is free", so an answered question never looks ignored.
• Cancelled or expired. The prompt says so, and tells the agent to read the thread, then go ahead with the most conservative option or ask again.
• A missed event. Every fourth loop, a ticket still waiting has its question fetched directly, in case the event was lost.
• The answer is progress. It resets the ticket's automatic-continuation count to zero, and so does any comment from me.

Cost stays honest across resumes

Claude Code reports total_cost_usd cumulatively across --resume. In a session that asked two questions, summing the reported totals would count the first turn's cost three times. So the receipt for a turn is the difference from the session's last reported total, and the session's running total lives in the orchestrator's state.

What it costs me

A question costs no compute while I'm away: there is no process to keep alive. The price is latency: the ticket sits until I answer, and nothing else picks it up. I'm fine with that. The questions an agent asks are the decisions I said were mine.

How does your agent setup handle a question it can't answer itself: does the session wait, or does it end the turn and come back?

I'm Panth, and I lead the software team at Oizom. The tracker is ticket-tracker, my open-source board where coding agents work tickets next to people. This is the first decision in its plan, and the one that would cost the most to reverse.

The Firebase default I turned down

The usual Firebase shape is that clients write Firestore directly, security rules decide who may, and triggers do the side effects.

I kept half of it. Clients still read Firestore directly. Every write is a POST /api/{command}, and the rules deny writes except for a person's own pointers.

Two things decided it.

Nearly every write has a second document to keep in step: the key counter, the activity log, the other end of a link, notifications, webhooks.

And rules can't carry the permissions. Try writing this one as a rule somebody can maintain: a commenter may change a ticket's stage only between the stages in their grant, and only on tickets assigned to them.

The reason that mattered more

REST and MCP need the same logic anyway. With client writes, every rule would exist twice, once in firestore.rules and once in the API, and the two would drift.

So a command is defined once, as a spec in the shared package:

export const messagePost = defineCommand({
  name: 'messagePost',
  scopes: ['comments:write'],
  permission: "can(comment); ticket state == 'active'",
  errors: ['forbidden', 'not_found', 'conflict', 'too_large'],
  req: z.object({ /* ticket, body, replyTo, attachments … */ }),
  res: OkResSchema,
});

(The req is shortened here.) The registry holds 52 of these. The backend parses requests with the spec, and the frontend's command(name, input) is typed by the same map.

Three doors, one runner

All three end in one runner, and it does the same things in the same order:

registry lookup → scope narrowing → parse the request → idempotency claim
  → handler → parse the response → bump the board's revision → answer

The doors are routes in one HTTPS function. The alternative was a function per endpoint, and sixty functions means sixty cold starts, sixty deploy units and sixty minimum-instance bills. One service stays warm on the app's traffic alone, so REST and MCP calls land on warm instances.

Agents can do nothing people can't

A token only narrows. What it may do is its scopes intersected with can() for the principal it acts as, a person or one of their agents. The scope check at the door only answers early. The command decides.

A command that lists no scopes is app-only, and no token reaches it. There is no API-only write path that could drift from the app. And there is deliberately no MCP tool that deletes a ticket.

Every command also runs with who is acting and through which door (app, api or mcp), so a change made with an agent's token is attributed to that agent and to the token.

What the choke point bought later

Because every write passes through one runner, it became the place to announce that something on a board changed. After a command succeeds, the runner makes one small Realtime Database write to the board's revision. The app and the SDK listen to that node and ask for the delta, not poll for it. A replayed idempotent request bumps nothing, because nothing changed.

That came later, after I measured the bill and found the orchestrator's polling was 88% of all requests. It went into one place because the place existed.

The cost

A write takes about 150 to 400 ms, not about 50. Optimistic updates hide it in the app. Cold starts are covered by minInstances: 1 on the function in production.

That is the whole price: a slower write, hidden by optimistic updates, for one copy of every rule.

If your app has a UI, an API and an MCP server, do all three share one write path, or does each one have its own copy of the rules?

I'm Panth, and I lead the software team at Oizom. The tracker is ticket-tracker, my open-source board where each ticket runs its own headless Claude Code session. This post is about one small piece of it: how the board knows an agent is alive.

The status that lies

The obvious design stores a status on the ticket: working, then done. The agent writes working when it starts and done when it finishes.

If the process dies in between, nothing writes anything. The ticket says working for as long as nobody looks into it. A stored status is only as true as the last thing that managed to write it, and a dead process writes nothing.

What the beat carries

The orchestrator sends the beat, not the agent. It is one call:

POST /v1/heartbeat
{ ticket?: KEY,            // omit = an agent-level beat, with no ticket
  state: 'working' | 'idle' | 'done' | 'error',
  message?: string,        // "Running tests (3/12)", up to 200 characters
  progress?: number }      // 0 to 1

It goes out every minute while the agent works and once more when the work ends. The message comes from the agent's task list: the orchestrator reads the list on every fifth beat and repeats the last message on the four in between.

A beat never touches the ticket document. It has its own small record per agent and ticket, so a beat a minute doesn't retrigger the ticket's search indexing or re-render everything that listens to the ticket.

Health is computed where it is read

The stored state has four values. The health on screen has six, and the extra ones come from the clock:

export const HEARTBEAT_INTERVAL_MS = 60_000;
export const HEARTBEAT_STALE_MS = 75_000;

export function deriveAgentHealth(status, now, staleAfterMs = HEARTBEAT_STALE_MS) {
  if (!status) return 'none';
  if (status.state !== 'working') return status.state;
  return now - status.lastBeatAt > staleAfterMs ? 'stale' : 'working';
}

75 seconds is one missed beat plus a little slack.

Only working can go stale. Idle, done and error are statements that stay true until the next beat. working is a claim that has to be renewed.

This one function is in the shared package. The card, the ticket header, the People & roles page and the Agents page all call it, so they can't disagree about an agent. When several agents are on one ticket, the card shows the loudest: error first, then no signal, then working, idle and finished.

The 15-second clock

A derived value only changes when something recomputes it. If no new beat arrives, no data arrives either, so the screen needs its own clock:

export const HEALTH_TICK_MS = 15_000;

export const healthClock = readable(Date.now(), (set) => {
  const id = setInterval(() => set(Date.now()), HEALTH_TICK_MS);
  return () => clearInterval(id);
});

15 seconds is a quarter of the beat interval, so the longest a dot can be wrong is 15 seconds. It is one interval shared by every dot on screen, and it stops when the last subscriber goes.

What it cost, and what I changed

The first version kept each beat stream as a Firestore document. I measured the project's bill from 23 to 26 September 2026 and the heartbeat was on it twice:

• 1,440 writes per agent per day, each a Firestore transaction through a function, followed by one read in every open tab.
• A scheduled sweep that ran every minute to notice silence: 4,261 runs in three days.

Two changes came out of that.

Beats moved to the Realtime Database, which bills bandwidth and not operations. A beat is now about 100 bytes. The shape everything above the database reads stayed the same, so no screen changed.

The sweep stopped looking for silence. Noticing silence never needed a server, because staleness is now - lastBeatAt wherever the beat is read. The sweep now runs every three minutes and does the one thing that does need a server: if a working agent has been silent for five minutes, it tells the agent's owner, once per silence, because nobody may have the board open. The next beat re-arms it.

If you are building one

Store the facts: the last state the agent claimed and when it claimed it. Derive the judgement, alive or not, at read time, in one function every screen shares. Then give the screen a clock, or the judgement never updates.

How does your agent setup tell you a run died: a timeout on the server, or something you work out from the last thing it said?

I'm Panth, and I lead the software team at Oizom, where Envizom is the platform. This is why our agent pipeline isolates per ticket, and what the isolation buys beyond running tasks in parallel.

Where the mistakes were

With an agent, one developer can keep three tasks in flight. That holds while all three go into the same branch and the same release. Once they are due in three different releases, the developer juggles repos and branches and has to remember which change belongs to which PR.

The development was usually fine. The mistakes were a change on the wrong branch, or a fix that only worked on one machine. As agents got faster I saw more "works on my machine" on my team, and client expectations kept going up at the same time.

What was missing was a sandbox you could fork for one task and turn into a PR when it was done.

What a ticket gets

When a developer drags a ticket to Approved, orch, our orchestrator, provisions a workspace for it:

An agent does the work inside that workspace. The developer talks to it only through the ticket thread, tests the preview, and drags the ticket to Git Push when it is right. Until then, nothing the ticket does reaches the shared dev branch.

Isolation is the safety decision

A stack per ticket looks like a way to run things in parallel. That is half of it. The other half is safety.

An agent that runs a bad migration or deletes the wrong rows does it to the ticket's copy of the database. An agent that breaks the backend breaks the ticket's backend. Nobody else's work is touched, because nobody else's work is in that box.

The shared environment is reached only at Git Push, and that step is a script, not the agent. Orch commits the ticket branches, merges dev into them, opens and merges the PRs, waits for CI, rolls the new images onto the dev VM and runs health checks. If the VM is unhealthy afterwards, orch rolls it back to the previous images, reverts the merges and tells the thread what happened.

All of it stops at dev. Production goes through a separate path where people make the call.

Independent timelines

The second thing isolation gives us is time. A ticket can sit in QA for a week while three others ship. Its branch, its data and its thread wait for it, and nothing else waits on it. Three tickets due in three releases are three separate stacks, and each one ends in its own PR.

It is also why the audit trail holds together. What was asked, what was built, which preview was tested and what feedback went back all sit on one ticket, not mixed with another ticket's work.

What made the copies cheap

A database copy per ticket only works if copying is cheap. In Envizom V3, which we are rebuilding from scratch, the backend runs in dev without a single extra database credential. Every store has a dev driver:

SQLite runs inside the Go process, so the dev kit needs no extra containers, and copying a database means copying a few files. That change made our sandboxes about 10x cheaper and faster than sandboxes with full databases.

The dev VM that holds the devboxes and the ticket stacks costs about ₹4–6k a month and handles six tickets in parallel.

If you are building one

Start with isolation, not with the agent. If a task can't run in its own branch, with its own services and its own data, nothing else in the pipeline is safe. Then put the agent inside the box, and keep every step that leaves the box scripted.

If your team runs a sandbox per branch or per ticket, which part was hardest to copy: the code, the services or the data?

I'm Panth, and I lead the software team at Oizom. This is from ticket-tracker, the tracker I open-sourced, where AI agents are board members and a small orchestrator turns each ticket into a headless Claude Code session.

How a ticket becomes a session

The orchestrator (workspaces/lib/orch.mjs, about 600 lines, zero dependencies) runs one headless Claude Code session per ticket. Each time the agent has something to do, it starts claude -p once: a fresh ticket, rework, an answer to a question, a new comment, or carrying on by itself. That is one turn. The process exits, and the last line of its stream-json output is a result line.

The first turn creates the session with --session-id. Every later turn on the same ticket continues it:

const resume = e.session_id && fs.existsSync(sessionFile(e.session_id));
const sessionId = resume ? e.session_id : randomUUID();

const args = [
  '-p', prompt,
  '--output-format', 'stream-json', '--verbose',
  ...(resume ? ['--resume', sessionId] : ['--session-id', sessionId]),
  // ...
];

The session keeps its history, so the agent picks up where it stopped. That is the point of resuming. It is also why the cost number misleads.

The trap

The result line carries total_cost_usd, usage (input, output and cache tokens), num_turns and modelUsage. Before the change, the orchestrator already read total_cost_usd off every result line and wrote it to a local log. Nothing in the app showed it.

Putting the number on the ticket looks like a one-liner: post total_cost_usd when the run ends. It isn't, because that field is the session's running total, not the run's. Ticket OCZ-5 shows it plainly. Run 8 did almost nothing, one API call and zero output tokens, and reported $392.83. So did run 7.

Read naively, that line says run 8 cost $392.83. Add those lines up and every earlier run is counted again on each later turn.

The fix: keep the last total, post the difference

The orchestrator keeps per-ticket state in a JSON file. Two fields handle cost: session_usd, the last total that ticket's session reported, and session_usage, the last token counts. When a turn ends:

const prevUsd = e.session_usd ?? 0;
const costUsd = Math.max(0, r.total_cost_usd - prevUsd);

const now = {
  input: u.input_tokens ?? 0,
  output: u.output_tokens ?? 0,
  cache_read: u.cache_read_input_tokens ?? 0,
  cache_write: u.cache_creation_input_tokens ?? 0,
};
const prev = e.session_usage ?? { input: 0, output: 0, cache_read: 0, cache_write: 0 };
const usage = Object.fromEntries(Object.keys(now).map((k) => [k, Math.max(0, now[k] - prev[k])]));

e.session_usd = r.total_cost_usd;
e.session_usage = now;
saveState();

The turn's cost is the new total minus the old one, clamped at zero. The token counts get the same treatment. Then the new total is saved for the next turn.

Three cases decide whether the numbers stay honest:

The resume check matters more than it looks. The orchestrator only resumes if the session's .jsonl file still exists under ~/.claude/projects/. If it doesn't, the turn starts a new session, and a new session counts from zero. Subtracting the old total from it would wipe that turn's cost.

Where the number goes

Money spent on a ticket is a fact about the ticket, so it goes where the ticket's other facts go. When a run ends, the orchestrator posts one message on the ticket, the turn receipt. In the design doc's example, the body reads:

Turn 3 · review · $1.24 · 12 min

Beside the text it carries a run field: the turn number, the outcome (review, waiting, blocked, failed, stopped or timeout), cost_usd for this turn, session_usd as Claude Code reported it, duration, API calls, model and the token deltas. Both numbers are kept on purpose. The turn cost is what the person reading the thread cares about. The session total is what you check the arithmetic against.

The server stores the receipt on the message and, in the same write, adds it to the ticket's cost, the board's cost and the day's row the Analytics view draws. So the card shows the spend next to its other signals, and the board shows cost per day and which tickets cost the most.

Only the orchestrator posts receipts. The app's comment box never does. Each receipt goes out with an Idempotency-Key made from the ticket and the run number, so a retried request doesn't count a turn twice.

What changed

Cost per turn in the thread is a deliberate design choice, not a debug feature. The person who asked for the work sees what each turn of it cost, in the same thread where they asked.

The lesson for anyone running Claude Code headless: treat total_cost_usd as a meter reading, not a bill. Store the last reading per session and bill the difference.

If you run Claude Code from scripts or CI, where does the cost of a single run end up for you, and does it account for --resume?

Neither I nor my developers need a laptop to ship code anymore. That sounds like a stunt, so here is the boring version: on our team a developer's working day happens inside a ticket thread, and the parts that make that possible are mostly scripts, not AI.

This post is the long version of the short video above. It covers what the pipeline does, how the pieces fit, what we deliberately kept dumb, and where we drew the line on automation.

The problem we were actually solving

I lead the software team at Oizom, an environmental monitoring company. Our platform is called Envizom, and it is several repos: hardware drivers, a simulator, a backend, a frontend. Getting a working local setup took a new developer days, and even for the rest of us "does it work on your machine" was a real question.

When we started using AI coding agents, the first thing we noticed was not the speed of the agent. It was the waiting. One developer, one agent, one task at a time, sitting there watching a terminal. The agent was fast. The developer was the bottleneck.

We wanted five things:

1. A developer should be able to hand off a task and go do something else.
2. Every task should run in isolation, so an agent cannot break anyone else's work, and so features can live on different timelines. One ticket can sit in QA for a week while three others ship.
3. Nothing about "we use agents" should weaken the normal engineering process: branches, commits, pull requests, review, CI.
4. Every prompt should be stored with the work it produced, so we can review and coach how people ask for things.
5. The ticket should be the audit trail: what was asked, what was built, what was tested, what feedback was given, in one place.

The ticket is the interface

We already had our own ticket tracker. So we made the ticket the only thing a developer touches.

A developer writes what they want in a ticket, in plain English, and drags it to Approved. That is the whole spec. There is no IDE open, no terminal, no prompt window.

On our dev VM each developer has their own devbox: VS Code running in the browser, with our orchestrator inside it. We call it orch. Every developer is paired with their own agent, and their orch only picks up tickets assigned to that agent. So the board is shared, but the machines and the agents are not.

Each ticket gets its own full stack

When orch sees an approved ticket it provisions a workspace for it:

• a fresh branch in every repo, named after the ticket
• its own backend, drivers, simulator and frontend, running in that workspace
• a copy of the dev database, so the ticket can change data freely
• a private preview URL for the ticket, with an HTTPS certificate issued on demand

This is the part that made everything else possible. Because every ticket is fully isolated, a developer can have several tickets in flight at once, and agents cannot step on each other. If an agent does something silly to the database, it did it to a copy.

An agent does the work, but only from the thread

Inside that workspace an AI coding agent picks up the ticket. Orch gives it the ticket text and the workspace, and the agent does the actual engineering work.

The developer only ever talks to the agent through the ticket thread. When the agent is done it posts the preview link and moves the ticket to QAQC. The developer opens the preview, tests it, replies with feedback in the same thread, and moves the ticket back. The same agent, with the same context, picks up where it left off.

In the video I asked for a camera layout change on a device page. The agent shipped it, posted the preview, I replied "also handle more than 4 cameras with a horizontal scroll, and set up 8 test cameras on this device", and it did both, including creating the test data in the ticket's own database copy.

Two of the design goals pay off here, and they are the ones people underestimate:

• Every prompt is saved. The thread is the prompt history. I can read how each person on the team asks for things and coach them on it. That has been more useful than any prompting guide.
• The thread is the audit trail. What was asked, what was built, which preview was tested, what feedback was given. All in one place, attached to the ticket. Because each ticket has its own stack, that trail is also self-contained: a ticket can pause in QA for days without blocking anything else.

Shipping is a drag-and-drop

When the developer is happy they drag the ticket to Git Push. From here on, no agent is involved. Orch:

1. commits the work on the ticket branches
2. merges the current dev branch into them
3. opens a pull request in each affected repo
4. merges the pull requests
5. posts the PR and commit links back into the ticket thread
6. waits for CI
7. rolls the new images onto the dev VM
8. runs health checks against the VM

If the VM is not healthy afterwards, orch rolls the VM back to the previous images and reverts the merges. On its own. Then it tells the thread what happened.

When it is healthy, the ticket moves to Completed and the developer gets a notification. In the video that is the whole ending: two notifications, and the change is live on dev.

What is AI and what is not

This is the part I am proudest of, and the part I think most teams get backwards.

Roughly 90% of this pipeline is plain scripted code. Watching the board, provisioning the stack, copying the database, issuing certificates, driving git, opening PRs, waiting on CI, deploying, health checks, rollback. None of that is done by an agent. It is deterministic, it is tested, and when it breaks I can read a log and fix it.

The agent does exactly one thing: the engineering work inside an isolated workspace, in response to a human's words in a ticket.

We tried the other way first, with the agent doing more of the operational steps. It was slower, more expensive, and worse at exactly the tasks that need to be boring. An agent deciding how to merge is not a feature. A script that always merges the same way is.

What it gave us

• One developer runs several tickets in parallel instead of babysitting one agent.
• No local setup. New people are productive on day one. The laptop can be closed.
• Every feature still gets its own branch, commits, PR and review. Nothing about our process got looser.
• Every prompt the team writes is stored with the ticket.

Three deliberate choices

People sometimes read these as gaps. They are decisions.

• Cost is visible on every turn. Each agent turn posts its cost into the ticket thread. That is intentional. The person asking sees what the ask cost, right where they asked it, and it changes how they write the next one.
• QA is human. The developer tests the preview and decides whether the ticket moves forward. That step is where we check the thing that was built is the thing that was asked for. An agent grading its own work would remove the one review that matters most.
• This pipeline is for dev only. Rollback, auto-merge, VM updates: all of it stops at the dev environment. Production goes through a separate path where real people make the call, on a slower cadence. We do not want automation deciding what reaches production, and we do not want a fast feature list to become the goal.

What is next

The one thing I keep looking at is the dev VM itself. The per-developer devboxes and the per-ticket stacks all run on one rented box. It costs about ₹4–6k a month and comfortably handles six tickets in parallel, which is fine at our size, but it is still one machine and it is still rented.

The plan is almost embarrassingly simple. We have a pile of laptops. Turn them into VMs. The whole point of this pipeline is that your main laptop can be closed while the work happens, so the machine doing the work might as well be an old laptop on a shelf in the office. A one-time purchase of in-house hardware will never cost what a cloud VM costs over a year, and it keeps the same isolation model: each developer's devbox on its own box.

We have not done this yet. When we do, I will write it up.

If you want to build something similar

Start with isolation, not with the agent. If every task cannot run in its own branch, its own services and its own data, none of the rest is safe. Then make the handoff surface something your team already uses. For us that was the ticket. Then, and only then, put an agent inside the box.

And keep the box scripted. The agent should be the smallest part of the system.

If you have questions, ask in the comments. I am collecting them for a follow-up post.

I lead the software team at Oizom, where Envizom is the platform that turns data from air-quality and environmental monitors in 80+ countries into something cities and industries can act on. The walkthrough video is on YouTube; find me on LinkedIn.

I'm Panth, and I lead the software team at Oizom, where we build air-quality monitoring. This is the written version of a four-part screen recording I made while building it. The videos are below if you would rather watch me get it wrong first.

What a data point looks like

A single reading from a device looks roughly like this:

type DataPoint = {
  device: string;
  t: number;                       // epoch seconds
  d: Record<string, number>;       // decimals: co2, o2, pm25, temperature …
  s: Record<string, string>;       // strings: location, firmware, tag …
};

d holds the gas concentrations and other numbers. s holds whatever labels the device or the user attached. Any key can be present or absent on any reading. Thousands of devices report every minute, so a report query can easily touch a million of these.

The backend ran on a DigitalOcean droplet with 2 GB of RAM, and it kept dying with out-of-memory errors on exactly those queries.

Why a million objects is the problem

Three things happen when V8 holds a million of those objects, and none of them show in the code.

Every property access is pointer chasing. points[i].d.co2 is not one memory read. The array holds a reference to the point, which holds a reference to d, which is a hash map that hashes "co2", finds the slot and hands back the number. In C the compiler knows the offset of co2 before the program runs. In JavaScript every value is a reference to somewhere else.

Every object is a hash map. An object with ten keys is buckets plus a hash on every access. A million of them is ten million hashed lookups for one pass over the data.

Every object is garbage. Each {} is an allocation the garbage collector tracks. Create a million per request and the collector works harder than your code. The numbers are boxed doubles spread across the heap, so the working set is far bigger than the data.

So the fix is not to optimize the loop. The fix is to stop creating objects.

Treat the data as an image

Put timestamps on the Y axis and gases on the X axis, and the readings are a grid. Some cells are empty because that device didn't report that gas at that time. That is what an image is: a rectangle of numbers in one contiguous array, addressed by row * width + column.

            co2    o2    pm25   temp
t1        [ 412   20.9   35.2   26.1 ]
t2        [ 415    –     36.0   26.3 ]
t3        [  –    20.8   34.1    –   ]

Instead of an array of objects, the table keeps:

• one Float64Array for all decimal values, times.length * gases.length long
• one Int32Array for string values, holding an index into a dictionary of unique strings, because labels repeat on almost every row and there's no reason to store "Bangalore" a million times
• three small arrays for the axes: timestamps, gas names, label names

A reserved value marks an empty cell. Time 0 is an unused row; an empty string is an unused column.

No object per reading, one allocation for the whole table. Reading co2 at row i is values[i * gases.length + gasIndex.co2]: the offset arithmetic JavaScript normally won't give you.

One class on top

The buffers are ugly to use directly, so everything goes through one class. This is the surface from the first video, and it barely changed later:

class DataPointTable {
  static create(rows: number, gases: number, labels: number): DataPointTable;
  static fromJSON(points: DataPoint[]): DataPointTable;
  static fromBuffer(buf: ArrayBuffer): DataPointTable;

  // axes
  times(order?: 'asc' | 'desc'): Iterable<number>;
  gases(): Iterable<string>;
  labels(): Iterable<string>;
  timeIndex(t: number): number;         // -1 if absent
  gasIndex(name: string): number;
  addTime(t: number): number;           // returns the row index
  addGas(name: string): number;
  removeTime(t: number): void;

  // cells
  get(ti: number, gi: number): number | undefined;
  set(ti: number, gi: number, v: number): void;
  row(ti: number, onlyExisting?: boolean): Iterable<[gas: string, v: number]>;

  // whole table
  toJSON(): DataPoint[];
  toBuffer(): ArrayBuffer;              // for sending to the browser
  compress(): DataPointTable;           // drop unused rows and columns
  stats(): { bytes: number; rows: number; used: number };
}

Nobody else in the codebase has to know there are buffers underneath. A report generator asks for a row and gets [gas, value] pairs. It never sees a Float64Array.

Version one: correct, and 8x slower than JSON

The first version kept rows sorted by timestamp, descending, because queries always want time order. So addTime had to find where the new row belonged, find the nearest free slot on either side, and shift every row in between by one. Adding a gas column was worse: every row moves to make room for the new cell, so the copy runs from the back of the buffer to the front, zeroing new cells as it goes.

I over-allocated to soften it: five spare rows and five spare columns at a time, so a burst of inserts doesn't reallocate each time. When the table is full, allocate a bigger one, copy, carry on.

Every function test passed, toJSON(fromJSON(x)) round-tripped, and compress() reclaimed the spare cells. Then I loaded a million points with thirty columns each. It never ran out of memory, which JSON always did at that size. It was also 8x slower than parsing the JSON.

What was slow

Not the typed arrays. The shifting.

The backend loads rows from the database, maybe runs one or two operations, then sends the data to the browser or feeds it into a report. It almost never inserts a row into the middle of a table. I was paying for ordered insertion on every load and getting nothing for it.

Two smaller things added to it. Finding a column by name was a linear scan of the gas array on every cell write, so a name → column index hash map came back, but only for the axes, which have thirty entries, not a million. And a binary search on the sorted timestamps didn't help, because the search was never the cost.

Version two: stop keeping order

The rewrite dropped the one assumption that hurt. Rows are no longer sorted. The table keeps two free lists instead:

private freeRows: number[];      // indexes of rows with t === 0
private freeCols: number[];      // indexes of columns with name === ''
private rowOf = new Map<number, number>();   // timestamp → row index
private colOf = new Map<string, number>();   // gas name → column index

addTime is now: if the timestamp is known, return its row; otherwise pop a free row and record it in the map. No search, no shift. If the free list is empty, allocate a bigger table, copy once and push the new rows onto the free list. removeTime zeroes the row and pushes its index back. No loop in the hot path.

A query knows how many rows it will return before it reads them, so the table is created at the right size up front, and the grow-and-copy branch almost never runs.

Order comes back only where it is read. times('desc') collects the live timestamps, sorts them once and yields them. One sort per query instead of one shift per insert.

The figure in the video description, roughly a 400% gain, is the load-and-serve path measured against the JSON version on the same box.

Skipping JSON entirely

That is why the class has toBuffer and fromBuffer. If the point is to stop building a million objects, serializing them to JSON for the browser undoes it. So the table serializes itself as its own buffers: a small header with the dimensions and the axis arrays, then the two value buffers as they already sit in memory.

On the server, database rows go straight into the table without a DataPoint[] ever existing. The buffer goes to the browser as is (base64 where it has to cross a text boundary), and the frontend rebuilds the same class from it. fromJSON and toJSON stay for tests and the odd tool; the production path never touches them.

What I'd tell past me

• The win came from removing allocations, not from typed arrays as such. Typed arrays are just the only way JavaScript lets you allocate a million numbers in one go.
• Don't maintain an invariant the read path doesn't need. Sorted rows cost me a full version.
• Keep the buffers behind one class. Every place that touched a Float64Array directly would have been a place the rewrite had to visit.
• Measure against what you're replacing, on the same box. "Never runs out of memory" isn't a win if it's 8x slower.

I had Claude write the function and performance tests for the second version. In the fourth video, the round-trip comparison catches a bug in compress() I wouldn't have found by hand.

Have you hit the point where plain objects stopped scaling in Node.js, and what did you move to?

Related reading

• IDW heatmaps, 5 seconds to 30 ms: the same instinct on a 1 km grid, where the "look at it as an image" idea came from.
• Postgres triggers and our Redis cache at Oizom: the other half of the same backend.
• await on a sync function in Node.js: another thing that isn't free even when it looks free.

That is Node.js. In Chrome the same loop went from 3 ms to 1,500 ms. The function does nothing and is not async; the whole difference is the await.

I'm Panth, and I lead the software team at Oizom. I measured this in May 2025, after Prime said in a video that even sync code behind a Promise waits for the event loop. I didn't believe it, so I tested it.

An async function starts synchronously

My first guess was that an async function with nothing to wait for would just run inline.

async function a() {
    console.log('a');
}
console.log('1');
setTimeout(() => console.log('t1')); // runs later, after the current code
a(); // sync body, so it should run right here
setTimeout(() => console.log('t2'));
console.log('2');

1
a
2
t1
t2

It does. a prints between 1 and 2.

.then and await do not

That could just be the async keyword, so the next test used a Promise constructor, .then and await.

async function a() {
    console.log('a1');
    const p = new Promise((r) => {
       console.log('p>>>')
       r();
    }).then(() => console.log('p<<<'));
    console.log('a2');
    await p;
    console.log('a3');
}
console.log('1');
setTimeout(() => console.log('t1'));
a();
setTimeout(() => console.log('t2'));
console.log('2');

1
a1
p>>>
a2
2
p<<<
a3
t1
t2

The Promise body runs at once (p>>>), but the .then callback and everything after await wait until the current synchronous code has finished: p<<< and a3 print after 2. They still run before the timers (t1, t2). The Promise resolved immediately and the work was still deferred.

Measuring what the deferral costs

Five ways to call an empty function a million times:

1. a sync function, called directly
2. a sync function, called with await
3. an async function, called without await
4. an async function, called with await
5. an async function, called without await, then await Promise.all on all the results

function sf() {}
async function af() {}
function run(fn, next) {
  const start = Date.now();
  function done() {
    console.log(Date.now() - start + "ms");
    if (next) next();
  }
  fn(done);
}
function case1() {
  run((done) => {
    for (let index = 0; index < 1000_000; index++) {
      sf();
    }
    done();
  }, case2);
}
function case2() {
  run(async (done) => {
    for (let index = 0; index < 1000_000; index++) {
      await sf();
    }
    done();
  }, case3);
}
function case3() {
  run((done) => {
    for (let index = 0; index < 1000_000; index++) {
      af();
    }
    done();
  }, case4);
}
function case4() {
  run(async (done) => {
    for (let index = 0; index < 1000_000; index++) {
      await af();
    }
    done();
  }, case5);
}
function case5() {
  run(async (done) => {
    const promises = [];
    for (let index = 0; index < 1000_000; index++) {
      promises.push(af());
    }
    await Promise.all(promises);
    done();
  });
}
case1();

Every runtime pays for the await, and Chrome pays the most. A fresh Chrome start did not change that, so it was not my open tabs.

With a body in the function

Runtimes might special-case empty functions, so the second run gave both functions one line of work, cnt++, and reset the counter after each case:

let cnt = 0;
function sf() {
  cnt++;
}
async function af() {
  cnt++;
}
function run(fn, next) {
  const start = Date.now();
  function done() {
    cnt = 0;
    console.log(Date.now() - start + "ms");
    if (next) next();
  }
  fn(done);
}

Same picture. The cost is the await, not the empty body.

What I took from it

I didn't know the cost of this before I measured it. It also explained why people who write Rust and Go say async is hard, when it had always looked easy to me in JavaScript. It is easy to write, and it has a price: in these runs, putting await in front of code with nothing to wait for made the loop at least 5x slower in every runtime, and several hundred times slower in Chrome.

For high-performance code, that pushed me towards sync callbacks on hot paths. I am thinking about changing my libraries and framework to support sync callbacks fully, and maybe rewriting the backend from scratch to use them.

Where has an await on something that was already there cost you time?

I'm Panth, and I lead the software team at Oizom. Our Node.js backends sit on Postgres and Redis. This is how we got from plain cache-aside to trigger-driven invalidation, step by step, including the bug in the first version (every backend instance heard every change) and the table-backed listener that fixed it.

Here is where we started, the usual cache-aside:

async function getUser(userId) {
    const cacheResult = await cache.get(`USERS:${userId}`);
    if (cacheResult !== null) return JSON.parse(cacheResult);
    const result = await (...) // do some db call
    await cache.set(`USERS:${userId}`, JSON.stringify(result), "exp", 3600 * 24);
    return result;
}

So, how would you clear the cache? Because when a user updates their profile name, the changes don't seem to show up immediately.

async function updateUser(userId, data) {
    await (...) // do some db calls
    await cache.del(`USERS:${userId}`);
}
async function deleteUser(userId) {
    await (...) // do some db calls
    await cache.del(`USERS:${userId}`);
}

So, you think you've nailed caching, huh? But have you really? Let's break it down:

1. Why is the cache invalidation code stuck in updateUser and deleteUser? It's super easy to overlook, especially when you create a new function like updateUserForSomeNewTask.
2. What if updating or deleting a user also affects other stuff, like the user's organization or module expiry? If you're caching all that too, you'll need to handle invalidation for everything. Who can keep track of all that?
3. Even with a simple cache get, you have to add two layers of code—one before and one after your implementation—to check if the cache exists and then set it. It's a bug waiting to happen.
4. What if you change the output schema for getUser? You either change the key prefix from USERS: to USERS_V2: or flush the USERS: cache. Who's going to remember that?
5. The big one: what if a user is updated from another service? Like someone directly edits the database, or it happens through a different backend, admin backend, or background workers? You'll be stuck with a caching bug for 24 hours, wasting time debugging.
6. And what about when you want to create a function like getUsers(userIds)? If you think that's easy, try making caching logic for getModulesForUser(userId, moduleIds) where moduleIds = undefined to get all modules.

See how managing this can get tricky as you go along? These are real problems that happen more often than you'd think.

All of this can be sorted out with a good design system.

I'm going to talk about my caching and invalidation tricks using Redis and Postgres with NodeJs.

Let's tackle everything step by step.

1. Let's start by turning the caching code into a function.

// caching function for single object
// we can similarly create a abstract wrapper to cache multiple objects
function CacheFunc(func, getKey, expInSec) {
    async wrappedFn(...args) {
        const key = getKey(...args);
        const cacheResult = await redis.get(key);
        if (cacheResult !== null) return JSON.parse(cacheResult);
        const result = await func(...args)
        await redis.set(key, JSON.stringify(result), "exp", expInSec);
        return result;
    }
    async invalidate(...args) {
        const key = getKey(...args);
        await redis.del(key);
    }
    return Object.assign(wrappedFn, { invalidate })
}

const getUser = CacheFunc(async function (userId) {
    const result = await (...) // do some db call
    return result;
}, (userId) => `USERS:${userId}`, 3600 * 24)
async function updateUser(userId, data) {
    await (...) // do some db calls
    await getUser.invalidate(userId);
}
async function deleteUser(userId) {
    await (...) // do some db calls
    await getUser.invalidate(userId);
}

this looks way more promising.

2. But there's still a question: What do we do when a new lightweight user function needs caching and invalidation? so let's move all the invalidation stuff to an onUserChange function.

async function onUserChange(userId) {
    await Promise.all([
        getUser.invalidate(userId),
        getUserWithFewColumns.invalidate(userId),
        // add all the other, user caching functions
    ])    
}

const getUser = CacheFunc(async function (userId) {
    const result = await (...) // do some db call
    return result;
}, (userId) => `USERS:${userId}`, 3600 * 24)
const getUserWithFewColumns = CacheFunc(async function (userId) {
    const result = await (...) // do some db call
    return result;
}, (userId) => `LIGHT_USERS:${userId}`, 3600 * 24)

async function updateUser(userId, data) {
    await (...) // do some db calls
    await onUserChange(userId);
}
async function deleteUser(userId) {
    await (...) // do some db calls
    await onUserChange(userId);
}

It's like having a pub/sub system, where all the subscribers are inside the onUserChange function, and any functions that change user content, like updateUser and deleteUser, are doing the publishing.

3. There's still the problem of having to manually invalidate in updateUser and deleteUser. Someone might forget to invalidate the user cache when creating a new userUpdateWithModule function. So, we'll send an event from pg whenever changes happen and trigger the onUserChange function based on that event.

-- run this once
CREATE FUNCTION notify_change()
RETURNS TRIGGER AS $$
BEGIN
    PERFORM pg_notify(
        'internal_change', 
        jsonb_build_object(
            'old', row_to_json(OLD),
            'new', row_to_json(NEW),
            'table', TG_TABLE_NAME,
            'event', TG_OP
        )::text
    );
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

-- for all the tables you do caching for.
CREATE TRIGGER on_change
AFTER INSERT OR UPDATE OR DELETE ON public."USERS"
FOR EACH ROW EXECUTE FUNCTION notify_change();

const pool = new pg.Pool({...});
const client = await pool.connect();
async function listenToPgChanges() {
    const callbacks = {
        USERS: onUserChange,
        // add other tables
    }
    client.on('notification', function (e) {
        if (e.channel !== 'internal_change') return;
        try {
            const data = JSON.parse(e.payload);
            await callbacks[data.table](data);
        } catch (err) {
            console.error('Err in internal_change notification:', err)
        }
    })
    await client.query(`LISTEN internal_change`)
}
async function onUserChange(msg) {
    if (msg.event === 'INSERT') {
        // new possibilities.
        await sendWelcomeEmail(msg.new)
        return;
    }
    const userId = msg.old.ID;
    await Promise.all([
        getUser.invalidate(userId),
        getUserWithFewColumns.invalidate(userId),
        // add all the other, user caching functions
    ])    
}

const getUser = CacheFunc(async function (userId) {
    const result = await (...) // do some db call
    return result;
}, (userId) => `USERS:${userId}`, 3600 * 24)
const getUserWithFewColumns = CacheFunc(async function (userId) {
    const result = await (...) // do some db call
    return result;
}, (userId) => `LIGHT_USERS:${userId}`, 3600 * 24)

function updateUser(userId, data) {
    await (...) // do some db calls
    // No need to do any thing for invalidation.
}
function deleteUser(userId) {
    await (...) // do some db calls
    // No need to do any thing for invalidation.
}

This is awesome. We've tackled a bunch of issues all at once. If anyone updates the table from the database or some other backend, we can count on pg to send a notification about the changes, and our server will handle them accordingly. Plus, we've totally removed the need for invalidation in updateUser and deleteUser.

With just 3 simple steps, we've solved 90% of our problems. But there are still a few things we haven't figured out yet:

1. The code design isn't perfect yet. The invalidation call getUser.invalidate happens in the onUserChange function, but we want a proper callback using Pub/Sub.This all can be further solved using my library, i have abstract the code for anyone to use. so that you can spent time writing important business logic and not wonder the caching bugs.
2. We only have one caching wrapper to cache the function output. We need a solution for functions like getUsers that handle multiple users.
3. Changes in the output schema for caching functions. We need to check the schema before deciding if the cached output is valid, without changing the cache key prefix or manually flushing keys from Redis.

NOTE BUG: In the current setup, when we use pg_notify, it sends notifications to all the backend instances you have running. This means you'll end up invalidating the same cache multiple times because each backend instance listens to the event separately. It might be okay to invalidate the cache several times, but if you're using this for something like sendWelcomeEmail, it could be a problem. Don't worry, my solution will fix all these issues for you.

Here's the solution I came up with in the end:

1. https://jsr.io/@panth977/tools - This lets us set up Pub/Sub for calling internal functions.
2. https://jsr.io/@panth977/functions - This helps us define functions with zod input/output schemas and wrappers. It makes our code more consistent and easier to read.
3. https://jsr.io/@panth977/cache - An abstract wrapper with a bunch of hooks for single object caching, multiple object caching, collection caching, and more. (You'll need a cache that supports key → value and key → hash (field → value))
4. https://jsr.io/@panth977/cache-redis - A Redis adapter for the /cache package.

Let's tackle the rest step by step.

1. Let's use Pub/Sub from /tools (we'll be using zod as the schema validation library).

const zUSER = z.object({
    ID: z.number(),
    NAME: z.string(),
    // ...
})
const zEvent z.enum(['INSERT', 'UPDATE', 'DELETE'])
function zChanges(schema) {
    return z.object({ event: zEvent, old: schema.nullable(), new: schema.nullable() });
}
const TablesChanges = {
    // from @panth977/tools
    USERS: TOOLS.CreatePubsub(zChanges(zUSER))
}

async function onPgChanges() {
    client.on('notification', function (msg) {
        if (msg.channel !== 'internal_change') return;
        try {
            const data = JSON.parse(msg.payload);
            await TablesChanges[data.table].publish(data);
        } catch (err) {
            console.error('Err in internal_change notification:', err)
        }
    })
    await client.query(`LISTEN internal_change`)
}

TablesChanges.USERS.subscribe((msg) => msg.old && getUser.invalidate(msg.old.ID));
const getUser = CacheFunc(function (userId) {
    const result = await (...) // do some db call
    return result;
}, (userId) => `USERS:${userId}`, 3600 * 24);
TablesChanges.USERS.subscribe((msg) => msg.old && getUserWithFewColumns.invalidate(msg.old.ID));
const getUserWithFewColumns = CacheFunc(async function (userId) {
    const result = await (...) // do some db call
    return result;
}, (userId) => `LIGHT_USERS:${userId}`, 3600 * 24);

Perfect, this lets us directly subscribe to our changes without needing a middle function like onUserChange.

2. Let's use /cache. It already has the code abstracted with CacheFunc for Single Object and Multiple Object outputs, as well as Single Collection (and even Multiple Collection for those tricky situations).

// from @panth977/cache
const cache = new CACHE.CacheController({
    // from @panth977/cache-redis
    client: new RedisCacheClient(redis.createClient({...}), { delayInMs: 10, label: 'Redis' }),
    separator: ':',
    defaultExpiry: 3600 * 24,
    prefix: 'Dev',
    allowed: { '*': true },
    log: true,
});
TablesChanges.USERS.subscribe((msg) => msg.old && getUser.getHooks(msg.old.ID).del());
const getUser = CACHE.Wrap((userId) => new CACHE.HOOKS.SingleObject({
    cache: cache
        .addPrefix('USERS')
        .addPrefix(JSON.stringify({ userId }))
        .setDefaultExp(3600 * 24),
}), function ($, userId) {
    const result = await (...) // do some db call
    return result;
})
getUser(1001)
getUser(1002)

this might seem a bit much, but keep in mind CACHE.HOOKS has other hooks like:

const getUsers = CACHE.Wrap((userIds) => new CACHE.HOOKS.MultipleObject({
    cache: cache
        .addPrefix('USERS')
        .setDefaultExp(3600 * 24),
    ids: userIds,
}), function ($, userIds) {
    userIds = $.notFound; // only run for ids not found in cache
    // make sure you fetch all user data in single db request.
    const result = await (...) // do some db call
    // NOTE: use TOOLS.oneToOneMapping from @panth977/tools
    const dictResult = (...) // Make sure result is { [userId: number]: UserObject }
    return dictResult;
})
// lets say user:1001 & user:1003 exists;
getUsers([1001, 1002]) // { 1001: USER_DATA }
// --> will set cache for 1001
getUsers([1001, 1003]) // { 1001: USER_DATA, 1003: USER_DATA }
// --> will get 1001 from cache & set cache for 1003

// ------------ // ------------ //
const getUserDevices = CACHE.Wrap((userId, deviceIds) => new CACHE.HOOKS.SingleCollection({
    cache: cache
        .addPrefix('USER_DEVICES')
        .addPrefix(JSON.stringify({ userId }))
        .setDefaultExp(3600 * 24),
    subIds: deviceIds ? deviceIds : '*',
}), function ($, userId, deviceIds) {
    deviceIds = $.notFound === '*' ? undefined : $.notFound;
    const ignoreDeviceIds = $.found;
    // make sure you fetch all user devices data in single db request. 
    // if deviceIds = undefined, then fetch all the device
    // and ignore ones already found from cache.
    const result = await (...) // do some db call
    // NOTE: use TOOLS.oneToOneMapping from @panth977/tools
    const dictResult = (...) // Make sure result is { [deviceId: string]: DeviceObject }
    return dictResult;
})
// lets say user:1001 has device:OZ001, device:OZ002 & device:OZ003
getUserDevices(1001, ['OZ001', 'OZ005']) // { 'OZ001': DEVICE_DATA }
// will set cache for (1001 -> OZ001)
getUserDevices(1001) // { 'OZ001': DEVICE_DATA, 'OZ002': DEVICE_DATA, 'OZ003': DEVICE_DATA }
// will set cache for (1001 -> OZ002) & set cache for (1001 -> OZ003) 
getUserDevices(1001, ['OZ003']) // { 'OZ003': DEVICE_DATA }
// will get (1001 -> OZ003) from cache
getUserDevices(1001) // { 'OZ001': DEVICE_DATA, 'OZ002': DEVICE_DATA, 'OZ003': DEVICE_DATA }
// will get fully from cache.
getUserDevices(1001, ['OZ004']) // {}
// will figure form cache.

These APIs will help us cache almost all our cases.

3. Let's use /functions to make our code design more consistent. This will also make sure your cache has the right schema.

const getUsers = FUNCTIONS.AsyncFunction.build({
    input: z.number().array(),
    output: z.record(z.number(), zUSER),
    wrappers: (_params) => [
        CACHE.Wrapper({
            _params,
            getHook: ({ input }) => new CACHE.HOOKS.MultipleObject({
                cache: cache.addPrefix('USERS'),
                ids: input,
            }),
            updateInput: ({ info }) => info.notFound,
            useHook(hook) {
                // NOTE: [useHook] function gets executed once 
                // for this case it is when [getUsers] is being build.
                TablesChanges.USERS.subscribe(
                    (msg) => msg.old 
                        && hook({ input: [msg.old.ID] }).del()
                );
            },
        }),
    ],
    func({ input }) {
        const result = await (...) // do some db call
        const dictResult = (...) // Make sure result is { [userId: number]: UserObject }
        return dictResult;
    }
})

This takes care of all our code issues. We'll deal with the pg_notify problem next.

Now, let's tackle the last problem: listening to database changes.

-- drop the earlier function & trigger
DROP TRIGGER IF EXISTS on_change ON public."USERS";
DROP FUNCTION IF EXISTS notify_change();

-- we will store all the changes to a table
CREATE TABLE public."INTERNAL_CHANGES_V2" (
    "ID" SERIAL PRIMARY KEY,
    "TXID" TEXT NOT NULL DEFAULT txid_current(),
    "TABLE" VARCHAR(255) NOT NULL,
    "EVENT" VARCHAR(255) NOT NULL,
    "OLD" JSONB NOT NULL,
    "NEW" JSONB NOT NULL,
    "CREATED_AT" TIMESTAMP NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_internal_changes_txid_table_created_at
ON public."INTERNAL_CHANGES_V2" ("TXID", "TABLE", "EVENT", "IDENTITY", "CREATED_AT");

-- we will maintain a table about which all changes have already been observed by a subscriber.
CREATE TABLE public."INTERNAL_CHANGES_LISTENED_V2" (
    "INTERNAL_CHANGES_ID" INTEGER NOT NULL REFERENCES public."INTERNAL_CHANGES_V2"("ID") ON DELETE CASCADE,
    "LISTENER_ID" VARCHAR(255),
    PRIMARY KEY ("INTERNAL_CHANGES_ID", "LISTENER_ID") 
);
CREATE INDEX idx_internal_changes_listened
ON public."INTERNAL_CHANGES_LISTENED_V2" ("INTERNAL_CHANGES_ID", "LISTENER_ID");

-- we will delete all the changes that past 20 mins
CREATE OR REPLACE FUNCTION delete_rows_in_internal_changes()
RETURNS trigger
AS $$
BEGIN
    DELETE FROM public."INTERNAL_CHANGES_V2"
    WHERE "CREATED_AT" < NOW() - INTERVAL '20 min';
    RETURN NULL;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER trigger_delete_rows_internal_changes
AFTER INSERT ON public."INTERNAL_CHANGES_V2"
FOR EACH ROW EXECUTE FUNCTION delete_rows_in_internal_changes();

-- we will delete cleanups, can be avoided by CASCADE DELETE
CREATE FUNCTION delete_rows_in_internal_listened()
RETURNS trigger
AS $$
BEGIN
    DELETE FROM public."INTERNAL_CHANGES_LISTENED_V2"
    WHERE "INTERNAL_CHANGES_ID" = OLD."ID";
    RETURN NULL;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER trigger_delete_rows_internal_listened
AFTER DELETE ON public."INTERNAL_CHANGES_V2"
FOR EACH ROW EXECUTE FUNCTION delete_rows_in_internal_listened();

-- on any changes added to the table notify all the clients
CREATE FUNCTION on_internal_change_added()
RETURNS TRIGGER AS $$
BEGIN
  PERFORM pg_notify('internal_change', '1');
  RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER on_change_added
AFTER INSERT ON public."INTERNAL_CHANGES_V2"
FOR EACH ROW EXECUTE FUNCTION on_internal_change_added();

-- create on change generic function.
CREATE FUNCTION insert_changes()
RETURNS TRIGGER AS $$
BEGIN
    INSERT INTO public."INTERNAL_CHANGES_V2"("TABLE", "EVENT", "OLD", "NEW")
    SELECT TG_TABLE_NAME, TG_OP, row_to_json(OLD), row_to_json(NEW);
    RETURN NULL;
END;
$$ LANGUAGE plpgsql;

-- add the trigger to all your desired table.
CREATE TRIGGER on_change
AFTER INSERT OR UPDATE OR DELETE ON public."USERS"
FOR EACH ROW EXECUTE FUNCTION insert_changes();

function tableChangeSchema<S extends z.ZodRawShape>(
  schema: z.ZodObject<S>
) {
  return z.object({
    context: z.instanceof(FUNCTIONS.DefaultContext),
    event: z.enum(['INSERT', 'UPDATE', 'DELETE']),
    old: schema.nullable(),
    new: schema.nullable(),
  });
}
export const TablesChanges = {
  USERS: tableChangeSchema(zUSER)
  // add other tables...
}
const fetchLimit = 50;
const restartSec = 300;
const listenTimeout = 1000 * 30;
// this query will fetch all the new records form "INTERNAL_CHANGES_V2" 
// and add its observation entry in "INTERNAL_CHANGES_LISTENED_V2"
// it will also ignore all the rows added in transaction, but not yet commited. 
const query = `
WITH
SELECTED_CHANGES AS (
  SELECT "ID", "TABLE", "EVENT", "OLD", "NEW"
  FROM public."INTERNAL_CHANGES_V2" C
  WHERE 
    "TXID" NOT IN (
      SELECT backend_xmin::text
      FROM pg_stat_activity
      WHERE state IN ('active', 'idle in transaction') AND backend_xmin::text IS NOT NULL
    ) 
    AND NOT EXISTS (
      SELECT 1 
      FROM public."INTERNAL_CHANGES_LISTENED_V2" 
      WHERE "INTERNAL_CHANGES_ID" = "ID" AND "LISTENER_ID" = {{{SqlStr listenId}}}
    )
    AND "TABLE" IN (${Object.keys(InternalChanges).map(x => `'{x}'`).join(', ')})
  ORDER BY "ID" ASC
  LIMIT ${fetchLimit}
),
ADD_LISTENER AS (
  INSERT INTO public."INTERNAL_CHANGES_LISTENED_V2" ("INTERNAL_CHANGES_ID", "LISTENER_ID")
  SELECT "ID", ?
  FROM SELECTED_CHANGES
)
SELECT "TABLE", "EVENT", "OLD", "NEW" FROM SELECTED_CHANGES;
`;
// this function will constantly observe to pg changes.
export async function pgListenInternalChanges(): Promise<void> {
  console.log('LISTEN PG INTERNAL CHANGES');
  let lastResult: undefined | null | number | void = 1;
  while (typeof lastResult === 'number') {
    // if last observation was 0 rows
    // we will wait for any changes to happen
    if (!lastResult) await listenNewEntry();
    // we will create a [context]
    await FUNCTIONS.DefaultContext.Builder.forTask('Pg-Internal-Changes', async function (context, done) {
      // check & publish the changes.
      lastResult = await publish(context).catch(console.error);
      done();
    });
  }
  console.log('exited publishInternalChanges dew to some error, will start after a while');
  await new Promise((resolve) => setTimeout(resolve, 1000 * restartSec));
  return pgListenInternalChanges();
}
const pool = new pg.Pool({...});
const client = await pool.connect();
async function listenNewEntry() {
  // a promise that resolves on 1 msg received successfully!
  const newEntry = new Promise<any>(async (resolve, reject) => {
    try {
      client.on('notification', resolve);
      client.on('error', reject);
      // start listning!
      await client.query('LISTEN internal_change');
    } catch (err) {
      reject(err);
    }
  });
  newEntry.then(() => console.log('Listen a new Change!'));
  // fallback of some timeout, if no msg was reveived.
  const timeout = TOOLS.delay(listenTimeout);
  // check which one resolves first.
  await Promise.race([newEntry, timeout]);
  // stop listning!
  client.query('UNLISTEN internal_change').catch(console.error);
}
async function publish(context: FUNCTIONS.DefaultContext) {
  // run the query.
  const result = await pool.query(query, CONFIG.env.LISTEN_PG_INTERNAL_CHANGES);
  const jobs: Promise<void>[] = [];
  for (const row of result) {
    // publish the change.
    const promise = TablesChanges[row.TABLE as keyof typeof TablesChanges].publish({
      event: row.EVENT,
      old: row.OLD, 
      new: row.NEW,
      context: context,
    });
    jobs.push(promise);
  }
  // wait for all jobs to complete.
  await Promise.all(jobs);
  // return changes observed.
  return result.length;
}

That's it!

Now you know how we clear the cache.

How does your team invalidate a cache when the write comes from a service that doesn't know the cache exists?

I lead the software team at Oizom, an air-quality monitoring company (the platform is Envizom). This is how the heatmap backend was rebuilt there, including the bottleneck I didn't expect: PNG encoding.

How the old version worked

We used the IDW (inverse distance weighting) algorithm with wind speed and direction to figure out the value for each spot.

1. We set up a heatmap config in our database.
2. A cron job ran every hour.
3. We grabbed all the configs.
4. For each config, we had boundaries, known devices, limits, and color codes.
5. We pre-computed a grid of small 1km x 1km boxes and stored it in the CDN.
6. We used the center of each box in the pre-computed grid as the unknown value.
7. We knew each gas parameter for all devices, plus real-time geolocation, wind speeds, and directions.
8. Using IDW with wind effects, we calculated the value for each box in the grid.
9. Colors were added to the grid based on user-defined limits.
10. A Base64 PNG image was created and stored in a time-series database.
11. This image was served to the frontend when requested.

This was written in Python and used up a ton of resources (4GB RAM, 2 CPUs, 1-2 instances depending on the load), and it still wasn't enough. Each image took 10-20 seconds to process. It just wasn't fast. Back then, heatmap was a brand new feature, and we only had 3 configs, so it clearly wasn't scalable. Plus, on closer look, the wind effects weren't even used correctly.

A weekend proof of concept in the browser

When I found out about this, I suggested generating the heatmap on the frontend. My boss thought the idea was nuts. He didn't believe it was even possible. My argument was simple: today, there are tools like Canva and Figma, and many others that let you edit photos and videos right in the browser. There are even proper 2D games and simulations being played in the browser, and some resume websites are like 3D games. Technology has improved, and coding patterns and styles have changed a lot in the past 2 years, so I thought it was doable. He was convinced and decided to hire an intern to do the R&D. Typical corporate.

This really bugged me because I've always wanted to dive into image generation projects, and this seemed like my golden opportunity. So, I spent a weekend putting together a POC for a heatmap in the browser. It worked.

https://youtu.be/2wsYBaBtCY4

This was just the start. I still needed to add wind effects, map (long, lat) to (x, y) coordinates, include user-defined boundaries (from a GeoJson polygon), and finally, add gradient colors.

Boss saw it, boss liked it, and everyone was happy.

He still had a good point. We don't want to do this on the frontend because our frontend runs on old phones, corporate laptops, and sometimes even TVs—in other words, places without much computing power. So, maybe we can set up a proxy server to handle all the heatmaps and generate them on the fly.

Node.js, then Go, then the real bottleneck

So, I went ahead and set up a Node server and built the whole thing. It worked great. Turns out, when you use gradient colors, there's not much difference between a 150px resolution and a 1000px resolution, except for having sharper edges.

Test case: each request with 100px resolution, 10 known points with 40 images. Each load ran for 5 seconds.

In Node.js the results were already so much better than anything we had before. After this, I decided that to make it even faster, we should use a lower-level language. We also wanted to ensure the language is easy for other developers to understand. So, I chose GoLang. Go was faster and throttled less.

Something still felt off; it shouldn't take this long. After adding time logs for each function, I discovered the bottleneck was Base64 PNG encoding. Each image took 1-3ms, which was 80% of the time. Switching to JPEG boosted performance dramatically.

The only downside to JPEG is that it cannot create transparent images, as it lacks an alpha channel. However, we have a straightforward solution: let the backend handle all the computational tasks, while the frontend focuses on one task—masking the image and selecting only the required portions. This can be accomplished in the frontend using JavaScript, within 10-30ms. If the client can render GeoJSON, maps, and images, it can also perform image masking.

This approach also offers an advantage: the difference between a 150px and a 1000px image is that the 1000px image has sharper edges. To achieve sharper edges for a 150px image, we generate an image with an additional 2px around the boundary and crop the necessary section in the frontend. This results in sharper edges.

What the new pipeline does

1. The frontend creates a config from backend APIs and stores it in the database.
2. The frontend requests images for a config, gas parameters, and given time bounds.
3. The backend fetches all the resources and sets up a payload to send to the GoLang heatmap server.
4. In the GoLang server: It takes all the boundary points in (long, lat), maps them to (x, y) within 0-1, creates a grid of boxes based on resolution, and finds the ones within the polygon.
5. It converts all the known points/devices' locations from (long, lat) to (x, y) based on grid transformation.
6. This grid context is used to precompute the weight of each known point on each target pixel.
7. Based on the given colors, we create a gradient of values ranging from 0-256 integers.
8. We set up the encoder and precompute the (x, y) to (pixel position) in the data image array.
9. For each snapshot of values, we quickly compute the grid's value using precomputed weights.
10. We magnify each value to bring it between 0-256.
11. We fill the RGB values in the data image array.
12. We use the encoder for the base64 image.
13. These images are sent back to the core server.
14. The core server sends them back to the client.
15. The frontend masks the required portion using geo boundaries. Done!

Before and after

Images are generated on the fly, so we can now create past images too. We use fewer resources, which makes everything cheaper overall. Plus, it's finally scalable and versatile.

1. Heatmaps were only available after they were created → Now we can get heatmaps for any time range.
2. For a fixed 1-hour average → You can completely customize it.
3. 5 seconds per image → Now it's 30 ms for 20 images.
4. Python → GoLang.
5. Complex scheduler-based solution → Simple request/response-based solution.
6. Fixed options → Custom options for resolution, distance power, wind power, and wind effect.

Finally:

Of course, the data used to create this heatmap is fake and random. But this is how it would look.

The fix I didn't see coming wasn't the language or the algorithm. It was a time log on every function, which showed the image encoder taking 80% of the time. What was the last bottleneck you found that way, somewhere you weren't looking?

I'm Panth, and I lead the software team at Oizom, where our devices send tens of thousands of air-quality readings to the cloud every minute. This post is how that logging works in our Node.js and Express backends, with the code. Hosted logging services can collect all of this, but none of them group logs by request out of the box, and running a separate service per request is not an option.

The problem

A client makes a request and the API answers with a 500. To debug it you would normally try to recreate the bug, probably by logging in to the frontend as that user. The faster route is to keep the logs of every request that answered 500, keyed by its requestId, and read them.

The idea

1. Every time a request comes in, generate a UUID and include it in the response headers. Whenever anything fails, the API sends a requestId that the client can show, in a sandbox view or similar.
2. Create a context object (with requestId as context.id) that includes a method context.log, which works like console.log.
3. When creating the context, ensure there is an implementation of context.log where you can access context.id. This allows you to group the logs by requestId effectively.

Solution

My libraries @panth977/functions and @panth977/routes do this out of the box.

Create Context

// Create Context like following
app.get('/v1/user/:userId', function (req, res) {
    const id = null; // either set id, to string or leave it null for auto generate UUID
    DefaultContext.Builder.forTask(id, async function (context, done) {
        res.on("finish", done);
        context.log('My task started!');
        const user = await getUser({context, input: req.params}); // pass context to all the functions, for access to context.log
        res.send(user);
        context.log('My task completed!');
    });
});
// OR
// use @panth977/routes api to skip the process of creating functions!
export const GET_USER = ROUTES.Http.build([WebsiteAuthValidator], 'get', '/v2/user/{userId}', {
    request: ROUTES.z.HttpRequest({path: getUser.input}),
    response: ROUTES.z.HttpResponse({body: getUser.output}),
    async func({context, path}) {
        context.log('My route started!');
        const user = await getUser({context, input: path}); // pass context to all the functions, for access to context.log
        context.log('My route completed!');
        return {body: user};
    }
})

Make every function take context

That way every function can use context.log.

const getUser = FUNCTIONS.AsyncFunction.build({
    input: z.object({userId: z.coerce.number()}),
    output: z.object({name: z.string(), ...}).nullable(),
    async func({context, input}) {
        context.log('Getting user', input);
        const [user] = pg.query(`SELECT * FROM users WHERE id = ? LIMIT 1`, [input.userId]);
        context.log('Found user', user);
        return user ?? null;
    }
});

Capture context.log

Every call to context.log runs your subscribed callback.

const unsub = DefaultContext.onLog(function (context, args) {
    console.log(context.id, ...args);
    // send the log to socket 
    // store to database
    // write to a file!
});
// incase you like to unsubscribe!
unsub();

By default, context.log doesn't write to the console at all, so add console.log in the callback if you want it.

How we use it at Oizom

At Oizom our devices send calibrated air-quality data to the cloud, tens of thousands of entries every minute, and clients read their live data through our APIs. With that many requests, debugging anything in production by reading a shared log stream was close to impossible.

What I wanted

1. Keep logs only for failed requests, so successful ones don't clutter the logs database.
2. A way to list the request IDs made by user from the website, user from third-party APIs, device from the device itself, and system-{type} for any internal system calls like cron jobs, reports, alerts, etc.
3. The full logs for any one request ID.

What I built

1. Define a log payload

    type log = {
        ids: string[];
        msg: string;
        ts: number;
        line: number;
    }
   

   Many logs belong to the same ID, so why can one log belong to many IDs? Most logs do have a single ID, context.id. Multiple IDs are there for filtering: the same log line can be found by request, by user, by organisation or by login session. More on that below.

2. A secure socket app with three channels:

   1. /send_log - every server sends its logs through this channel.
   2. /stream_all_logs - streams all logs in real time as the servers send them.
   3. /stream_logs - streams only the logs that match its room (one room per id).

3. A service that listens to /stream_all_logs and stores every log. For us that is Cassandra, self-hosted in a container, with a ttl of a few weeks.

    CREATE TABLE logs.logs_v2 (
      id text,
      ids set<text>,
      ts bigint,
      msg text,
      line bigint,
      PRIMARY KEY ((id), ts)
    ) WITH CLUSTERING ORDER BY (ts ASC) AND ...
   

4. A service with an HTTP GET endpoint /fetch_logs that reads logs by id from Cassandra, ordered, with limit/offset (Cassandra has token-based pagination built in).

5. A small frontend with two modes: [ real-time , database ]

   1. A text box for the id and a submit button.
   2. Depending on the mode, it connects to the socket /stream_logs or fetches from /fetch_logs, with a "fetch next" button for more.
   3. Each log has an expand button that shows all its ids, which link to the other ids that share the log.

6. That is the logging system. The core is in the backend. Every time a context is created:

   1. It gets a logs state.
   2. Every context.log entry goes into that state.
   3. On an error, or when something needs to be kept, we set .importance = true.
   4. On dispose, if .importance === true, all the logs are streamed to /send_log.

    type Logs = {
        logs: { ids: string[]; msg: string; ts: number; line: number }[];
        important: boolean;
    };
    const LogsStateKey = FUNCTIONS.DefaultContextState.CreateKey<Logs>({
        label: 'Logs',
        scope: 'global',
    });
    const stringifyLogs = LOGS.createStringifyLogger({ forEachLine: false });
    // exposing this function for set the importance level.
    // this determines wether logs should be sent to socket or not!
    export function needToSendLogsToSocket(context: Pick<FUNCTIONS.Context, 'useState'>) {
        const state = context.useState(LogsStateKey).get();
        if (!state) return;
        state.important = true;
    }
    // exposing this function for more versatile use. (link different ids to same log)
    export function logToSocket(
        context: Pick<FUNCTIONS.Context, 'useState'>,
        { ids, line, ts }: { ids: string[]; ts?: number; line?: number },
        ...args: any[]
    ) {
        const state = context.useState(LogsStateKey).get();
        if (!state) return;
        state.logs.push({
            ids: ids,
            msg: stringifyLogs(null, args),
            ts: ts ?? Date.now(),
            line: line ?? state.logs.length,
        });
    }
    // set a state on context
    FUNCTIONS.DefaultContext.onCreate(function (context) {
        context.useState(LogsStateKey).set({ logs: [], important: false });
    });
    // store all the logs in the state
    FUNCTIONS.DefaultContext.onLog(function (context, args) {
        // every context.log will be mapped to 
        // ids=[context.id] 
        // so single id for all the context.log 
        logToSocket(context, { ids: [context.id] }, ...args);
    });
    // at end, check if it is important, and stream them to socket.
    FUNCTIONS.DefaultContext.onDispose(async function (context) {
        try {
            const state = context.useState(LogsStateKey).get();
            if (!state.important) return;
            for (const log of state.logs) socket.send(JSON.stringify(log));
        } catch (err) {
            console.error('Socket Log Error:', err);
        }
    });

7. Whenever a user logs in from the website, a third party, a device or a system-{type}, the login session gets its own UUID in the auth token.

8. Every request goes through an auth check. That is where I work out what kind of login it is and which categories the request belongs to.

    const WebsiteAuthValidator = ROUTES.Middleware.build({
        request: ROUTES.z.MiddlewareRequest({
            headers: z.object({}).passthrough(),
        }),
        response: ROUTES.z.MiddlewareResponse({
            headers: z.object({}).passthrough(),
        }),
        async func({ context, headers }) {
            // frontend wants to store logs!
            if (headers['x-store-logs'] === 'true') needToSendLogsToSocket(context);
            const decoded = ...; // decode the token from headers.
            if (decoded.mode === 'client') throw createHttpError.Unauthorized('This is not a client Api, You only have access to client APIs');
            const ids = [context.id];
            if (decoded.mode === 'system') {
                if (decoded.id) ids.push(`Core:User:System:${decoded.id}`);
            } else {
                ids.push(`Core:User:${decoded.userId}`);
                ids.push(`Core:User:Org:${decoded.orgId}`);
                if (decoded.id) ids.push(`Core:User:${decoded.userId}:${decoded.id}`);
            }
            const { req } = context.useState(ExpressStateKey).get();
            logToSocket(context, { ids }, req.method, req.path);
            context.useState(DecodedStateKey).set(decoded);
            if (headers['x-store-logs'] === 'true') return { headers: { 'x-request-id': context.id } };
            return {};
        },
    })
   

   This way, when you get logs for id = Core:User:2141, you see a list of all the API calls made by this user. Each log includes a list of other relevant ids, and one of them will be context.id. That way, you can get the requestId, and by fetching the logs of that requestId, you get the complete logs for that specific API call.

9. When a request fails, we keep its logs.

   const onError: ExpressHandler.onError = function ({ context, error: err }) {
     if (createHttpError.isHttpError(err)) {
       return { 
         body: err.message,
         headers: { 'x-request-id': context.id, ...(error.headers ?? {}) },
         status: err.statusCode
       };
     }
     if (context) {
       const { req } = context.useState(ExpressHandler.ExpressStateKey).get();
       // log the request, could be helpfull for recreating error, in debugging.
       context.log(
         'Req::',
         JSON.stringify({ method: req.method, path: req.params, query: req.query, body: req.body, headers: req.headers }, null, '\t'),
       );
       context.log('Err While Req:', err);
       console.error('500 Error at:', context.id);
       // set the importance.
       needToSendLogsToSocket(context);
     } else {
       // nothing we can do further!
       console.error('Err:', err);
     }
     return { 
       body: 'Something went wrong!', 
       headers: { 'x-request-id': context.id }, 
       statusCode: 500 
     };
   };
   const v2 = ExpressHandler.serve({ bundle: ExposedRoutes, onError });
   app.use(v2);
   

The log viewer

I built the viewer with AI's help: I designed the system and let AI write the boring parts. Here it is:

Later I needed a list of every email sent to one address, or every SMS sent to one phone number. All it took was a new template for id. Each of those logs carries the requestId, so you can see which request sent the email.

Each log shows 4 things:

1. A toggle button to see other linked ids
2. The line number of the log
3. The ts in local time
4. The msg, which is the log message.

The extra ids are links, so you can jump from a user to a request to an email while reading.

If you build something like this, build it yourself rather than buying it: the ids you will want to filter by are specific to your system.

How do you find the logs of one failed request in production today?

The constraint: the declaration is not a server. @panth977/routes only describes routes. Serving them is a separate handler, @panth977/routes-express for Express, or one you write for another framework.

I'm Panth, and I lead the software team at Oizom. @panth977/routes is one of my open-source libraries on JSR. This post walks through it with code: middleware, an HTTP route, an SSE route, and the calls that turn a folder of routes into docs and a client.

What one declaration gives you

Because every route is already declared with types, the docs don't need a second source. The OpenAPI JSON can serve a docs UI, and the same JSON generates the frontend's code. Nothing is written twice.

Routes are built the same way as @panth977/functions, so you keep its wrappers, context, static, name and namespace, build, and input/output schemas. The functions post covers those.

Middleware

A middleware declares its own request schema and its security rule. Both end up in the docs of every route that uses it.

const DecodedStateKey = FUNCTIONS.DefaultContextState.CreateKey<ReturnType<typeof decodeToken>>({ label: 'Decoded',scope: 'global' });
const authorize = ROUTES.Middleware.build({
  security: {
    JwtAuth: {
      type: "apiKey", // this rules will be used for documentation of routes
      description: "Authorize ** /users/login **.",
      name: "x-auth-token",
      in: "header",
    },
  },
  request: ROUTES.z.MiddlewareRequest({ // this schema will be used to address route request schema, in documentation
    headers: z.object({ "x-auth-token": z.string() }).passthrough(),
  }),
  response: ROUTES.z.MiddlewareResponse({}),
  async func({context, headers}) {
    const token =
      headers["x-auth-token"] ??
      headers["authorized"] ??
      headers["x-token"];
    const decoded = await decodeToken(token);
    if (!decoded) throw createHttpError.Unauthorized("Token not found / Token got Expired / Invalid Token!");
    // set state to context for others to make use of it!
    context.useState(DecodedStateKey).set(decoded);
    return {};
  },
});

This one looks for the token in three headers (x-auth-token, then authorized, then x-token), decodes it, and throws a 401 if that fails. On success it stores the decoded token in the request's context under DecodedStateKey. Any route that lists authorize can read it back from there, which is what the next example does.

An HTTP route

const getProfile = ROUTES.Http.build([authorize], "get", "/profile", {
  description: `Get user profile.`,
  request: ROUTES.z.HttpRequest({
      query: z.object({
        getFriends: z.enum(['true', 'false']).default('false'),
      })
  }),
  response: ROUTES.z.HttpResponse({
    body: z.object({
      id: z.number(),
      username: z.string(),
      address: z.string().nullable(),
      noOfProjects: z.number().int().gt(0),
    }),
  }),
  static: {
    query: pg.query(`Select * from users where id = ? LIMIT 1`),
  },
  async func({context, query, build}) {
    const { userId } = context.useState(DecodedStateKey).get();
    const result = await build.query.execute([userId]);
    if (!result.rows.length) throw createHttpError.NotFound("Given user was not found in db, could be because someone has deleted the user.");
    if (query.getFriends === 'true') {...}
    return result.rows[0];
  },
});

The route reads the user id that authorize stored, runs a query declared once in static, and returns a row. The getFriends query parameter arrives as the string 'true' or 'false', with 'false' as the default, because that is what the schema says.

Compared with @panth977/functions, input becomes request and output becomes response. The request's path, query, headers and body arrive in func as named arguments. The response has headers and body.

ROUTES.z.HttpRequest and ROUTES.z.HttpResponse stand in for z.object so the types stay exact. You can still pass your own. For the docs, a route also takes description, tags, summary, security, reqMediaTypes and resMediaTypes.

A Server-Sent Events route

An SSE route is an async generator. This one yields the request's logs 30 at a time until there are none left.

export const getLogs = ROUTES.Sse.build([systemAuthorized], "get", "/logs/{requestId}", {
  request: ROUTES.z.SseRequest({
    path: z.object({
      requestId: z.string(),
    }),
  }),
  response: ROUTES.z.SseResponse(
    z
      .object({ id: z.number(), msg: z.string(), ts: z.date() })
      .array()
      .transform((x) => JSON.stringify(x))
  ),
  async *func({context, path: { requestId }}) {
    let logs = [];
    let offset = 0;
    do {
      const result = await pg.query(`SELECT * FROM logs where id = ? ORDER BY ts OFFSET ? LIMIT 30`, [requestId, offset]);
      logs = result.rows
      offset += logs.length;
      yield logs;
    } while (logs.length);
  },
});

Chaining middleware instead

If you prefer app.get('/profile', …) with middleware chained in front, ROUTES.Endpoint gives you that shape.

const initMiddleware = ROUTES.Middleware.build(...);
const userAuthMiddleware = ROUTES.Middleware.build(...);
const thirdPartyAuthMiddleware = ROUTES.Middleware.build(...);

const EndpointFactory = ROUTES.Endpoint.build().addMiddleware(initMiddleware);
const endpoints = {
  userAuthorized: EndpointFactory.addMiddleware(userAuthMiddleware),
  thirdPartyAuthorized: EndpointFactory.addMiddleware(thirdPartyAuthMiddleware),
}

const getProfile = endpoints.userAuthorized.HTTP('get', '/profile', ...);
const getProfile3rdParty = endpoints.thirdPartyAuthorized.get('/api/{userId}/profile', ...);

From a folder of routes to OpenAPI

ROUTES.getEndpointsFromBundle takes a module's exports and keeps only the endpoints (HTTP and SSE), dropping helpers and middleware. The detailed types are lost at this step, which is what you want when you are serving or filtering every route at once.

// --- routes/index.ts ---
export * from './m.ts'; // middleware
export * from './a.ts'; // route1, route2, and a helper variable
export * from './b.ts'; // route3, route4 (SSE), and a helper function
// --- server.ts ---
import * as routes_ from './routes/index.ts';
const routes = ROUTES.getEndpointsFromBundle({bundle: routes_}); // strong type will be lost
console.log(routes) // {route1: ..., route2: ..., route3: ..., route4: ...}

Then the OpenAPI JSON, with routes tagged internal-apis left out:

const routes = ROUTES.getEndpointsFromBundle({
    bundle: await import('./routes/index.ts'),
    excludeTags: ['internal-apis'],
});
const OpenApiJson = ROUTES.getRouteDocJson(routes, {
    info: {
        title: 'My Apis',
        version: '0.0.1',
    }
});

Serving a route

ROUTES.execute runs one endpoint through lifecycle hooks you supply. This is the seam where a framework plugs in. The example uses Express; routes-express is the full reference handler.

type MyFrameworkOpt = { req: Express.Request, res: Express.Response };
const MyEndpointExecutionLifeCycle: ROUTES.LifeCycle<MyFrameworkOpt> = {
    init(...) {...},
    onStatusChange({...}) {...},
    onExecution({...}) {...},
    onResponse({...}) {...},
    onComplete({...}) {...},
}
app.get('/getUser', (req, res) => {
    FUNCTIONS.DefaultContext.Builder.forTask(null, async (context, done) => {
        ROUTES
            .execute({
                context,
                build: getUserEndpoint,
                lc: MyEndpointExecutionLifeCycle,
                opt: { req, res },
            })
            .finally(done);
    })
})

The TypeScript client

The last output comes from the OpenAPI JSON. ROUTES.CodeGen turns it into frontend code with a typed call for every API. Here it is served from an endpoint:

app.get('/code-gen', (_, res) => {
  FUNCTIONS.DefaultContext.Builder.forTask('CodeGen', function (context, done) {
    res.on('finish', done);
    try {
      const { code } = ROUTES.CodeGen.genTsCode({ context, input: { json, options: {} } });
      res.setHeader('content-type', 'text/x.typescript').send(code);
    } catch (err) {
      console.log(err);
      res.status(500).send('Something went wrong!');
    }
  });
});

The schema is written once, and the handler, the docs and the client are all read from it.

The rest of the packages are at jsr.io/@panth977.

Where do your API docs come from today: the route code itself, or a spec someone keeps in step by hand?

The constraint: TypeScript types are for development. They don't validate a function's inputs or outputs once the code runs, so the checking has to be a layer you add on purpose. In @panth977/functions that layer is FUNCTIONS.WRAPPERS.SafeParse.

I'm Panth, and I lead the software team at Oizom. @panth977/functions is one of my open-source libraries on JSR. Once a function is declared this way, three other things come with the same shape: wrappers for caching and throttling without touching the function's code, static variables kept out of the global scope, and a context that travels through the call tree with logs, cleanup and state.

One declaration

const getUser = FUNCTIONS.AsyncFunction.build({
    namespace: 'PgQuery', // optional
    name: 'getUser', // optional
    input: z.object({ userId: z.number().int().gt(0) }),
    output: z.object({ name: z.string(), ... }).optional(),
    static: { // add static variables
        db: new Pg.Pool(...),
    },
    buildContext: YourCustomContextBuilder, // optional
    wrappers: (_params) => [FUNCTIONS.WRAPPERS.SafeParse({ _params })], // optional
    async func({ context, input, build }) {
        context.log('getting user', input.userId);
        const result = await build.db.query(`SELECT * FROM users WHERE id = ? LIMIT 1`, [input.userId]);
        const user = result.rows[0];
        if (!user) return undefined;
        context.log('found user', user.name);
        return user;
    }
});
...
getUser.setNamespace('PgQuery');
getUser.setName('getUser');
getUser.getRef(); // 'PgQuery.getUser'
...
const user = await getUser({ context, input: { userId } });

What each part is for:

There is a single argument, so every argument is named, and a call reads getUser({ context, input: { userId } }) at every call site. name and namespace don't have to be known when the function is built; they can be set later in the code, which the last section uses. FUNCTIONS builds four kinds: SyncFunction, AsyncFunction, SyncGenerator and AsyncGenerator. The schemas give you types while you write the code. SafeParse is what makes them hold at runtime.

Wrappers

Caching, throttling and logging are usually higher-order functions wrapped around the original. Here they are a list on the declaration, and they run in the order written:

const fib = FUNCTIONS.SyncFunction.build({
    input: z.number(),
    output: z.number(),
    wrappers: (_params) => [
        // your custom wrapper
        function ({ context, input, func, build }) {
            console.log('start w1', input);
            const result = func({ context, input, build });
            console.log('end w1', input);
            return result;
        },
        function ({ context, input, func, build }) {
            console.log('start w2', input);
            const result = func({ context, input, build });
            console.log('end w2', input);
            return result;
        },
    ],
    func({ context, input, build }) {
        console.log('fib', input);
        if (input <= 2) return 1;
        return build({ context, input: input - 1 }) + build({ context, input: input - 2 });
    },
});
fib({ context, input: 3 });
// CONSOLE
start w1 3
start w2 3
fib 3
start w1 2
start w2 2
fib 2
end w2 2
end w1 2
start w1 1
start w2 1
fib 1
end w2 1
end w1 1
end w2 3
end w1 3

The recursive calls go through build, so each one passes through both wrappers again. That is the point of having them on the declaration: fib itself has no idea that anything is wrapped around it, and you can add or remove a wrapper without editing func. Caching is a wrapper, the runtime check is a wrapper, and so is your own logging. The ready-made ones:

• FUNCTIONS.WRAPPERS.SafeParse parses input and output.
• FUNCTIONS.WRAPPERS.MemoData remembers results in AsyncFunction and SyncFunction.
• FUNCTIONS.WRAPPERS.Debug logs input and output, and checks whether the call took longer than the throttle time you set.
• FUNCTIONS.WRAPPERS.CloneData copies inputs and outputs (and yields and next arguments in a generator).

Static variables

Anything you put in static stays attached to the function, outside the global scope. A query builder, a connection pool, a constant the function needs: it lives with the function, the implementation reads it through build, and code outside can still reach it as a property:

const getUser = FUNCTIONS.AsyncFunction.build({
    ...
    static: {
        query: new QueryBuilder(`SELECT * FROM users WHERE ID = {{{SqlNum userId}}}`),
    },
    ...
    async func({ context, input, build }) {
        const rows = await pg.query(build.query.compile(input))
        ...
    }
});
getUser.query; // QueryBuilder

Context: logs, cleanup and state down the call tree

The context you pass in goes to buildContext, which makes a new one for the call. A context is identified by its id, and several contexts can share an id. I create one context with a unique id per task and pass it to every call under it.

await FUNCTIONS.DefaultContext.Builder.forTask('myId', function (context, done) {
    // use parent context as reference
    const context1 = FUNCTIONS.DefaultContext.Builder.fromParent(context, 'ref1');
    const context2 = FUNCTIONS.DefaultContext.Builder.fromParent(context, 'ref2');
    console.log(context.id) // 'myId'
    console.log(context.id === context2.id) // true
    console.log(context === context2) // false
    console.log(context.path) // []
    console.log(context2.path) // ['ref1', 'ref2']
    done();
}); // create a new context

A context made from a parent keeps the task's id and gets a new object, and its path records the references it came down through. Every call in one task logs and cleans up under the same id.

context.log. console.log can't tell you which task or thread wrote a line. context.log can. Nothing reaches the console until you add a logger:

FUNCTIONS.DefaultContext.onLog(function (context, args) {
    console.log(context.id, ...args);
});
// CONSOLE:
(0 sec) myId Hi World

context.dispose. Cleanup that shouldn't slow the main task goes into a dispose callback. Whoever created the context disposes of it:

await FUNCTIONS.DefaultContext.Builder.forTask('myId', function (context, done) {
    // just before this function is called, FUNCTIONS.DefaultContext.onCreate is called
    console.log('my implementation');
    context.dispose(() => TOOLS.delay(2_000).then(() => console.log('MyDispose1')));
    context.dispose(() => TOOLS.delay(2_000).then(() => console.log('MyDispose2')));
    TOOLS.delay(2_000).then(done);
    // after the done call, FUNCTIONS.DefaultContext.onDispose is called
});
console.log('task completed!');

The order: every onCreate hook is awaited (Promise.allSettled), then the task runs. When it calls done, every context.dispose callback is awaited, and then every onDispose hook.

context.useState. Local state belongs to one context instance. Global state is shared by all parent and child contexts. In the example, the child sees the parent's global value but not its local one, and when the child changes the global value, the parent sees the change:

const GlobalStateKey = FUNCTIONS.DefaultContextState.CreateKey<string>({
    label: 'MyGlobalState',
    scope: 'global',
});
const LocalStateKey = FUNCTIONS.DefaultContextState.CreateKey<number>({
    label: 'MyLocalState',
    scope: 'local',
});
FUNCTIONS.DefaultContext.Builder.forTask('myId', function (context, done) {
    context.useState(GlobalStateKey).set('s1');
    context.useState(LocalStateKey).set(1);
    const context1 = FUNCTIONS.DefaultContext.Builder.fromParent(context, 'ref1');
    const context2 = FUNCTIONS.DefaultContext.Builder.fromParent(context, 'ref2');
    console.log(context2.useState(GlobalStateKey).get()) // 's1'
    console.log(context2.useState(LocalStateKey).get()) // undefined
    context2.useState(LocalStateKey).set(2);
    context2.useState(GlobalStateKey).set('s2');
    console.log(context.useState(GlobalStateKey).get()) // 's2'
    console.log(context.useState(LocalStateKey).get()) // 1
    console.log(context2.useState(LocalStateKey).get()) // 2
    done();
});

Names from the folder

name and namespace can be set after the build. I export every function in a directory from one index.ts and set them in a loop: the directory is the namespace, the export name is the name.

// --- pg/users.ts ---
export const getUser = FUNCTIONS.AsyncFunction.build({ ... });
export const getSubUsers = FUNCTIONS.AsyncFunction.build({ ... });
// --- pg/devices.ts ---
export const getDevices = FUNCTIONS.AsyncFunction.build({ ... });
export const updateDevice = FUNCTIONS.AsyncFunction.build({ ... });
// --- pg/_index.ts ---
export * from './users.ts';
export * from './devices.ts';
// --- pg/index.ts ---
export * as Pg from './_index.ts';
import * as _Pg_ from './_index.ts';
const Pg = (_Pg_) as Record<string, any>;
for (const name in Pg) {
    Pg[name].setNamespace('Pg');
    Pg[name].setName(name);
}

Routes in @panth977/routes are built the same way, which the routing post covers.

Do you parse a function's input and output at runtime, or only at the edges of the system, where requests come in?

Rate limit function invocation

Sometimes you just want to make sure you don't go over the max number of calls to a service. To keep things in check and avoid hitting the rate limit, try using TOOLS.CreateParallelTaskManager.

const serviceToSq = TOOLS.CreateParallelTaskManager({
    maxParallelExecution: 2,
    async implementation(x) {
        console.log('start', x);
        await TOOLS.delay(x * 1000);
        console.log('end', x);
        return x ** 2;
    },
});
serviceToSq(5);
serviceToSq(2);
serviceToSq(4);
serviceToSq(10);
serviceToSq(9);
/** CONSOLE **/
(00 sec) start 5
(00 sec) start 2
(02 sec) end 2
(02 sec) start 4
(05 sec) end 5
(05 sec) start 10
(06 sec) end 4
(06 sec) start 9
(15 sec) end 10
(15 sec) end 9

Batch multiple Invocations

Maybe you've got a speedy API call for the backend that handles multiple operations in one go, but you don't want to complicate your development process. In that case, try using TOOLS.CreateBatchProcessor.

const RedisMethods = {
    get: TOOLS.CreateBatchProcessor<string, string | null>({
        delayInMs: 0,
        async implementation(args) {
            console.log('<0>', args);
            return await redis.mget(args);
        },
    }),
    getAfter10ms: TOOLS.CreateBatchProcessor<string, string | null>({
        delayInMs: 10,
        async implementation(args) {
            console.log('<10>', args);
            return await redis.mget(args);
        },
    }),

}
RedisMethods.get('key1');
RedisMethods.get('key2');
delay(12).then(() => RedisMethods.get('key3'));
delay(12).then(() => RedisMethods.get('key4'));
RedisMethods.getAfter10ms('key5');
RedisMethods.getAfter10ms('key6');
delay(5).then(() => RedisMethods.getAfter10ms('key7'));
delay(12).then(() => RedisMethods.getAfter10ms('key8'));

/** CONSOLE **/
(00 ms) <0> ['key1', 'key2']
(10 ms) <0> ['key3', 'key4']
(10 ms) <10> ['key5', 'key6', 'key7']
(22 ms) <10> ['key8']

Internal PubSub

We've all seen code where we pile on tons of features after creating a user. Sure, you could set up a PubSub, but that takes time. Unless you're handling something big, you can just do it on the API server. No need to set up a separate background worker service and build a whole PubSub messaging system. Try using TOOLS.CreatePubsub to keep your code nice and clear.

const userChanges = TOOLS.CreatePubsub(z.object({
    event: z.enum(['CREATE', 'UPDATE', 'DELETE']),
    data: z.object({
        id: z.number().int(),
        email: z.string().email(),
        name: z.string().max(225),
        ...
    }),
}));
function createUser(userData) {
    const rows = pg.query(`INSERT INTO users VALUES (...) RETURNING *`);
    await userChanges.publish({ event: 'CREATE', data: rows[0] });
}
userChanges.subscribe(function ({ event, data }) {
    if (event === 'CREATE') {
        // send welcome email
    } else if (event === 'DELETE') {
        // send good bye email
    }
});
userChanges.subscribe(function () {
    if (event === 'CREATE') {
        // setup records
    } else if (event === 'DELETE') {
        // archive records
    }
});

Data Structures One To One Mapping

You often deal with databases that return rows, and you probably want a Dictionary structure to make life easier. We all know how annoying it is to use .find on rows every single time. Give TOOLS.oneToOneMapping a shot. We also have TOOLS.oneToManyMapping and TOOLS.oneToOneToOneMapping.

function getUsers(userIds) {
    const rows = pg.query(`SELECT * FROM users WHERE id IN (...)`)
    const users = TOOLS.oneToOneMapping({ rows, keyPath: "id" });
    return users;
}
const users = getUsers([userId1, userId2]);
const user1 = users[userId1];
const user2 = users[userId2];

Data Structures One To Many Mapping

function getUsersFriends(userIds) {
    const rows = pg.query(`
SELECT f.* 
FROM users u 
JOIN friends f ON f.userId = u.id
WHERE u.id IN (...)
    `)
    const friends = TOOLS.oneToManyMapping({ rows, keyPath: "id" });
    return friends;
}
const friends = getUsersFriends([userId1, userId2]);
const friendsOfUser1 = friends[userId1];
const friendsOfUser2 = friends[userId2];

Data Structures One To One To Many Mapping

I honestly doubt you'll ever need this monster! But hey, I've got it here just in case. (NOTE: If you find yourself needing this in more than one place, you really should rethink your code architecture.)

Encode Connection String

Have you checked out env.PG_HOST, env.PG_PORT, env.PG_DB, env.PG_USER, env.PG_PASS? What if we could simplify all these different environment variables into a single env.PG_CONNECTION_STRING? That would be awesome! Here are the TOOLS.encodeConnectionString encoder and TOOLS.decodeConnectionString decoder for that.

MY_CONNECTION_STRING: TOOLS.encodeConnectionString({
    protocol: 'myprotocol',
    user: 'myuser',
    pass: 'mypass',
    host: 'myhost',
    port: 1234, // optional
    db: 'mydb', // optional
    params: { //optional
      log: 'someval',
      ssl: 'true',
      max: '20',
    },
})
// myprotocol://myuser:mypass@myhost:1234/mydb?log=someval&ssl=true&max=20
MY_CONNECTION_OPTIONS: TOOLS.decodeConnectionString(MY_CONNECTION_STRING)

No worries, user, pass, and host are all encoded with encodeURIComponent and decoded with decodeURIComponent. Plus, params are handled with URLSearchParams, making life easier for everyone!

Sort

A quick sort function for an array to sort based on some deeply nested field, just the way you need it.

const myArray: {
    id: number;
    age: number; // int values only
    stats: {
        score: number;
        goals: number;
        ...
    };
    ...
}[] = [...];
const sortedOnScore = TOOLS.sortList({
    rows: myArray,
    mode: 'ASC',
    keyPath: 'stats.score',
});
const sortedOnAge = TOOLS.sortList({
    rows: myArray,
    mode: 'ASC',
    keyPath: 'age',
    algorithm: 'bucket',
});

Dynamic destructuring

You might need to grab a bunch of data from a SQL select query and turn it into a JSON for the frontend. Here's how you can do it easily using TOOLS.destructure

const rows = await pg.query(`
    SELECT 
        v.id            "videoId",
        c.email         "creator.email",
        c.name          "creator.name",
        c.handle        "creator.handle",
        v.url           "video.url",
        v.share_id      "video.share_id",
        v.description   "meta.description"
        ...
    FROM ...
    JOIN ...
    WHERE ...
`);
const videosJson = TOOLS.destructure({ rows });
type Video = {
    videoId: number;
    creator: {
        email: string;
        name: string;
        handle: string;
    };
    video: {
        url: string; 
        share_id: string;
        description: string;
    };
    ...
}
return videosJson as Video[];

All Promise Stats

Let's say you're running a bunch of promises at the same time, and this is for a worker API. You probably want to track some stats to see which jobs were successful and which ones weren't. Here's how you can do it using TOOLS.SettleAllPromise

const jobs: Record<string, Promise<any>> = {};
for (const cluster of clusters) {
    jobs[cluster.id] = runCluster(cluster);
}
const stats = TOOLS.SettleAllPromise(jobs);
stats.errors; // = Record<string, errors>
stats.results; // = Record<string, errors>
stats.stats; // {total, status, error?, errorRate?}

Delay

You've got a super long sync process happening, and if you didn't know, JavaScript runs on a single thread. Unless you're using workers, you're stuck with just one core, and none of your incoming requests will get processed because that one thread is busy with a sync task.

If you're familiar with the JavaScript event loop, you know there's a way out. Just figure out your checkpoints where you can let the main thread handle other tasks and then come back to finish the rest of the process. TOOLS.delay comes to rescue.

function heavyTask1() {
    //
}
function heavyTask2() {
    //
}
function heavyTask3() {
    //
}
function typicalImplementation() {
    heavyTask1();
    heavyTask2();
    heavyTask3();
}
async function betterImplementation() {
    heavyTask1();
    // complete all the other task from task que & event que
    await TOOLS.delay();
    heavyTask2();
    // complete all the other task from task que & event que
    // if not already 250ms completed, wait of it to complete!
    await TOOLS.delay(250);
    heavyTask3();
}

Check out https://jsr.io/@panth977 to make your project clearly coded instead of that clean coding nonsense...

Who in their right mind would think that console.log({} > []) yields true? It's absurd because JavaScript decides to convert the object into a string before comparing. And that's why console.log({} > ['x']) results in false. This kind of behavior is baffling and highlights the quirks of JavaScript.

The concept of the prototype and the this keyword can be absolutely mind-blowing 🤯! Imagine being able to change how an existing class method works (super handy when you don't have lifecycle hooks) or eliminating the need to create new extended classes. JavaScript is packed with powerful features for developers to explore, but only if you have the right knowledge 💪. Instead, you often see people relying on 💉 AI. AI isn't bad, but without a solid grasp of your concepts, AI can be like muscles made of oil—yikes!

class SomeClassFromLib {
    someMethod(...args) {
        return someVal;
    }
}
// this lib should have given you event for when property1 is called,
// but seems like [prototype] can fix this.
const original_someMethod = SomeClassFromLib.prototype.someMethod;
SomeClassFromLib.prototype.someMethod = function (...args) {
    console.log('someMethod was called.');
    return original_someMethod(...args);
}
// and now any instance of [SomeClassFromLib] class will use 
// this newly provided function.

You have eval, which is super powerful. You can do all sorts of cool stuff without needing to create a compiler. How about creating a context, passing it to all the function calls, collecting all the console logs in that context, and maybe storing them somewhere?

// all the functions exported from all the files.
const myFunc = {
  add: function (context, { a, b }) {
    const calc = a + b;
    console.log(calc);
    return calc;
  },
  diff: function (context, { a, b }) {
    const calc = Math.abs(a - b);
    console.log(calc);
    return calc;
  },
  calcAll: function (context, input) {
    return {
      add: myFunc.add(context, input),
      diff: myFunc.diff(context, input),
    };
  },
};
// wrap all the functions to magical functions that make sure
// every thing is [console.log] invokes [context.log]
for (const key in myFunc) {
  myFunc[key] = eval(`(function (context) {
    const originalConsole = console;
    $: {
      const console = {...originalConsole, log: (...args) => context.log(...args)}
      const func = ${myFunc[key].toString()};
      return func(...arguments);
    }
  })`)
}

const context = {
  logs: [],
  log(...args) {
    this.logs.push({ ts: Date.now(), log: JSON.stringify(args) });
  }
}
myFunc.add(context, {a: 2, b: 3})
myFunc.diff(context, {a: 2, b: 3})
myFunc.calcAll(context, {a: 5, b: 10})
console.log(context.logs);
/* Console */
[
  { ts: ..., log: '[5]' },
  { ts: ..., log: '[1]' },
  { ts: ..., log: '[15]' },
  { ts: ..., log: '[5]' }
]

Until I figure out a way to pass down context without code, you might need a compiler 😜. (Of course, you're now stuck with Error.stack or any outside references made.)

Then there's the event loop! You can totally use this to make space for other sync programs to run. How useful is that?! 🎉 This technique can help you share your main thread with other tasks throughout your app.

function syncLoop(x) {
    let s = 0;
    for (let i = 1; i <= x; i++) {
        s += i;
    }
    console.log(s);
    return s;
}
syncLoop(1_000_000_000); // 1st
syncLoop(100); // 2ed

async function asyncLoop(x) {
  let s = 0;
  while (x > 0) {
    if (x < 10_000) {
      for (; x > 0; x--) {
        s += x;
      }
    } else {
      const n = x - 10_000;
      for (; x > n; x--) {
        s += x;
      }
      await new Promise((res) => res());
    }
  }
  console.log(s);
  return s;
}
asyncLoop(1_000_000_000); // 5th
asyncLoop(600); // 1st
asyncLoop(9_999); // 2ed
asyncLoop(10_001); // 4th
asyncLoop(200); // 3rd
asyncLoop(1_000_000_000); // 6th

But there's an obvious downside to offering so many features. There's no "one way to code," and the human mind is creative, so it will find every possible way to use all these features in their scrap yard projects. When the beginners or interns continue working on the project, you end up discovering world-class misuse of features and classic coding pattern errors. This flexibility sometimes leads to such chaos.

Callback hell, indeed. JavaScript supports async/await syntax, yet for some inexplicable reason, and than they use .then/.catch/callbacks and keeps writing code in the callback function body. Why? Nothing good can come from this! And what's with all the .catch or try/catch everywhere? Seriously, unless you're actually handling the response, stop catching everything—it's not mandatory. And naturally, instead of using the async function syntax, they opt for new Promise(...) syntax, as if that simplifies anything.

function badImplementation() {
    return new Promise((resolve, reject) => {
        someOldLibWithCb((data, err) => {
            if (err) {
                reject(err);
                return;
            }
            myDbCall(data).then((result) => {
                const output = processResult(result);
                resolve(output);
            }).catch(reject);
        });
    });
}
// OR
async function goodImplementation() {
    // Convert any old cb syntax to promise using [new Promise]
    const data = await new Promise((resolve, reject) => {
        try { // put a try/catch just to be extra safe.
            someOldLibWithCb((data, err) => {
                if (err) reject(err);
                else resolve(data);
            });
        } catch (err) {
            reject(err);
        }
    });
    // keep awaiting, and don't catching unless you have a reason to.
    const result = await myDbCall(data);
    const output = processResult(result);
    return output;
}

We can chat all day about clean code and whether it's better or not, but honestly, I think it's kind of pointless. This isn't about clean code; it's about Clear Code. To achieve that, you need to decide how your code should look and which features are most important. Otherwise, we're no better than the next desk intern! 😰

In my future blogs, I will share a JS design system I have created, specifically for the backend! This system can greatly enhance your debugging and coding time, while ensuring your code is type-safe and free from runtime type errors. Let's build a community focused on code design! We'll make JS code design so great that AI will be all over it!

function badImplementation() {
    return new Promise((resolve, reject) => {
        someOldLibWithCb((data, err) => {
            if (err) {
                reject(err);
                return;
            }
            // Implementation...
        });
    });
}
// OR
async function goodImplementation() {
    // Convert any old cb syntax to promise using [new Promise]
    const data = await new Promise((resolve, reject) => {
        try { // put a try/catch just to be extra safe.
            someOldLibWithCb((data, err) => {
                if (err) reject(err);
                else resolve(data);
            });
        } catch (err) {
            reject(err);
        }
    });
    // Implementation...
}

Keep your function implementation in the main body instead of inside a callback.

async function badImplementation() {
    return getUser(userId).then((user) => {
        if (!user) throw new Error('User not found');
    });
}
// OR
async function goodImplementation() {
    const user = await getUser(userId);
    if (!user) throw new Error("User not found");
    // ...
}

Make sure to avoid catching errors unless you have a plan to handle them.

async function badImplementation() {
    const user = getUser(userId).catch((err) => {console.error(err)});
    if (!user) throw "User not found";
    // ...
}
// OR
async function goodImplementation() {
    const user = await getUser(userId);
    // ...
}

Always throw an HTTP response-supported error for better API responses.

class HttpError extends Error {
  constructor(status, message) {
    this.status = status;
    super(message);
  }
}
app.get('/profile', async (req, res) => {
    try {
        const profile = await getProfile(req.query.username);
        res.send(profile);
    } catch (err) 
        if (err instanceof HttpError) {
            res.status(err.code).send(err.message);
            return;
        }
        console.error(err);
        res.status(500).send("Something Went Wrong!");
    }
});
async function getProfile(username) {
    const user = await getUser(username);
    if (!user) throw new HttpError(400, "No such user found!");
    // implementation
    return profile;
}

Consider sending a request ID with all incoming requests and storing all console.log entries in a database for debugging purposes.

function myLog(requestId, ...args) {
    try {
        await db.logs.insert(requestId, Date.now(), JSON.stringify(args));
    } catch (err) {
        cosnole.error(err);
    }
}
app.get('/profile', async (req, res) => {
    const requestId = randomUuid();
    const profile = await getProfile(requestId, req.query.username);
    res.setHeaders('x-req-id', requestId);
    res.send(profile);
});
async function getProfile(requestId, username) {
    const user = await getUser(username);
    if (!user) throw new HttpError(400, "No such user found!");
    // implementation
    myLog(requestId, "Found Profile", username, profile);
    return profile;
}

Add schema validation for all incoming requests and outgoing responses, as they cannot be fully trusted and might have a schema version mismatch.

app.get('/user', async (req, res) => {
    if (!isUsername(req.query.username)) {
        res.status(400).send('Invalid username!')
        return;
    }
    const user = await getUser(req.query.username);
    if (!isUsername(user.username)) {
        res.status(500).send('Invalid username!')
        return;
    }
    if (!isEmail(user.email)) {
        res.status(500).send('Invalid email!')
        return;
    }
    if (!isName(user.name)) {
        res.status(500).send('Invalid name!')
        return;
    }
    res.send(user);
});
app.post('/user', async (req, res) => {
   if (!isUsername(req.body.username)) {
        res.status(400).send('Invalid username!')
        return;
    }
    if (!isEmail(req.body.email)) {
        res.status(400).send('Invalid email!')
        return;
    }
    if (!isName(req.body.name)) {
        res.status(400).send('Invalid name!')
        return;
    }
    const profile = await getProfile(requestId, req.query.username);
    res.send('User added successfully!');
});

Cache your frequently accessed data. There's no need to cache everything!

async function getDevices(userId) {
    // maybe cache this!
    return await db.user(userId).devices.get();
}
async function getDevicesData(userId, gte, lte) {
    const devices = await getDevices(userId)
    return await db.devices(devices).data(gte, lte);
}
async function getDevicesStatus(userId) {
    const devices = await getDevices(userId)
    return await db.devices(devices).data.lastEntry().wasInLast15Min();
}
async function getDevicesExpiry(userId) {
    const devices = await getDevices(userId)
    // ...
}
// Make sure to invalidate the cache when changes are made!

Create an internal Pub/Sub system; it can be useful. Also, consider creating a batching function and a max parallel allowance function, similar to Pub/Sub. These can make your work easier.

class PubSub {
  cbs = new Set()
  subscribe(cb) {
      this.cbs.add(cb);
      return () => {
          this.cbs.delete(cb);
      }
  }
  publish(...args) {
      for (const cb of this.cbs) {
          setTimeout(() => cb(...args));
      }
  }
}

const userChanges = new PubSub();
userChanges.subscribe((username) => {
  // Invalidate user cache
})
async function getUser(username) {
    // cache this!
}
async function updateUser(username) {
    // Implementation
    userChanges.publish(username)
}
async function deleteUser(username) {
    // Implementation
    userChanges.publish(username)
}

You might want a way to generate OpenAPI JSON, and there are various UIs available to assist with presentation.

const generatedOpenAPIjson = {...};
app.use('/docs/', swagger.ui, swagger.serve(generatedOpenAPIjson));

Ensure all your GET APIs are fast, and make your POST, PATCH, DELETE, and PUT operations transactional. This means that either everything succeeds or everything fails.

I think I've found a way to handle these problems. Check out https://jsr.io/@panth977 for how to use it. It's the best approach for building a backend easily.

This package lets you do many things easily.

1. https://jsr.io/@panth977/tools offers useful tools, such as creating internal Pub/Subs and higher-order functions that let you batch multiple function calls. It also includes a higher-order function to set a limit on the maximum number of parallel invocations allowed, as well as one-to-one and one-to-many mappings, among others.
2. https://jsr.io/@panth977/functions offers a builder to create type-safe functions using Zod, along with static variables and local wrappers. You can build synchronous functions, asynchronous functions, synchronous generators, and asynchronous generators. It also includes helpful wrapper implementations for easier use.
3. https://jsr.io/@panth977/cache offers a standard cache class, giving you full control over key prefixes, and the ability to read, write, and remove keys and hash fields. It also includes standard hooks to cache a response, whether it's an Object, MultipleObject, Collection, or MultipleCollection. This should cover all your caching needs.
4. https://jsr.io/@panth977/routes provides builders to create your endpoints. It supports Middleware, HTTP, and SSE techniques. If you correctly use the Zod schema with the routes, it will automatically generate OpenAPI JSON for free.
5. https://jsr.io/@panth977/cache-redis provides a Redis implementation for /cache.
6. https://jsr.io/@panth977/routes-express provides an Express implementation for /routes.

Stay tuned for more insights.