Skaldborn devlog

The colleague who hallucinates (and the fences that keep it honest)

Mon, 01 Jun 2026 00:00:00 GMT

One morning in April I almost shipped a fix for a problem that didn’t exist.

I’d sent three agents off to audit the project’s own governance — read the rules, run the checks, report what’s broken. Routine housekeeping. One of them came back certain: four of my rule files were missing a required field, and a check that was supposed to guard exactly that was failing in CI. It wrote this up cleanly. Specific files. Specific field. A failing gate. It even proposed the four commits to fix it.

None of it was true. The four files had the field. The gate passed — twenty-seven of twenty-seven. The agent had run an audit, looked at green, and reported red. Not maliciously. It just produced the shape of a finding that sounded right, the way the tavern-keeper in my first post produced a dead brother who never existed.

I’d already started consolidating its report into a patch plan when something felt off and I went and looked myself.

The thing that saved me was a human glance

Sit with that, because it’s the whole problem.

The only reason a fabricated bug report didn’t turn into four real commits against my codebase is that I happened to be paying attention that morning. I read the agent’s confident paragraph, felt a flicker of wait, didn’t I just fix that?, and pulled the file up by hand.

That is not a control. That’s luck wearing the costume of a control. It doesn’t scale past the days I’m sharp, it doesn’t survive me being tired, and it definitely doesn’t survive the thing I actually want — more agents, doing more work, while I’m not watching every keystroke. If my safety net is “Carlos notices,” then every hour the machine works unsupervised is an hour with no net at all.

So the question stopped being how do I get the AI to stop hallucinating — you don’t; that’s like asking the dice to stop rolling — and became something I could actually build against:

You don’t make it honest. You make it checkable.

This is the same move I made when I deleted two months of working code. Back then the lesson was about contracts: an aspirational contract is one you wrote down and hope is true; a mechanical contract is one that breaks the build when it’s violated. I’d been trusting aspirational contracts and calling it engineering.

The collaborator is the same story, one level up. I can’t audit the AI’s intentions — it doesn’t really have any, and even if it did I can’t see them. What I can do is build the workspace so that the dangerous things it might do are either impossible or self-reporting. Not “please be careful.” Structurally unable to be careless without it showing.

An honest colleague you can’t verify is worth less than a careless one you can. Skaldborn is built by an AI I assume is careless, inside fences that make carelessness bounce off.

Here’s what those fences actually are.

What an AI colleague actually is on this project

There isn’t an AI on Skaldborn. There are several, and they’re deliberately not the same actor.

The work is split into roles, each with its own context and its own blinders:

The architect plans and writes the decision records. It never implements.
The writer implements exactly one scoped slice. It never plans.
The reviewer reads the writer’s diff against the contract. It never writes code.
The verifier runs the scenario and checks pass/fail. It doesn’t reason about implementation.
The steward keeps the trackers and manifests honest.
The orchestrator runs the loop — and is forbidden, in writing, from reading source code at all.

That last one sounds insane until you see why. The orchestrator’s job is to decide what gets dispatched to whom. The moment it starts reading source to “just check something,” it’s no longer scoping work — it’s doing the work, in the one context that’s supposed to stay clean enough to catch when a piece of work is mis-scoped. So its rule file says, flatly: contracts, logs, health output, packet history — never source. If it can’t figure out a dispatch from those, the correct output isn’t a guess. It’s the sentence “contracts are underspecified,” and a stop.

I launch these as separate sessions with a one-line script. The role isn’t a vibe — it’s bound at launch, down to which model runs it:

$ start-claude.sh writer-query-api --role writer -p "Implement the bounded slice in the packet ..."

Remote-control name: sklaude-writer-query-api-04417
Claude Code session 'skaldborn-writer-query-api-04417' started.
Role: writer (model: claude-opus-4-8)
Expected handoff: state/coordination/writer-query-api-handoff.md

start-claude.sh spins up a detached session, pins the model to the role, drops a sentinel file that says this slug is running, and tells me exactly which file will appear when the session is done. Completion isn’t “the agent said it finished.” Completion is a handoff file exists on disk. The session can crash, run out of context, or wander off — the only thing that counts as done is the artifact landing where the script said it would.

None of this matters, though, without the part that makes the writer’s blinders real.

The fence around the writer

A writer agent is handed a packet — a scoped work order. The packet names the component it’s allowed to touch and, critically, lists allowed_source_roots and forbidden_source_roots: the exact directories it may read and write, and the ones it may not.

I used to hope the agent would respect that. Now a hook enforces it, and the agent’s own good intentions are irrelevant.

Before any writer is dispatched, the orchestrator stages a tiny scope file naming the component contract. The first time the writer tries to touch a file, a PreToolUse hook — enforce-component-boundary.sh — wakes up, reads that contract’s allowed roots, and checks the path. If the file is outside the fence, the tool call doesn’t happen:

{
  "hookSpecificOutput": {
    "hookEventName": "PreToolUse",
    "permissionDecision": "deny",
    "permissionDecisionReason": "Component boundary violation: <path> is outside allowed roots for component '<name>'."
  }
}

That’s not a log line written after the fact. It’s a refusal before the edit. A writer scoped to the query API physically cannot open a simulation file, no matter how reasonable its plan to do so sounds. And there’s a meaner case the hook handles too: if a writer is dispatched with no scope staged at all — the orchestrator forgot — it doesn’t shrug and allow everything. It denies the very first file operation, with a reason that says exactly what went wrong: component scope missing — the orchestrator must stage scope before calling the writer.

This is the whole philosophy in one design choice: fail closed. A misconfigured dispatch — the exact kind of mistake a busy orchestrator makes at 1am — doesn’t quietly grant an agent the run of the repo. It stops the agent at the door. The unsafe default is the one that can’t happen.

(People ask why I don’t just give each writer its own git worktree. I tried. Eleven stale worktrees once stranded eighteen commits, and I went looking for them like lost luggage. The boundary hook plus packet scope gives me the isolation without the graveyard. Writers commit straight to main, inside their fence.)

The fence around the investigator

The boundary hook guards writing. But my April near-miss wasn’t a writer. It was an investigator — an agent sent to look around and report. Those are the dangerous ones, because their output isn’t a diff you can review line by line. It’s a confident paragraph, and confident paragraphs are exactly what these models are best at producing whether or not they’re true.

So that incident got its own fence — and it’s my favorite one, because of what it forces.

Now, before any investigation-class agent can be dispatched, the dispatcher has to write a sidecar that says, in advance, how the finding will be proven. If it doesn’t, the hook refuses the dispatch outright — no sidecar staged at state/coordination/pending-subagent-audit.json (or its freeform sibling), no Agent call fires at all.

The audit sidecar isn’t bureaucracy. It makes the claim falsifiable by construction:

{
  "packet_type": "audit",
  "canonical_commands": ["make validate-governance-enforcement-declared"],
  "canonical_files": ["docs/governance/playbooks/..."],
  "expected_evidence_format": "raw_stdout"
}

canonical_commands is the exact command whose output is the evidence. (In this case, a check that walks every governance rule in the repo and asserts each one declares how it’s enforced — a real script, a hook, a CI target — versus being a nice paragraph nobody runs. It’s the gate the April agent claimed was failing.) expected_evidence_format: raw_stdout means the agent’s report has to quote that command’s actual output, not summarize it, not characterize it — paste it. You can’t claim a gate is red if the gate’s own stdout, sitting in your report, says green.

Think about what this would have done in April. The agent that fabricated a failing check would have been forced, before it ever ran, to declare: the proof of this finding is the verbatim output of this command. And the verbatim output was twenty-seven of twenty-seven, passing. The lie and its own disproof would have been in the same report. I’d have caught it in a glance — but now the glance is guaranteed to have something to catch on, every time, whether or not I’m sharp that morning.

There’s an escape hatch, because not every look-around is a fact-claim. “Find me the usages of this function” isn’t an audit; it’s a search. For those, the dispatcher writes a freeform sidecar with one required field — a plain-English reason — and the hook lets it through. That’s deliberate. The discipline I want isn’t “fill out forms forever.” It’s: if you’re going to assert a fact about my repo, name your proof up front, and if you’re just poking around, say so out loud. The escape hatch is visible in the audit trail too, so “freeform everything” can’t quietly become the lazy default.

How the fences come together: the loop

Day to day, the orchestrator runs a loop that’s almost boring, which is the point. Boring is what you want from the thing holding the leashes.

It reads the next unblocked packet. It stages the writer’s scope. It dispatches the writer — which can now only touch its fence. When the writer’s done, it dispatches a separate reviewer against the diff and the contract. When the reviewer approves, a verifier runs the regression profile. Only then does the packet close.

No single agent both writes the code and blesses it. No agent that’s reasoning about the system is also the one searching it and reporting facts about it. The investigator that could hallucinate can’t dispatch without declaring its proof. The writer that could overreach can’t open a file outside its lines. And the orchestrator coordinating all of it can’t read source to form opinions of its own.

Every one of those is a structural separation, not a polite request. The agents are good colleagues. I just don’t build as though they are.

What it feels like to work this way

I won’t pretend it’s frictionless. It isn’t.

Every writer dispatch needs its scope staged first. Every fact-finding mission needs its proof named first. There are mornings the boundary hook denies something I genuinely wanted, and I have to stop and ask whether the fence is wrong or my plan is — and embarrassingly often, the fence is right and I was about to let an agent reach across a boundary I’d drawn for a reason.

But here’s the trade I actually made. The friction is paid by me, up front, in small visible amounts — a scope file, a named command. The alternative cost is paid later, invisibly, in fabricated bug reports that become real commits, in an agent quietly rewriting a file three components away, in the slow rot of a codebase edited by something I trusted instead of checked. I’ve paid the second kind. It’s much more expensive, and you never see the bill until it’s overdue.

The fences don’t make the AI smarter. They make its mistakes cheap and loud instead of expensive and silent. That’s the entire deal.

What I’d tell myself in February

If you’re about to hand real write access to an agent — and more of us are, every month — here’s what I wish I’d known before the April morning:

Assume carelessness, not malice, and definitely not competence. Design for the agent that confidently reports red on green. It will happen. Plan for it instead of being surprised by it.
A human glance is not a control. If your only safety net is that you’ll notice, you have no net the moment you look away — and the whole point of an agent is to look away.
Fail closed. A misconfigured dispatch should stop the agent at the door, not silently grant it the keys. Make the unsafe default impossible, not merely discouraged.
Separate the roles that could collude. Don’t let one agent both write code and approve it, or both reason about the system and report facts about it. Cheap separations prevent expensive failures.
Make every factual claim name its own proof. “The gate is failing” is worthless. “The gate is failing; here is its verbatim output” disproves itself when it’s wrong. Force the second shape.
Pay the friction up front, on purpose. A scope file and a named command are small, visible costs. The thing they prevent is a large, invisible one.

The agents do real work on Skaldborn — most of it, by volume. I sleep fine about that, not because I trust them, but because I stopped needing to.

The companion to this one is the technical build guide: How to let an agent write your code without giving it the keys — a line-by-line walk through the launch script, the two fail-closed hooks, and the dispatch loop, close enough to the real files that you could stand up your own version of these fences over a weekend. This post is the why; that one’s the how.

If you want to follow along, subscribe via the form at the bottom of any page — one short email when the next post lands. If you want to argue — or just tell me what you’re wiring up — write to devlog@skaldborn.com.

Everything else is the boring engineering of making it true.

How to let an agent write your code without giving it the keys

Mon, 01 Jun 2026 00:00:00 GMT

This is the technical companion to The colleague who hallucinates. That post told the story — an audit agent that reported a passing gate (twenty-seven of twenty-seven) as failing and nearly turned a hallucination into four real commits — and the principle that fell out of it: you don’t make a coding agent trustworthy, you make its output checkable, and you build the workspace so the dangerous moves are either impossible or self-reporting. This one is the build guide. It walks through the actual machinery — a ~150-line launcher, two fail-closed hooks, and the dispatch loop — close enough to the real thing that you can stand up your own version.

It’s built on Claude Code (the CLI), and the specific mechanism it leans on — a hook that runs before a tool call and can veto it — is Claude Code’s PreToolUse hook. If your agent harness has an equivalent pre-execution interception point, the same shapes port over. Linux and bash are assumed. The snippets are trimmed for the page — error handling and a few escape hatches are elided — but the control flow is what runs.

You don’t have to have read the companion to use this. The launch pattern, the deny-by-JSON hook contract, the scope-sidecar trick, and the audit-sidecar discipline generalize to any setup where you’re handing real write access to something that occasionally lies with confidence.

Why this is hard

Here’s the asymmetry that makes the whole problem.

When you hand an agent a structured work order — implement this, in these files, validated by this command — you have a lot of surface to enforce against. The scope is a list of paths. The acceptance check is a command with an exit code. You can gate all of it.

When you send an agent to look around and tell you what it found, you have almost nothing. The input is a freeform prompt. The output is a paragraph. And a paragraph that says “the gate is failing” is indistinguishable, at the dispatcher’s end, from a paragraph that says “the gate is failing” when it isn’t. The model is best at producing fluent, confident prose — which is exactly the output you can least verify.

So the strategy splits in two:

For agents that write — fence the filesystem. Make it physically impossible to touch a file outside the work order, and fail closed when the work order is missing.
For agents that investigate — force every factual claim to name its own proof, up front, before the agent even runs.

Both are enforced at the same chokepoint: a hook that fires before the tool call and can refuse it. Everything below is variations on that one move.

The one mechanism everything hangs on

A Claude Code PreToolUse hook is a command that runs before a tool executes. It receives the tool call as JSON on stdin, and it can allow the call (exit 0, no output) or deny it by printing a small JSON decision and exiting 0:

deny() {
  cat <<EOF
{
  "hookSpecificOutput": {
    "hookEventName": "PreToolUse",
    "permissionDecision": "deny",
    "permissionDecisionReason": "$1"
  }
}
EOF
  exit 0
}

That’s the entire contract. Print that, and the tool call never happens; the permissionDecisionReason is fed back to the agent as the reason it was blocked. Note the convention: you exit 0 even when denying — the decision is in the JSON, not the exit code. (Exiting 2 with a message on stderr does the same thing; the JSON form is just easier to read and lets you write a precise reason.)

Every fence in this post is a hook that decides whether to call deny. Hold onto that and the rest is detail.

Piece 1 — the launcher

Before any fences matter, you need a clean way to start an agent in a known role. On Skaldborn that’s a ~150-line bash script, start-claude.sh. It does three things worth copying.

It binds the role to a model at launch. A “writer” and a “reviewer” aren’t the same actor with different instructions — they’re different sessions, on deliberately different models, so the thing that reviews the work was never the thing that wrote it:

role_to_model() {
  case "$1" in
    architect|steward|orchestrator|writer) echo "claude-opus-4-8" ;;
    reviewer|verifier)                      echo "claude-opus-4-7" ;;
    *) return 1 ;;   # unknown role → hard error, don't guess
  esac
}

It launches detached, with the role’s model and a remote-control handle:

INNER="claude --dangerously-skip-permissions --remote-control $RC_NAME"
[[ -n "$MODEL"  ]] && INNER="$INNER --model $MODEL"
[[ -n "$PROMPT" ]] && INNER="$INNER $(sq_escape "$PROMPT")"

tmux new-session -d -s "$SESSION" -e "CLAUDE_SESSION_SLUG=$SLUG" "$INNER"

Yes, --dangerously-skip-permissions. That flag is what makes the agent able to run unattended — and it’s also exactly why the hooks below have to be airtight. Skipping the interactive permission prompt doesn’t skip the hooks. The hooks are the permission system once the human is no longer in the loop. If you’re going to use that flag, the fences aren’t optional; they’re the replacement for the thing you turned off.

(sq_escape is a POSIX single-quote escaper. The prompt travels bash → tmux → sh -c → claude, and a prompt with a stray quote or newline will shred the argv chain. Wrap in '...', replace embedded ' with '\'', move on with your life.)

It makes “done” a fact on disk, not a claim. The launcher prints — and a sentinel file records — the exact path that will exist when the session finishes its work:

Role: writer (model: claude-opus-4-8)
Expected handoff: state/coordination/writer-query-api-handoff.md
Active-session sentinel: state/coordination/active-sessions/writer-query-api.json

This matters more than it looks. An agent that says “I’m done!” and an agent that crashed mid-task produce the same silence from the outside. By contract, completion is the handoff file landing at the declared path — nothing else counts. A caller can block on it without parsing any chat:

until [ -f state/coordination/writer-query-api-handoff.md ]; do sleep 60; done
echo "DONE: writer-query-api"

The sentinel (written at launch, removed by a tmux session-closed hook when the session dies) answers the other question — is it still running? Present means alive; absent means stopped. Together they give you a cheap, file-based view of a fleet of agents without a scheduler.

Piece 2 — the fence around the writer

Now the important one. A writer is dispatched with a scope: the component it may touch, expressed as a set of allowed directory roots in that component’s contract file. The boundary hook (enforce-component-boundary.sh, wired to fire before every Read, Edit, and Write) enforces it.

The first problem is mechanical and worth knowing if you build this yourself: environment variables don’t propagate to subagents, and at the moment the hook fires for a freshly spawned agent, the agent doesn’t exist yet — so you can’t key the scope on an agent id you don’t have. The fix is a single-shot file. The dispatcher drops one pending-scope.json before spawning the writer; the writer’s very first tool call claims it:

# No scope in the environment? Try to claim the pending scope file.
if [[ -z "${COMPONENT_CONTRACT:-}" ]]; then
  AGENT_ID=$(jq -r '.agent_id // ""'   <<<"$INPUT")
  AGENT_TYPE=$(jq -r '.agent_type // ""' <<<"$INPUT")
  SIDECAR="state/coordination/active-scope-$AGENT_ID.json"
  PENDING="state/coordination/pending-scope.json"

  # First tool call claims the pending scope as this agent's active scope.
  [[ ! -f "$SIDECAR" && -f "$PENDING" ]] && mv "$PENDING" "$SIDECAR"
  [[ -f "$SIDECAR" ]] && COMPONENT_CONTRACT=$(jq -r '.contract' "$SIDECAR")
fi

Then the part that makes it a fence and not a suggestion — fail closed. A writer that arrives with no scope resolved is a misconfigured dispatch (the orchestrator forgot to stage it). The wrong move is to shrug and allow everything; that silently re-opens the exact hole the fence exists to close. So:

# A writer with no resolved scope is a misconfiguration. Deny, don't default-open.
if [[ -z "${COMPONENT_CONTRACT:-}" && "$AGENT_TYPE" == writer-* ]]; then
  deny "Component scope missing for $AGENT_TYPE. The orchestrator must stage \
state/coordination/pending-scope.json before dispatching the writer."
fi

With a scope resolved, the actual check is almost boring. Read the allowed roots out of the contract; if the target path is inside one, allow; otherwise, deny:

ALLOWED_ROOTS=$(jq -r '(.source_roots + .test_roots + .shared_dependencies)[]' "$CONTRACT_PATH")

while IFS= read -r root; do
  [[ -z "$root" ]] && continue
  [[ "$root" != /* ]] && root="$PWD/$root"
  if [[ "$FILE_PATH" == "$root" || "$FILE_PATH" == "$root/"* ]]; then
    exit 0          # inside the fence — allow
  fi
done <<<"$ALLOWED_ROOTS"

COMPONENT_NAME=$(jq -r '.component // "unknown"' "$CONTRACT_PATH")
deny "Component boundary violation: $FILE_PATH is outside allowed roots for \
component '$COMPONENT_NAME'."

There’s a small universal allowlist on top of this — every agent can always read the shared contracts, the coordination directory, the build files — so the fence constrains source, not the agent’s ability to orient itself. And reads of the agent’s own component directory are always fine. But the spine is the loop above: a writer scoped to the query API cannot open a simulation file, full stop, no matter how reasonable its plan to “just check one thing” sounds.

One deliberate non-choice: I don’t isolate writers in git worktrees. I tried; eleven stale worktrees once stranded eighteen commits and I spent an afternoon recovering them like lost luggage. The boundary hook plus the scope file gives the isolation without the graveyard. Writers commit straight to the main branch, inside their fence.

Piece 3 — the fence around the investigator

The boundary hook guards writing. It does nothing for the more insidious case: an agent sent to investigate, which can’t damage a file but can hand you a confident, fabricated finding that you then act on. (That’s the failure the companion post opens with — an audit agent that reported a passing gate as failing and nearly turned a hallucination into four real commits.)

The second hook (enforce-subagent-dispatch-discipline.sh, wired to fire before the Agent tool — the one that spawns sub-agents) gates exactly the investigation-class spawns and lets everything else through:

SUBAGENT_TYPE=$(jq -r '.tool_input.subagent_type // ""' <<<"$INPUT")
case "$SUBAGENT_TYPE" in
  Explore|general-purpose) ;;   # the fact-finding agents — gated below
  *) exit 0 ;;                  # writers, reviewers, etc. — not our problem here
esac

The rule for the gated ones: you may not dispatch a fact-finder without first declaring, on disk, how its finding will be proven. Two sidecars satisfy that. An audit sidecar — for any dispatch that will assert a fact — or a freeform sidecar with a stated reason, for genuine open-ended search (“find the usages of this function”) where there’s no fact to prove yet.

AUDIT="state/coordination/pending-subagent-audit.json"
FREEFORM="state/coordination/pending-subagent-freeform.json"

if [[ -f "$AUDIT" ]]; then
  if ! err=$(validate_audit "$AUDIT"); then
    deny "audit sidecar invalid: $err"
  fi
  consume "$AUDIT"; exit 0      # claim succeeds — dispatch allowed
fi

if [[ -f "$FREEFORM" ]]; then
  reason=$(jq -r '.reason // ""' "$FREEFORM")
  [[ -n "$reason" && "$reason" != "null" ]] || deny "freeform bypass requires a non-empty reason"
  consume "$FREEFORM"; exit 0
fi

deny "audit sidecar required for Explore/general-purpose dispatch. Write \
pending-subagent-audit.json or pending-subagent-freeform.json before the call."

The whole trick is in what validate_audit insists on. An audit packet has to name the evidence, and the evidence has to be the kind you can’t fake:

validate_audit() {
  local file="$1" errors=""
  jq empty "$file" 2>/dev/null || { echo "not valid JSON"; return 1; }

  local fmt;  fmt=$(jq -r '.expected_evidence_format // ""' "$file")
  local nc;   nc=$(jq '(.canonical_commands // []) | length' "$file")
  local nf;   nf=$(jq '(.canonical_files    // []) | length' "$file")

  case "$fmt" in raw_stdout|parsed_jsonl|file_read) ;;
    *) errors+="expected_evidence_format must be raw_stdout|parsed_jsonl|file_read; " ;;
  esac

  # Must name at least one evidence source.
  (( nc == 0 && nf == 0 )) && errors+="name at least one canonical_command or canonical_file; "
  # "I read a file" must name the file; "I ran a command" must name the command.
  [[ "$fmt" == "file_read" ]]                            && (( nf == 0 )) && errors+="file_read needs canonical_files; "
  [[ "$fmt" == "raw_stdout" || "$fmt" == "parsed_jsonl" ]] && (( nc == 0 )) && errors+="$fmt needs canonical_commands; "

  [[ -n "$errors" ]] && { echo "${errors% }"; return 1; }
  return 0
}

So a real audit sidecar looks like this:

{
  "packet_type": "audit",
  "canonical_commands": ["make validate-governance-enforcement-declared"],
  "expected_evidence_format": "raw_stdout"
}

expected_evidence_format: raw_stdout is the load-bearing field. It means the agent’s report must quote that command’s actual output verbatim — not summarize it, not characterize it, paste it. You cannot claim a gate is red when the gate’s own stdout, sitting three lines down in your own report, says green. The lie and its disproof end up in the same document. The fence doesn’t stop the agent from being wrong; it stops the agent from being wrong invisibly.

consume is the audit trail. The sidecar doesn’t get deleted — it’s moved, stamped with the tool-call id that claimed it, so afterward you can match “what did the dispatcher promise” against “which spawn actually consumed it”:

consume() {
  local src="$1"
  mkdir -p state/coordination/consumed
  mv "$src" "state/coordination/consumed/${TOOL_USE_ID}-$(basename "$src")"
}

The freeform escape hatch is deliberate, and deliberately visible. The discipline I want isn’t “fill out a form forever” — it’s “if you’re going to assert a fact about the repo, name your proof; if you’re just poking around, say so out loud.” Because every freeform reason is logged to the same trail, “freeform everything” can’t quietly become the path of least resistance without leaving fingerprints.

Wiring it together

The hooks are inert until you register them. In .claude/settings.json, you attach each to the tool it guards via a matcher:

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Read|Edit|Write",
        "hooks": [
          { "type": "command", "command": ".claude/hooks/enforce-component-boundary.sh" }
        ]
      },
      {
        "matcher": "Agent",
        "hooks": [
          { "type": "command", "command": ".claude/hooks/enforce-subagent-dispatch-discipline.sh" }
        ]
      }
    ]
  }
}

Read|Edit|Write is a regex over tool names — the boundary hook fires before any file op. Agent is the spawn tool — the dispatch-discipline hook fires before any sub-agent is created. That’s the entire integration. (Skaldborn runs a few more hooks at each point — one that keeps the orchestrator out of source code, one that blocks destructive git, an audit logger — but those two are the load-bearing pair this post is about.)

The loop that drives it

The fences are passive — they only say no. Something has to do the dispatching, and on Skaldborn that’s the orchestrator: a session whose entire job is to run a phase of work by handing scoped packets to fresh agents. Its loop, stripped to the spine:

Read the next unblocked work packet.
Stage the writer’s scope: write pending-scope.json naming the component contract.
Spawn the writer. The boundary hook now confines it to that component.
When the handoff file lands, spawn a separate reviewer against the diff and the contract.
On approval, spawn a verifier to run the regression check.
Only then mark the packet done.

Two constraints fall out of the mechanics. Writers are serial — there’s one pending-scope.json at a time, so you stage, spawn, wait for the first tool call to claim it, then stage the next. (Parallel writers need a per-spawn keying scheme the single-shot file doesn’t give you.) And the orchestrator itself is forbidden from reading source — enforced by yet another boundary hook — because the moment the coordinator starts reading code to “just check,” it stops being the clean context that can notice when a packet is mis-scoped. If it can’t dispatch from contracts, logs, and packet history alone, the correct output isn’t a guess. It’s “the contract is underspecified,” and a stop.

No single agent both writes code and approves it. No agent that reasons about the system is also the one reporting facts about it. Those separations are the point, and they’re structural, not polite.

Gotchas worth knowing before you build this

A few things cost me time. Skip the lessons:

Env vars don’t reach sub-agents, and the sub-agent doesn’t exist when the parent’s spawn hook fires. This is the reason scope is passed by single-shot file instead of an environment variable. Don’t fight it — drop a file, claim it on first tool call.
exit 0 is how you deny. The decision lives in the JSON on stdout, not the exit code. An uncaught error that exits non-zero is not a deny — depending on config it can fail open. Which is why the next point matters.
Decide your failure direction on purpose. These hooks shell out to jq. If jq is missing, what happens? Skaldborn’s dispatch-discipline hook fails closed — no jq, no dispatch — with a narrow, explicit environment-variable escape hatch for the rare case you need to override. A security fence that fails open the moment a tool is missing isn’t a fence.
Per-session tmux hooks don’t fire on session close — by the time session-closed runs, the session and its attached hooks are already gone. Use a global hook that extracts the session name from the hook context. (This is the bit that cleans up the sentinel files.)
Quote-escape the prompt across the whole chain. bash → tmux → sh -c → claude will happily destroy a prompt containing a quote or newline. POSIX single-quote escaping survives it; bash-only tricks don’t, because tmux runs the inner command through sh.

How Skaldborn consumes this

None of this is a side project — it’s how the game actually gets built. The substrate work, the content pipeline, the client: most of it, by volume, is written by agents running inside these fences, dispatched by the loop above, while I’m reviewing something else or asleep. The launcher starts them in role. The boundary hook keeps each one inside its component. The dispatch-discipline hook keeps the fact-finders honest. The handoff files tell me what finished. I read diffs and verdicts, not keystrokes.

The fences don’t make the agents smarter. They make the agents’ mistakes cheap and loud instead of expensive and silent — a misconfigured dispatch stops at the door, an out-of-bounds edit never lands, a fabricated finding arrives stapled to its own disproof. That’s the whole trade, and it’s the only reason I’m comfortable handing real write access to something I’ve watched confidently report green as red.

What’s next

There’s a piece I deferred and still owe: a post-run check that reads the agent’s transcript and verifies each promised command’s output actually appears in its report — turning “detectable in review” into “mechanically impossible to fake.” The pre-dispatch sidecar forces the agent to name its proof; it doesn’t yet force the agent to quote it truthfully. That’s the next fence.

If you want to follow along, subscribe via the form at the bottom of any page — one short email when the next post lands. If you build a version of this and something breaks — or you want to argue — write to devlog@skaldborn.com. I read everything.

Skaldborn's art content pipeline

Tue, 12 May 2026 00:00:00 GMT

There is no AI judging the AI. No vision-language model rates the output. No aesthetic-classifier auto-promotes the best of a batch. Every image the Skaldborn art pipeline ships passes my review before it reaches the manifest, and the pipeline is built so only my review can promote it.

The pipeline generates pixel art with two AI backends — a local Stable Diffusion stack and PixelLab’s hosted pixel-art API — under a governance discipline borrowed from the simulation side of the engine. This post is about why we built it that way, what’s underneath, and where the receipts are.

Want to skip straight to the technical bits? Set up ComfyUI for your own content pipeline walks the install end to end on a 4 GB consumer GPU, plus the full PixelLab API integration. This post is the architectural why; that one’s the operational how.

The thirty-second version

A recipe is a JSON file that describes what to make. It declares a small set of levers (constrained to enums) and a larger set of locked fields (everything else — checkpoint, LoRA strengths, seeds, canvas size, anchor offsets). Calling the CLI with a recipe and lever values queues one job per backend into a Postgres-backed saga. The saga drives each job through a state machine: generate, stage to a content-addressed path, wait for human approval at two gates, copy the approved file into a frozen “approved” slot, and emit a manifest entry.

Every randomness source in the pipeline lives upstream of my approval. By the time an asset is in the manifest, it’s a frozen file pinned to a content-addressed path. The simulation consumes the frozen manifest. It never sees the generation pipeline.

That is the spine. The rest of this post is what’s underneath.

The stack

Everything in the pipeline is open source or publicly purchasable, so a reader who wants to rebuild a piece of it can:

.NET 9.0 — worker host, CLI, saga state machine, all pipeline C#
PostgreSQL — durable art_jobs queue (claim ordering via SELECT FOR UPDATE SKIP LOCKED)
Npgsql — Postgres driver
ComfyUI — local Stable Diffusion orchestrator; runs on a host GPU and is reachable from inside Docker via host.docker.internal:8188
PixelLab — hosted pixel-art API; eight-direction character rotations, isometric tiles, map objects
Docker + Compose — the worker runs as a long-lived compose service against a hermetic multi-stage Dockerfile
OpenTelemetry (OTLP) — metrics out of the worker; a Datadog Agent receives them when the opt-in env flag is set
System.CommandLine — CLI argument parsing
Bash — validate-recipe-coverage.sh, promote-assets.sh, process-character-animations.sh
free-tex-packer — sprite-atlas packing (queued, not yet wired)

On the local Stable Diffusion side, the production stack is concrete:

Checkpoint: aZovyaRPGArtistTools v4VAE — an SD 1.5 fine-tune by Zovya for illustrative RPG concept art (game art, tabletop, book covers). VAE baked in.
Style LoRA: NorseViking_v10 by Nontime — Norse warrior / berserker style, applied at strength_model: 0.7 / strength_clip: 0.5.
Upscaler: 4xFoolhardyRemacri by FoolhardyVEVO — a 4× ESRGAN, run after FaceDetailer to bring concept output up to 2048×3072.
Negative embedding: negative_hand-neg by Nerfgun3 — corrects bad hand anatomy without dragging the style with it.
Face detector: face_yolov8n.pt — Bingsu’s nano-scale YOLOv8 fine-tune (mAP50 0.660 across multiple face datasets), used inside FaceDetailer for the bounding box.
Segmenter: sam_vit_b — Meta AI’s smallest Segment Anything ViT (the sam_vit_b_01ec64.pth checkpoint), used by FaceDetailer to mask the detected face for the inpainting pass.

All public, all linkable. Reproducing the stack is one shell script and one recipe; Set up ComfyUI for your own content pipeline walks through it end to end.

The hero artifact: the recipe

The single most useful file to understand the pipeline is a recipe. Here is one, with in-world strings genericized:

{
  "recipe_id": "<age>.<kind>.<slug>.v<N>",
  "kind": "<kind>.<sub-kind>",

  "lever_schema": {
    "type": "object",
    "additionalProperties": false,
    "required": ["<lever_a>", "<lever_b>"],
    "properties": {
      "<lever_a>": { "enum": ["val1", "val2", "val3"] },
      "<lever_b>": { "enum": ["val1", "val2"] }
    }
  },

  "locked_fields": {
    "checkpoint": "<base-model>.safetensors",
    "loras": [
      { "name": "<style-lora>.safetensors",
        "strength_model": 0.7,
        "strength_clip":  0.5 }
    ],
    "cfg_scale": 7.5,
    "sampler": "dpmpp_2m",
    "scheduler": "karras",
    "steps": 30,
    "canvas_width": 512,
    "canvas_height": 768,
    "seed_production": 31337,

    "composition_mode": "standalone",
    "anchor_offsets": {
      "feet":      { "n": [0, 0],   "s": [0, 0],
                     "e": [0, 0],   "w": [0, 0] },
      "hand_main": { "n": [16, -24], "s": [-16, -24],
                     "e": [20, -20], "w": [-20, -20] }
    }
  },

  "prompt_templates_by_backend": {
    "pixellab": {
      "endpoint": "/v2/...",
      "description": "..., {lever_a} ..., {lever_b} ..."
    },
    "comfyui": {
      "workflow_ref": "content/recipes/workflows/<id>.workflow.json",
      "positive_prompt_template":
        "..., {lever_a} ..., {lever_b} ..."
    }
  },

  "valid_output_kinds": ["<kind>.<sub-kind>"],
  "default_palette_enforcement": true
}

Three things to notice.

Levers are the only caller-adjustable parameters, and they are constrained to enums. A recipe with three hair colors and two moods has a total output space of exactly six combinations. The CLI rejects any value that isn’t in the enum. This makes the total surface enumerable — you can pre-compute every possible output path before generating a single image — and it makes the fan-out hash deterministic.

Everything else is locked. Checkpoint, LoRA strengths, sampler, scheduler, steps, seed, canvas size, anchor offsets. The recipe author commits to a generation profile at recipe-authoring time and the profile doesn’t shift run-to-run. If you want to change a locked field, you create recipe.v2.json next to recipe.v1.json. In-place mutation of locked fields is forbidden by convention; the validator catches drift; the v1 directory tree stays addressable for audit.

prompt_templates_by_backend is the multi-backend fan-out. A character recipe declares both a ComfyUI workflow (for concept art, generated by a local Stable Diffusion stack) and a hosted pixel-art API template (for the eight-direction sprite rotations the game actually renders). At queue time, the CLI reads the map keys and inserts one row per backend into the saga’s job table. The saga worker picks each row up and routes it to the matching runner via reflection-based registry discovery. A closed enum here would mean editing the engine to add a new backend; the registry means a new backend is a new class with an attribute.

This last property — adding a content type without editing engine code — is the rule that triggered the rebuild covered in our previous post. Recipe-as-type-object is the same pattern, applied at the content layer.

The receipt: when we added the second backend kind to the pipeline, it landed as a new JSON recipe and zero edits to the runner, the saga, the registry, the queue, or the dependency-injection wiring. Recipe-as-type-object passed the same extensibility test that closed-enum dispatch had failed.

The pipeline, top to bottom

Here is what happens to an asset, end to end.

graph TD
    Recipe["Recipe (JSON)"]
    CLI["CLI: art generate"]
    Queue[("art_jobs (Postgres)")]
    Concept["Concept generation<br/>(local Stable Diffusion)"]
    ConceptGate{"Concept gate<br/>(human)"}
    Rotation["Rotation generation<br/>(hosted pixel-art API)"]
    Stage["Art-pool staging<br/>(content-addressed path)"]
    ReviewGate{"Final review gate<br/>(human)"}
    Promotion["Promotion<br/>(copy to approved/)"]
    Projection["Manifest entry projection"]
    Manifest[("Age manifest")]

    Recipe --> CLI
    CLI -->|"one row per backend"| Queue
    Queue --> Concept
    Queue --> Rotation
    Concept --> ConceptGate
    ConceptGate -- "approved" --> Rotation
    Rotation --> Stage
    Stage --> ReviewGate
    ReviewGate -- "approved" --> Promotion
    Promotion --> Projection
    Projection --> Manifest

Walking the stages:

Recipe authoring. A human writes a JSON file under content/recipes/. A bash validator (validate-recipe-coverage.sh) runs on every push and refuses the commit if a required field is missing, a lever isn’t an enum, or a component contract declares a required_recipe_id that doesn’t have a matching file. Human time: minutes.
Job submission. The CLI command art generate <recipe-id> --<lever> value validates the lever values against the recipe’s schema, computes a content-addressed hash of the lever binding, and inserts one art_jobs row per declared backend. --dry-run short-circuits the database write. Wall time: under a second.

3a. Concept generation (characters). The saga worker claims a pending row, substitutes lever values into the ComfyUI workflow template, posts it to the local ComfyUI daemon, polls until completion, and writes the resulting PNG to a content-addressed path. Wall time on a 4 GB consumer GPU: 30–90 seconds per concept (512×768 base + a face-detection inpainting pass + a 4× ESRGAN upscale).

3b. Tile / object generation (terrain, environment). Tiles and props skip the concept gate entirely. The saga worker posts directly to the hosted pixel-art API with the prompt template, polls the background-job endpoint, and writes the result. Wall time: 10–140 seconds depending on endpoint (tiles ~35 s, icons ~15 s, environment objects ~90 s).

Concept gate (characters only). I run art review-concepts <brief-id>. The terminal displays each concept image; I approve or reject. Approved concepts advance to rotation generation. Rejected concepts go to a terminal Rejected state and never reach the paid API. This is the most cost-load-bearing review step in the pipeline.
Rotation generation (characters). For each approved concept, the saga calls the hosted pixel-art API’s “create character with eight directions” endpoint, supplying the concept image as both reference and seed. Wall time: 60–120 seconds. Output is a ZIP containing eight 1×N rotation strips plus skeleton data.
Art-pool staging. The runner copies the bytes to art_pool/candidates/<recipe_id>/<hash16>/<backend>/000.png. The path is the content-addressed hash of the lever binding; multiple frames live as 000.png, 001.png, etc.
Final review gate. art review <job-id> shows me the staged candidate. I approve or reject.
Promotion. art promote <recipe-id> <hash16> --backend <name> --frame 0 copies the approved candidate to an approved/ subdirectory via copy-then-atomic-rename. Once promoted, the file is frozen on disk. Re-runs of upstream generation can land new candidates next to it; they cannot overwrite it.
Manifest entry projection. RecipeManifestProjector.Project() produces a ManifestEntry record carrying composition_mode, anchor_offsets, variant_sprites, and asset_path (pointing at the frozen file). The projector ships today; the actual write into content/ages/<age>/v<N>/manifest.json is the next thing queued for this surface area.

The Age manifest already carries entries for characters and tiles whose v2 generation hasn’t run yet — those entries hold placeholder asset paths. When promotion completes for a recipe, the projection updates the matching entry in place rather than appending a new one. The manifest is an authored spine; the pipeline fills in cells.

The sprite-atlas packing step lives in package.json as a stub today (pack:sprites is a placeholder). The plan is to wire free-tex-packer into the manifest-write step so promotion → atlas → client load is one motion. It isn’t shipped yet, and saying so out loud is part of the discipline of these posts.

I am the art director

The repo has assets/art-director.log and assets/art-directed/. The naming was deliberate, and it does not mean what it looks like.

There is no LLM judging the art. There is no vision model deciding which concept advances. The “art director” in this pipeline is a governance structure — two human-review gates that sit between every generation step and the shipped manifest, enforced by the saga’s state machine. I’m the human in question.

The assets/art-directed/ directory is a content-addressed cache. Each subdirectory is named by the SHA-256 hash of the inputs that produced it (dc61df083f4a49c5/...) and contains stage-numbered output files: stage-1-concept.png from the local stack, stage-3-rotations.zip from the hosted API. The content addressing is what makes the pipeline survive process death — when the worker comes back up, it checks “is the output already at this path?” before re-issuing a vendor call. Two runs with identical inputs collapse into one set of files.

The assets/art-director.log is the append-only execution transcript from the v1 CLI pipeline that preceded the saga rebuild — 787 lines of timestamped per-asset outcomes, costs, and diagnostic detail from spring 2026. It exists because in the old pipeline, my eye was the only place log information landed; in the v2 saga, every state transition is in Postgres and the log is supplementary.

I exercise the directorial role at exactly two stops: the concept gate and the final review gate. Both are CLI commands that transition art_jobs rows. Rejected jobs go to a terminal state without burning further vendor credits. The recipe’s levers (hair=red, mood=stoic) produce deterministic fan-outs that I evaluate as a batch — generate ten concept images by sweeping a lever’s enum, look at them, keep two, throw away eight. I’m the art director. The pipeline is my tool.

This isn’t an ideological choice. The cost of a wrong vision-model judgment in this domain is a generation that doesn’t look like the world; the cost of a right vision-model judgment is asset throughput. Until I have a judge calibrated on my taste — not a public CLIP-style scorer trained on internet aesthetics — the throughput win isn’t worth the calibration risk. My eye is cheaper, faster, and accurate by construction in a way no off-the-shelf scorer can be for a generational life-sim’s pixel art. So I stay in the loop.

Bounding the random

The pipeline uses generative models, which are probabilistic by construction. The simulation side of Skaldborn is built on a determinism guarantee. These two things have to coexist or the architecture is a lie.

They coexist through four mechanisms.

Seed pinning. Every recipe’s locked_fields carry a production seed (31337 in the canonical example). The ComfyUI workflow template hardcodes the seed in the KSampler node. Per-concept variation comes from a child_index field on each art_jobs row: the effective seed is recipe_seed + child_index, so a ten-concept batch produces seeds 31337 through 31346. Each seed is reproducible given the same checkpoint, LoRA, prompt, and parameters.

Content-addressed paths. The function LeverBindingHash.Compute() produces a SHA-256 of the lever binding map (keys sorted ordinal-ascending, compact JSON, no Unicode escaping). The first 16 hex characters become the directory name: art_pool/candidates/<recipe_id>/<hash16>/<backend>/000.png. Identical inputs produce the same hash; re-running with the same levers lands in the same directory. The stager writes there, the resolver reads from there, the worker checks for existence before issuing a vendor call.

Input:   {"hair":"red","mood":"stoic"}
SHA-256: a1b2c3d4e5f67890... (truncated to 16 hex chars)
Path:    art_pool/candidates/<recipe>/a1b2c3d4e5f67890/<backend>/000.png

Recipe versioning. If a checkpoint changes (new model, updated LoRA), the recipe is versioned: <recipe>.v1 becomes <recipe>.v2. In-place mutation of locked_fields is forbidden by convention and called out in the architecture decision that governs the pipeline. Old recipes remain addressable; old art_jobs rows still reference them; the directory tree separates outputs of different generation profiles cleanly.

Frozen-output promotion. The promotion step copies a candidate to approved/ via copy-then-atomic-rename. Once promoted, the file is immutable from the pipeline’s perspective. Upstream regeneration cannot overwrite it; new generations land in a new candidate slot and require an explicit second promotion from me to replace the approved file. The manifest entry produced by the projector carries the asset_path to the promoted file. The simulation consumes the frozen manifest. It never sees the candidate pool.

The honest gap: determinism at the hosted pixel-art API is not contractually guaranteed. The vendor’s seed-determinism story across the eight-rotation endpoint is a “probably yes” rather than a “documented yes.” The pipeline treats the hosted output as “potentially different on each call” and relies on the promotion gate as the determinism boundary. Once a sprite is in approved/, it doesn’t matter whether the same sprite would come back the same way next time.

The saga

The v2 pipeline is a nine-state saga backed by a Postgres art_jobs table. Each row tracks its job through:

stateDiagram-v2
    [*] --> Pending
    Pending --> ConceptGenerating
    ConceptGenerating --> AwaitingConceptApproval
    AwaitingConceptApproval --> RotationGenerating: approve
    AwaitingConceptApproval --> Rejected: reject
    RotationGenerating --> Persisting
    Persisting --> AwaitingReview
    AwaitingReview --> Approved: approve
    AwaitingReview --> Rejected: reject
    ConceptGenerating --> Failed: error
    RotationGenerating --> Failed: error
    Persisting --> Failed: error
    Approved --> [*]
    Rejected --> [*]
    Failed --> [*]

Two design choices carry the durability story.

Claim ordering uses SELECT FOR UPDATE SKIP LOCKED. Each worker claims exactly one job at a time, ordered by (state, priority DESC, created_at). Multiple workers can run concurrently without colliding. A TtlReclaimBackgroundService periodically sweeps for jobs whose processing_deadline_at has passed and re-queues them; after three reclaims a job is force-failed with error_class='stage_timeout_repeated' so a permanently stuck job can’t loop forever.

State output lands on disk before the row transitions. The saga writes generated bytes to the content-addressed path before updating the row to the next state. If the worker crashes between writing the file and updating the database, the next pickup finds the file already present and skips the vendor call. The cost ceiling is preserved across crashes; partial work survives.

The schema is built by an embedded migration runner that ships inside the worker. Five migrations live as embedded resources in the project (V1 through V5) and apply sequentially on startup. V1 creates the table with twenty-something columns. V2 drops an over-eager category CHECK constraint that was rejecting valid recipes. V3 adds cost-enforcement columns. V4 adds reclaim tracking. V5 adds the child_index column the seed-variation fix needed (more on that in a war story below). Six indexes — claim order, brief id, trace id, category-state, TTL reclaim, created-at — keep the common queries cheap.

A daily cost ceiling (ART_WORKER_COST_CEILING_DAILY_USD, default $25) is checked before each vendor call. If the day’s spend exceeds the ceiling, the worker stops claiming new jobs until UTC midnight. This is the failsafe behind any pipeline that touches a paid API: the per-call cost estimates are unreliable, the daily cap isn’t.

What we validate (and what we don’t)

The repo’s scripts/ directory has roughly twenty-five fail-close validators that run on every push. Most of them protect the simulation. The art side has one mechanical validator. We’re going to be honest about why.

validate-recipe-coverage.sh (Tier 1, wired into make validate-governance) checks that every component contract that declares required_recipe_ids has matching files in content/recipes/. It shape-validates each recipe: all eight required top-level fields present, lever_schema.additionalProperties === false, every lever property declares an enum keyword. The validator runs three controlled-failure scenarios against temp fixtures as a self-test (--self-test) so the validator itself stays falsifiable. If a recipe file is missing or a lever isn’t an enum, the push fails with a structured error.

Beyond that bash script, the pipeline relies on:

A C# LeverBindingValidator at job-submission time (rejects unknown levers, missing required levers, values outside the enum set).
The art generate --dry-run pre-flight (validates and prints what would queue, without writing).
A circuit breaker on the hosted API runner (opens at ten consecutive failures; subsequent calls return immediately).
The two review gates I described above.

Why not more mechanical gates? Because art quality is fundamentally a judgment call. You cannot write a bash script that determines whether a character portrait looks good. The pipeline compensates by making me the gate at two explicit review steps and ensuring no generated asset reaches the manifest without my “yes.” The mechanical validators catch structural errors — missing files, malformed levers, schema drift. Aesthetic errors are mine to catch. Pretending otherwise would be dishonest, and dishonesty here would mean shipping bad art under a fig leaf of automation.

A few additional validators (validate-tile-cohesion.ts, validate-composition.ts, validate-rendering.ts, validate-motion.ts) are scoped for tile variety, transition coverage, color-profile adherence, and motion presence. Tier 2 is in progress; Tiers 3–5 are not started. They will land before the manifest write does.

War stories

Three things broke instructively. They were teachers.

The “ten identical concepts” surprise. The first time we tested the concept-gate batch flow, art queue --concepts 10 produced ten identical PNGs. All ten child jobs shared the recipe’s pinned seed; the ComfyUI workflow hardcoded that seed in the sampler; the pipeline was faithfully and deterministically producing ten copies of the same image. Determinism was working too well. The fix, in commit 29f0028, added a child_index column to art_jobs (migration V5) and made the effective seed recipe_seed + child_index. Child 0 gets seed 31337, child 1 gets 31338, and so on. Each concept is reproducible (same child_index always produces the same image) but distinct from its siblings. Lesson: determinism and variety are not opposites, but they don’t reconcile by accident. You need an explicit variance axis, named, in the data.

The rotation_urls null surprise. Early character generation called the hosted API’s “create character with directions” endpoint, polled the resulting background job, fetched the character object, and then crashed trying to download rotation images. The API response had rotation_urls present in the schema but null in the actual response body. Every character generation failed identically across five consecutive batch runs in March; the log records each one. The fix was to read the full polling envelope rather than the immediate response: the character_id and rotation_urls materialize in a last_response field on the completed background job, not in the 202 Accepted from the original POST. The vendor’s OpenAPI schema typed last_response as opaque object, so the first implementation reasonably assumed the completion payload had the same shape as the synchronous one. Lesson: when an async API hands you a background job, the completion payload is not the same shape as the submission response. Poll the completion endpoint, dump the raw JSON, and then build the deserializer.

Saga CAS vs. TTL sweep. The saga used compare-and-swap on updated_at to prevent double-processing: UPDATE art_jobs SET state = ... WHERE id = ... AND updated_at = @expected. In overnight runs with the face-detection inpainting step enabled (longer generation times), the worker would claim a job, run the generation for sixty-plus seconds, then attempt the CAS update — which failed because the TTL reclaim sweep had touched the row’s updated_at in the meantime. The job was stuck: claimed but never advanced, never reclaimed (the sweep only touches un-claimed jobs). Commit 64c8e3e5 added a continuation loop: if the CAS fails, the worker re-reads the row, verifies it still owns the claim, and retries the transition. The same commit bumped polling ceilings for overnight workloads. Lesson: if your saga uses optimistic concurrency, make sure your background sweeps don’t silently invalidate the CAS token of in-flight work.

What it’s like to add an asset

A new character portrait, end to end, looks like this:

# Validate the recipe and lever values without queueing
art generate <recipe-id> --hair red --mood stoic --dry-run

# Queue the job
art generate <recipe-id> --hair red --mood stoic

# (~60 seconds: local concept generation)
art list-candidates <recipe-id>
art review-concepts <brief-id>            # me: approve/reject

# (~90 seconds: hosted API rotation generation)
art review <job-id>                       # me: approve/reject

# Promote to frozen
art promote <recipe-id> <hash16> --backend <name> --frame 0

About five commands, three to five minutes wall time, dominated by generation time. If the recipe doesn’t already exist, add roughly fifteen minutes of authoring (write the JSON, choose lever values, draft the prompt template). The validator catches structural mistakes before the push reaches CI.

There’s no hot-reload. Each iteration is a full generate-review-promote cycle, bottlenecked by the 30–140 seconds the model needs. The single change that would noticeably tighten the loop is wiring promotion directly into the manifest write and the atlas pack — collapsing today’s “promote → run a copy script → rebuild the client” chain into one art ship command. The substrate exists (the projector is shipped); the wiring is the next thing on the queue.

Receipts

A grab-bag of concrete numbers, current as of this post:

Manifest entries today: 157 across characters, portraits, buildings, terrain tiles and tilesets, environment objects, icons, and UI elements. All currently in a Generated state under the v1 path; none have been promoted through the v2 pipeline yet (we’re mid-cutover).
Raw sprite files in the client’s asset tree: ~2,360 files, ~9.6 MB total. Most of that is character animation frames.
v2 outputs in assets/art-directed/: 8 content-hash directories so far; ~38 MB total (concept PNGs are ~6 MB each at 2048×3072 post-upscale; rotation ZIPs are ~150 KB).
Concept generation time on a 4 GB consumer GPU under WSL2: 30–90 s per concept. The VRAM-safety flags on the ComfyUI launcher are load-bearing — --cpu-vae in particular, because the VAE decode spike will crash the entire WSL session on a 4 GB card without it. (Bare-metal Linux merely OOM-kills the process. WSL takes the whole subsystem down.)
Hosted-API generation time: tiles 35–105 s, portraits 25–135 s, icons 11–22 s, UI elements 15–50 s.
Hosted-API cost: $0.008–$0.010 per image. Daily ceiling: $25.
Recipe catalog: 2 canonical recipes shipped, 1 ComfyUI workflow template. Recipe files are 1.7–3.6 KB each.
Saga schema: 5 embedded SQL migrations, 20-plus columns per row, 6 indexes.

What we’d tell ourselves in February

A list, since this kind of advice tends to land best as a list.

Put the operator review gate before the expensive step, not after. Local concept generation costs nothing; the hosted rotation API costs money and time. Rejecting a bad concept before it reaches the paid API saved roughly 60% of vendor spend in the first batch runs. Design your pipeline so cheap evaluation precedes expensive generation.
Version your recipes; don’t mutate them. When locked_fields change — new checkpoint, new LoRA strength, different canvas — create recipe.v2.json next to recipe.v1.json. Old versions remain addressable for audit. In-place mutation makes “what produced this asset?” unanswerable after the fact. Immutable recipe files are a cheap form of experiment tracking.
Content-address your outputs by their inputs. The SHA-256 hash of the lever binding means re-running a generation with the same parameters is a no-op (the file already exists). This saves you from re-generating after worker crashes, and it makes the pipeline inherently idempotent. If you build any batch pipeline that talks to expensive APIs, compute the content hash before the API call and check the filesystem first.
Constrain your variance axes to enums, not free-form strings. Free-form parameters feel flexible until you try to answer “have we generated all the variants we need?” and discover you can’t define “all.” Enums make the total output space finite, the content hash deterministic, and the validation mechanical. The expressivity you give up was never load-bearing; the enumerability you gain is.
Make your queue durable independently of your interactive session. Our v1 pipeline ran in-process inside a CLI invoked from a terminal session. A lost tmux pane destroyed the batch state. The v2 pipeline stores every job in Postgres — process death loses at most one in-flight generation, and the TTL reclaim sweep picks it up automatically. If your pipeline takes more than five minutes end-to-end, it should not live inside the process that initiated it.
Separate “generate” from “ship.” Distinct generate, review, promote, and manifest-write steps mean you can generate fifty variants, review them over coffee, promote the three best, and only those three enter the game. Pipelines that auto-ship every generation force you to be conservative with parameters; pipelines with an explicit promotion step let you be exploratory with generation and selective with shipping.
Budget your API costs with a daily ceiling, not just a per-call estimate. A hard daily cap ($25 in our case) prevents a runaway batch from spending more than you planned. Per-call estimates are unreliable — our v1 log shows $0.000 cost for images the vendor actually charged for. The ceiling is the failsafe.
Don’t ask the AI to judge the AI unless your judge is calibrated on your world. A vision-model gate sounds like throughput. The failure mode is a generation that looks like something, just not yours — and that failure ships silently because the judge said yes. Until you can train and validate a judge against your own taste, the operator gate is cheaper than the cost of one shipped wrong-looking asset.

What’s next

The next post in the launch arc takes the manifest itself as its subject. The recipe’s composition_mode, anchor_offsets, and variant_sprites carry an entire visual-composition vocabulary; the renderer reads them as data, the projector emits them, the Age manifest binds them. That surface area is its own post.

Further out: the audio pipeline. A separate service following the same governance pattern — external producer, build-time snapshot, manifest-bound, no runtime authority — with a completely different stack.

Adjacent to both: validate-recipe-coverage.sh is one of roughly twenty-five fail-close validators wired into our pre-push hook. The broader story — how we turn architectural rules into CI gates — is its own post.

Companion to this one: Set up ComfyUI for your own content pipeline — a step-by-step walkthrough of the local Stable Diffusion install end to end, the PixelLab API integration, and the launch flags that keep VAE decode from OOM-ing on a 4 GB consumer GPU.

If you want to follow along, subscribe via the form at the bottom of any page — one short email when the next post lands. If you want to argue, write to devlog@skaldborn.com.

Everything else is the boring engineering of making it true.

Set up ComfyUI for your own content pipeline (on a 4 GB consumer GPU)

Sun, 10 May 2026 00:00:00 GMT

This is the companion tutorial to Skaldborn’s art content pipeline. That post explained the architecture; this one walks through the local Stable Diffusion half end to end, on the same 4 GB consumer GPU it runs on in production, and then shows how to drive PixelLab’s hosted API to turn a concept image into an 8-direction sprite. Ubuntu (or any modern Linux distro) is assumed.

You don’t have to have read post 04 to use this. The install pattern, the launch-flag set, the FaceDetailer + upscale graph, and the PixelLab integration generalize to any content pipeline that uses ComfyUI as a generation backend with a sprite-renderer downstream. The model choices below are tuned for fantasy/RPG art; substitute your own if your domain is different.

The canonical scripts and a tighter reference live at clunasco/skald-forge. This post is the narrative version with the lessons and the small landmines baked in. Time budget: about 30 minutes for the local stack if your hardware cooperates, plus a handful of minutes once you have a PixelLab API key.

Why this is hard on a 4 GB card
1. Prerequisites
2. Clone skald-forge and install ComfyUI
3. Install the FaceDetailer custom nodes
4. Configure your CivitAI token
5. Download the production model stack
6. Launch ComfyUI
7. Build the production workflow
8. From concept to sprite: PixelLab
9. Tuning for more VRAM
10. Troubleshooting
How Skaldborn consumes this
What’s next

Why this is hard on a 4 GB card

Default ComfyUI will OOM on SD 1.5 generation if you have 4 GB of VRAM. That’s annoying. What’s especially annoying is where it OOMs: the VAE decode at the end of generation produces a VRAM spike that fires after the 30+ seconds of sampling. You wait, you see the image almost complete, and the process dies on decode. Every generation. Until you add --cpu-vae to the launch script and the spike moves to system RAM.

Three of the launch flags in this guide are there specifically to make 4 GB work. Drop any of them on a small card and the symptoms range from “OOM during sampling, stack trace early” (you find out fast) to “OOM at decode, watch your generation die at the finish line, repeatedly” (you waste hours). The flag table in section 6 tells you which is which.

If you have more VRAM, you’ll want to drop some of these flags as you go up. Section 9 covers that.

↑ Back to top

1. Prerequisites

Hardware. An NVIDIA GPU with at least 4 GB of VRAM. Reference rig: RTX 3050 Laptop 4 GB.

Software.

Ubuntu 22.04+ (or any modern Linux distro)
A working NVIDIA driver — recent enough to support CUDA 12.4. On Ubuntu, sudo apt install nvidia-driver-550 (or newer) is the usual path; reboot after.
The handful of base packages this guide assumes:

sudo apt update
sudo apt install -y python3 python3-venv git tmux curl

Verify CUDA is alive before going further:

nvidia-smi

You should see your GPU and driver version. If this fails, fix the driver before continuing.

A CivitAI account + API token is needed for the production checkpoints/LoRAs (CivitAI gates downloads behind auth). Get one at civitai.com/user/account. Save it; you’ll need it in section 4.

A PixelLab account + API token is needed for the rotation step in section 8. Free trial credit is enough to test the flow end to end. Get one at pixellab.ai/signup; generate the token from your account settings.

↑ Back to top

2. Clone skald-forge and install ComfyUI

git clone https://github.com/clunasco/skald-forge.git ~/workspace/projects/skald-forge
cd ~/workspace/projects/skald-forge
./install_comfyui.sh

install_comfyui.sh is idempotent — re-run it any time to update. Six things happen:

Creates comfyui-venv/ (a Python venv, sibling to ComfyUI/).
Clones comfyanonymous/ComfyUI into ComfyUI/.
Installs PyTorch with the CUDA 12.4 wheels (torch, torchvision, torchaudio from the PyTorch CUDA 12.4 index).
Installs ComfyUI’s requirements.txt.
Installs ComfyUI-Manager into ComfyUI/custom_nodes/.
Prints a CUDA-visibility check at the end. You should see cuda avail : True and your GPU name.

If step 6 says False, you’re holding a CPU-only PyTorch wheel — usually a cached environment from a prior install. The script pins the CUDA 12.4 index, so this only happens if something else got there first. Force a clean reinstall:

source comfyui-venv/bin/activate
pip install --force-reinstall torch torchvision torchaudio \
  --index-url https://download.pytorch.org/whl/cu124

↑ Back to top

3. Install the FaceDetailer custom nodes

The production graph uses two custom-node packages from ltdrdata:

Impact Pack — provides FaceDetailer, the inpainting node that re-runs generation on the face region after the main pass. SD 1.5 at full-body 512×768 produces faces that are functionally mush; FaceDetailer is the fix.
Impact Subpack — provides the UltralyticsDetectorProvider node that loads face_yolov8n.pt for the face bounding box.

Install both:

cd ComfyUI/custom_nodes
git clone https://github.com/ltdrdata/ComfyUI-Impact-Pack.git
git clone https://github.com/ltdrdata/ComfyUI-Impact-Subpack.git
cd -

source comfyui-venv/bin/activate
pip install -r ComfyUI/custom_nodes/ComfyUI-Impact-Pack/requirements.txt
pip install -r ComfyUI/custom_nodes/ComfyUI-Impact-Subpack/requirements.txt
deactivate

Once ComfyUI is running you can also install missing custom nodes through the Manager UI (“Manager” button → “Install Missing Custom Nodes”). The git-clone path is what skald-forge documents because it’s reproducible from a fresh checkout without a running server.

↑ Back to top

4. Configure your CivitAI token

download_models.sh auto-loads .env from the repo root. Create it:

cat > .env <<'EOF'
CIVITAI_TOKEN=your_civitai_api_token_here
EOF
chmod 600 .env

.env is gitignored at the skald-forge level. Don’t commit it.

↑ Back to top

5. Download the production model stack

Six files. Total disk: ~6.2 GB. All paths are relative to ComfyUI/models/. Sizes and source URLs are also recorded in models_inventory.md.

# 1. Default SD 1.5 VAE — used by checkpoints that don't bake one in
./download_models.sh --vae-default

# 2. aZovyaRPGArtistTools v4VAE — the production checkpoint (~5.7 GB)
./download_models.sh --checkpoint \
  https://civitai.com/api/download/models/251729 \
  --name aZovyaRPGArtistTools_v4VAE.safetensors

# 3. NorseViking_v10 — the production style LoRA (~37 MB)
./download_models.sh --lora \
  https://civitai.com/api/download/models/31804 \
  --name NorseViking_v10.safetensors

# 4. negative_hand-neg — fixes hand anatomy without dragging the style (~25 KB)
./download_models.sh --embedding \
  https://civitai.com/api/download/models/60938 \
  --name negative_hand-neg.pt

# 5. easynegative — companion general-purpose negative embedding (~24 KB)
./download_models.sh --embedding \
  https://civitai.com/api/download/models/9208 \
  --name easynegative.safetensors

# 6. 4xFoolhardyRemacri — post-generation upscaler (~67 MB)
./download_models.sh --upscaler \
  https://civitai.com/api/download/models/164821 \
  --name remacri_original.safetensors

The face detector and segmenter aren’t on CivitAI — they live on Hugging Face and Meta’s S3:

mkdir -p ComfyUI/models/ultralytics/bbox ComfyUI/models/sams

# YOLOv8-nano face detector (Bingsu/adetailer canonical release)
curl -L -o ComfyUI/models/ultralytics/bbox/face_yolov8n.pt \
  https://huggingface.co/Bingsu/adetailer/resolve/main/face_yolov8n.pt

# Segment Anything ViT-B (Meta's official checkpoint)
curl -L -o ComfyUI/models/sams/sam_vit_b_01ec64.pth \
  https://dl.fbaipublicfiles.com/segment_anything/sam_vit_b_01ec64.pth

Verify everything landed at the right size:

ls -lh ComfyUI/models/checkpoints/aZovyaRPGArtistTools_v4VAE.safetensors  # ~5.7 GB
ls -lh ComfyUI/models/loras/NorseViking_v10.safetensors                   # ~37 MB
ls -lh ComfyUI/models/upscale_models/remacri_original.safetensors         # ~67 MB
ls -lh ComfyUI/models/embeddings/negative_hand-neg.pt                     # ~25 KB
ls -lh ComfyUI/models/ultralytics/bbox/face_yolov8n.pt                    # ~6 MB
ls -lh ComfyUI/models/sams/sam_vit_b_01ec64.pth                           # ~358 MB

If any are wildly wrong (a few KB instead of hundreds of MB), it’s almost always a CivitAI auth issue: your CIVITAI_TOKEN is missing or wrong. The download finished successfully but you got a redirect-to-login HTML page instead of the model. Re-check .env, re-run.

If you want to substitute models for your own domain, this is the surface to swap. The graph in section 7 doesn’t care about model identity, only file presence.

↑ Back to top

6. Launch ComfyUI

./start_comfyui.sh

This launches ComfyUI inside a detached tmux session named comfyui and binds the web UI to http://127.0.0.1:8188. Useful operations:

tmux attach -t comfyui          # see the live log
# Ctrl-b d                      # detach again
tmux kill-session -t comfyui    # stop
./start_comfyui.sh              # start again (kills the old session first)

The launch flags matter. Read them before changing anything:

Flag	Why
`--novram`	Maximum offload to system RAM. Without it, even SD 1.5 will OOM at 4 GB.
`--use-split-cross-attention`	Splits attention to reduce peak VRAM during sampling.
`--cpu-vae`	Load-bearing on 4 GB. The VAE decode at the end of generation spikes VRAM and OOMs the Python process at the very last step. CPU decode adds a few seconds per image but is the difference between “works” and “watch your image die at 99%.”
`--preview-method none`	Live previews cost VRAM. Off.
`--listen 0.0.0.0`	Reachable from Docker containers on the host bridge. Drop if you want loopback-only.
`--port 8188`	Default ComfyUI port.

Why a tmux daemon? Because if you run ComfyUI directly inside the shell that owns it, the server dies when the shell exits — and there’s no signal that “the shell exited” from inside an editor or an automation that thought it had ComfyUI running. tmux outlives any individual shell. The launcher kills any pre-existing session with the same name first, so re-running the script is safe.

Aside. Don’t try to wrap ComfyUI in an editor-managed background process — Claude Code background bash tasks, VS Code task runners, that shape of thing. Background bash tasks get reaped when their parent exits. Use the tmux daemon path so the server outlives whatever spawned it.

↑ Back to top

7. Build the production workflow

Once ComfyUI is up, open http://127.0.0.1:8188. Wire the production graph in the UI. Components, in the order signal flows:

CheckpointLoaderSimple → loads aZovyaRPGArtistTools_v4VAE.safetensors.
LoraLoader → loads NorseViking_v10.safetensors with strength_model: 0.7, strength_clip: 0.5. Connect MODEL/CLIP from the checkpoint to the LoRA loader; the LoRA loader’s outputs are what later nodes consume.
CLIPTextEncode (positive) — your prompt. Anything from “bearded warrior, fur cloak, snowy” to a more elaborate scene description.
CLIPTextEncode (negative) — include embedding:negative_hand-neg, embedding:easynegative plus any usual negative tokens.
EmptyLatentImage at 512×768 (the production canvas size).
KSampler — connect MODEL from the LoRA loader, conditioning from the two text encoders, latent from the EmptyLatentImage. Recommended params: steps: 30, cfg: 7.5, sampler_name: dpmpp_2m, scheduler: karras, seed: 31337 (or whatever you pin for reproducibility).
VAEDecode → uses the VAE baked into the v4VAE checkpoint.
FaceDetailer (Impact Pack):
- BBOX_DETECTOR ← UltralyticsDetectorProvider loading face_yolov8n.pt.
- SAM_MODEL ← SAMLoader loading sam_vit_b_01ec64.pth.
- Defaults are fine. The node detects faces in the decoded image, masks them with SAM, and re-runs generation on the face region — fixing the “face is mush” failure mode SD 1.5 has at full-body 512×768.
UpscaleModelLoader → remacri_original.safetensors → ImageUpscaleWithModel for the 4× pass to 2048×3072.
SaveImage.

Save the workflow JSON via Workflow → Export once it’s wired so you can reload it cleanly. ComfyUI also embeds the full graph in saved PNGs — dragging a generated image back into the canvas reconstructs the workflow. That’s a useful way to ship a reference graph without committing JSON.

Generation time on the reference rig (RTX 3050 Laptop 4 GB) is 30–90 seconds per concept. On a 12 GB+ card with the conservative flags dropped, expect 10–20 seconds.

↑ Back to top

8. From concept to sprite: PixelLab

You have a 2048×3072 concept image. To turn it into something a game engine can render as a moving character, you need eight rotations — one sprite per cardinal and intercardinal direction. PixelLab’s hosted API is what does that step in this pipeline. It takes your concept image as a reference and generates the 8-direction sprite set conditioned on it.

The flow is async: POST a job, poll until it’s done, download the rotations.

Get an API key and set the token

Sign up at pixellab.ai/signup, buy a credit pack (free trial credit is enough to test the flow), and generate an API token from your account settings. Authentication is a Bearer token:

export PIXELLAB_TOKEN=your_pixellab_api_token_here

Pricing scales with output size and mode. A character rotation through Pro mode runs roughly 20–40 generations per request — meaningful spend at production scale. PixelLab’s pricing page has the per-call estimates.

Preprocess your concept

PixelLab’s /v2/create-character-pro endpoint with method: "create_from_concept" takes two image inputs:

concept_image (max 1024×1024) — the visual you want the rotations to follow.
reference_image (max 168×168) — a low-res anchor that shapes the sprite-art treatment.

The production preprocessing for a 2048×3072 ComfyUI output:

Center-crop the concept to a square.
Resize the square to 512×512 with Lanczos for concept_image (well within the 1024 max — keeps the JSON payload manageable).
Resize the same square to 168×168 with Lanczos for reference_image (exactly at the max).

from PIL import Image

def preprocess(path: str) -> tuple[Image.Image, Image.Image]:
    src = Image.open(path).convert("RGB")
    side = min(src.size)
    left = (src.width - side) // 2
    top = (src.height - side) // 2
    square = src.crop((left, top, left + side, top + side))
    concept = square.resize((512, 512), Image.Resampling.LANCZOS)
    reference = square.resize((168, 168), Image.Resampling.LANCZOS)
    return concept, reference

Submit the rotation job

PixelLab takes images as base64-encoded PNGs inside the JSON body — the same pattern most image-input APIs use. In a shell, encoding any PNG to that string is one command:

base64 -w 0 concept_512.png > concept.b64
base64 -w 0 reference_168.png > reference.b64

The -w 0 disables line wrapping (base64 defaults to wrapping at 76 columns; JSON doesn’t tolerate embedded newlines in strings).

In Python the equivalent is two steps — write the in-memory PIL.Image to a PNG byte buffer, then base64-encode the bytes:

import base64, io, os, requests

def b64_png(img: Image.Image) -> str:
    """Encode an in-memory PIL image to a base64 PNG string for JSON."""
    buf = io.BytesIO()
    img.save(buf, format="PNG")
    return base64.b64encode(buf.getvalue()).decode("ascii")

concept, reference = preprocess("concept.png")

response = requests.post(
    "https://api.pixellab.ai/v2/create-character-pro",
    headers={"Authorization": f"Bearer {os.environ['PIXELLAB_TOKEN']}"},
    json={
        "method": "create_from_concept",
        "description": "norse warrior, fur cloak, axe",
        "image_size": {"width": 64, "height": 64},
        "view": "side",
        "concept_image": b64_png(concept),
        "reference_image": b64_png(reference),
    },
    timeout=30,
)
response.raise_for_status()
job_id = response.json()["background_job_id"]
print(f"job: {job_id}")

The response is 202 Accepted with a background_job_id. Pro mode typically takes 60–120 seconds end to end.

Poll the background job

import time

while True:
    poll = requests.get(
        f"https://api.pixellab.ai/v2/background-jobs/{job_id}",
        headers={"Authorization": f"Bearer {os.environ['PIXELLAB_TOKEN']}"},
        timeout=30,
    )
    poll.raise_for_status()
    body = poll.json()
    status = body["status"]
    if status == "completed":
        result = body["last_response"]
        break
    if status == "failed":
        raise RuntimeError(f"PixelLab job failed: {body}")
    print(f"status: {status}")
    time.sleep(2)

The async-shape gotcha. The completion payload is on body["last_response"], not on the immediate response from your POST. PixelLab’s OpenAPI types last_response as opaque object, so the shape isn’t statically discoverable. Early integrations reasonably assumed the completion response would echo the submission shape and crashed trying to read fields that weren’t there. The submission response is “your job is queued”; the completion response is “your job is done, here are the artifacts.” Different shapes. Read the completion endpoint, dump the raw JSON the first time you call it, then build your deserializer.

Download the rotations

last_response for character endpoints contains a rotation_urls dictionary keyed by direction:

import urllib.request
from pathlib import Path

DIRECTIONS = ["south", "south-east", "east", "north-east",
              "north", "north-west", "west", "south-west"]

Path("rotations").mkdir(exist_ok=True)
for direction in DIRECTIONS:
    url = result["rotation_urls"][direction]
    urllib.request.urlretrieve(url, f"rotations/{direction}.png")

You now have eight 64×64 sprite frames — one per cardinal and intercardinal direction. Wire them into your engine’s animation/rotation system however that works in your stack.

Bound your spend

Pro mode bills per generation, so a runaway loop or a sloppy retry can burn money fast. Two patterns from the production pipeline that protect against that:

Hard daily ceiling. Track total spend per UTC day; refuse new submissions when the day’s running cost exceeds a configured cap. The Skaldborn pipeline runs a $25/day default — when crossed, the worker stops claiming new jobs until the next UTC day.
Circuit breaker. Track consecutive vendor failures; stop submitting after N failures in a row. PixelLab does occasionally return “missing image data” errors on transient backend issues; the production pipeline retries each job up to three times before going terminal.

Neither is in the API itself. Both are your responsibility on the calling side.

Pure-shell smoke test

If you’d rather verify the API end to end before writing any Python — useful for confirming your token, the request shape, and the polling/download flow all work together — the whole sequence runs in shell with curl, jq, and imagemagick:

#!/usr/bin/env bash
set -euo pipefail

# 1. Preprocess (center-crop to square, then resize)
SIDE=$(identify -format '%[fx:min(w,h)]' concept.png)
convert concept.png -gravity center -crop "${SIDE}x${SIDE}+0+0" +repage \
  -resize 512x512 -filter Lanczos concept_512.png
convert concept.png -gravity center -crop "${SIDE}x${SIDE}+0+0" +repage \
  -resize 168x168 -filter Lanczos reference_168.png

# 2. Encode
base64 -w 0 concept_512.png   > concept.b64
base64 -w 0 reference_168.png > reference.b64

# 3. Submit
JOB_ID=$(jq -n \
  --rawfile concept   concept.b64 \
  --rawfile reference reference.b64 \
  '{
    method: "create_from_concept",
    description: "norse warrior, fur cloak, axe",
    image_size: { width: 64, height: 64 },
    view: "side",
    concept_image:   $concept,
    reference_image: $reference
  }' | curl -sX POST https://api.pixellab.ai/v2/create-character-pro \
       -H "Authorization: Bearer $PIXELLAB_TOKEN" \
       -H "Content-Type: application/json" \
       -d @- | jq -r .background_job_id)

echo "job: $JOB_ID"

# 4. Poll
while true; do
  BODY=$(curl -sX GET "https://api.pixellab.ai/v2/background-jobs/$JOB_ID" \
    -H "Authorization: Bearer $PIXELLAB_TOKEN")
  STATUS=$(echo "$BODY" | jq -r .status)
  echo "status: $STATUS"
  [[ "$STATUS" == "completed" ]] && break
  [[ "$STATUS" == "failed" ]] && { echo "$BODY" >&2; exit 1; }
  sleep 2
done

# 5. Download the rotations
mkdir -p rotations
echo "$BODY" \
  | jq -r '.last_response.rotation_urls | to_entries[] | "\(.key) \(.value)"' \
  | while read -r dir url; do curl -sL "$url" -o "rotations/$dir.png"; done

echo "done. rotations in ./rotations/"

Same flow as the Python above, with no language runtime to install. Use it to validate your environment, then graduate to the Python integration for anything you’d actually deploy.

↑ Back to top

9. Tuning for more VRAM

If you’re replicating this on a card with more headroom, relax the launch flags. Edit start_comfyui.sh:

8 GB: drop --novram, keep --cpu-vae. You may also drop --use-split-cross-attention if you don’t see OOMs.
12 GB+: drop all three of --novram, --use-split-cross-attention, --cpu-vae. GPU VAE decode is much faster.

--cpu-vae is the one to be most careful about removing. It’s there specifically because the VAE decode spike is the one that fires at the very end of generation — drop it on a marginal card and you’ll watch every generation die at 99%. If your ratio of “decode succeeded” to “decode OOMed” ever dips, put it back.

↑ Back to top

10. Troubleshooting

The four ComfyUI-side things that break for everyone, in roughly the order they break:

nvidia-smi doesn’t work or shows the wrong driver. Install or update the NVIDIA driver. On Ubuntu: sudo ubuntu-drivers autoinstall is the easy path; sudo apt install nvidia-driver-550 (or newer) is the explicit one. Reboot after, verify with nvidia-smi.

torch.cuda.is_available() returns False after install. You probably have a CPU-only PyTorch wheel. Re-run install_comfyui.sh; it pins the CUDA 12.4 index. If a previous environment cached the wrong wheel, force-reinstall:

source comfyui-venv/bin/activate
pip install --force-reinstall torch torchvision torchaudio \
  --index-url https://download.pytorch.org/whl/cu124

CivitAI download fails with 401. CIVITAI_TOKEN is missing or wrong. Tokens come from civitai.com/user/account, not from the session cookie.

ComfyUI shows red “missing node” boxes when loading a workflow. Custom nodes aren’t installed. Either use ComfyUI-Manager → “Install Missing Custom Nodes,” or git-clone Impact Pack + Subpack as in section 3.

Generation OOMs at the very end (after sampling completes). VAE-decode VRAM spike. Make sure --cpu-vae is in the launch command.

And the PixelLab-side ones:

401 Unauthorized on the POST. PIXELLAB_TOKEN is missing or wrong. Tokens come from your PixelLab account settings, not the signin cookie.

402 Payment Required. Out of credits. Buy a pack on pixellab.ai.

422 Unprocessable Entity. Validation error. The most common cause is an oversized concept_image or reference_image — verify the preprocessing produced 512×512 and 168×168 respectively, and that you’re sending base64 PNG bytes (not the raw PIL object or a path).

429 Too Many Requests. Rate or concurrency limit hit. Back off; if you’re running batches, throttle the submission rate.

Job sits at status: "processing" for many minutes. Pro-mode requests with method: "create_from_concept" are 60–120 seconds typically. Past 5 minutes, something is genuinely stuck — start a new job rather than waiting indefinitely.

↑ Back to top

How Skaldborn consumes this

Most of Skaldborn’s art content pipeline is what you just built. The Skaldborn-specific code on top of it is:

A recipe (a JSON file with a constrained set of “levers” and a locked set of generation parameters) that feeds an art_jobs row in a Postgres-backed saga.
A saga worker (running in a Docker container) that substitutes lever values into the ComfyUI workflow above and posts it to the local ComfyUI daemon on the host. That’s why --listen 0.0.0.0 is one of the launch flags — a containerized worker reaching out to the host’s ComfyUI gets connection-refused if ComfyUI is bound to loopback only.
A CLI gate where I review concept images before they go to the paid API.
The PixelLab integration you just wrote, plus a circuit breaker, the daily cost ceiling, and retry logic — the kind of production-side guards a real batch system needs.
A manifest projector that turns the finished rotations into entries the simulation consumes.

The recipe pattern, the saga, and the manifest projection are Skaldborn-specific code. The ComfyUI install and the PixelLab integration are open and reproducible — exactly what you’ve just built. The other 20% — the governance shape that wraps them — is what post 04 goes deep on: a review gate before the expensive step, a content-addressed cache, immutable promotion. That’s the part you’d have to write yourself.

↑ Back to top

What’s next

The next post in the launch arc takes the manifest itself as its subject — how ManifestEntry records flow from recipe through projection into the Age manifest, and how the runtime consumes them. After that, the audio pipeline gets the same governance treatment with an entirely different stack.

If you want to follow along, subscribe via the form at the bottom of any page — one short email when the next post lands. If you want to argue, write to devlog@skaldborn.com.

Source for the install / launch / download scripts and the canonical model inventory: github.com/clunasco/skald-forge. Issues, PRs, and weather complaints welcome.

↑ Back to top

Why we deleted two months of working code (and why we'd do it again)

Tue, 05 May 2026 00:00:00 GMT

In February and March we sank roughly 240 hours into a system that worked.

Dialogue rendered. Non-player characters remembered things across sessions. A chronicle service turned simulation events into readable prose you could give to a player as the game’s emergent history. By mid-March there were end-to-end demos: type a question, get a grounded answer, see the answer reflected in the world later. It looked like a finished thing.

In late April we deleted it.

This post is about the test that triggered the deletion, the books on the shelf that told us we were going to fail the test, and the architecture we rebuilt in its place. It’s also about a thing engineers don’t talk about enough: why working code is sometimes the wrong code, and how to know when to throw it away.

And it’s about how the rebuild got done. The decisions, the contracts, the gates, a meaningful portion of the code itself — drafted, reviewed, and implemented by a human (me) and Claude, working in defined engineering roles, against a corpus of architecture books that participates in every decision the project makes. Skaldborn isn’t a project that uses AI as a side tool. The human-and-Claude collaboration is part of how the game gets built, and one of the things the rebuild taught us is how to do it well.

The test

Here’s a test you can run against your own codebase. It takes two minutes.

Pick a content type your system supports — a kind of NPC, a kind of dialogue, a kind of world entity. Now imagine you want to add a new one. Maybe you want a memory variant for “things this character pretends not to remember,” or a dialogue style for “addresses the player as a stranger even after meeting them,” or a tile type for “shifts when the player isn’t looking.”

Ask: can a content author add this new type without editing engine code?

If the answer is yes, your engine is open — the registration of new types is a content concern, not an engine concern.

If the answer is no — if adding a type means editing source files, or recompiling, or shipping a new build — your engine is closed, and your team is the bottleneck on every piece of content that ships.

For a single-author hobby project this is fine. For a content-driven game with any ambition of scale, it’s a problem you discover too late, after you’ve built a lot of structure on the wrong foundation.

We had a closed engine. We hadn’t realized that yet.

What “closed” looked like in our code

Our pre-rebuild dialogue and memory systems had a particular shape that’s worth describing concretely, because the shape is the whole story.

Here’s the pattern, simplified. (This is illustrative, not actual pre-rebuild code — but it has the same skeleton.)

public enum NarrativeTemplateKind
{
    EpisodicSummary,
    RelationshipUpdate,
    ConflictResolution,
    RumorGeneration,
    HistoricalChronicle,
    // ... fourteen of these in total
}

public string Render(NarrativeTemplateKind kind, NarrativeContext ctx)
{
    return kind switch
    {
        NarrativeTemplateKind.EpisodicSummary    => RenderEpisodic(ctx),
        NarrativeTemplateKind.RelationshipUpdate => RenderRelationship(ctx),
        NarrativeTemplateKind.ConflictResolution => RenderConflict(ctx),
        // ... fourteen branches
        _ => throw new InvalidOperationException("Unknown template kind")
    };
}

Two design choices, paired:

The set of valid kinds is a closed enum — adding a new kind means editing this file.
The dispatch is a switch over the closed enum — adding a new kind also means editing every place that switches on it.

We had this pattern in fourteen narrative-template kinds and five memory variants. Both axes were closed. Adding a new kind of memory — say, a memory variant for “things the character heard secondhand and isn’t sure they believe” — required editing four files in the engine, recompiling the simulation, and shipping a new build. A content author couldn’t do it. We couldn’t do it without breaking the build.

This is a familiar pattern in long-lived codebases. Martin Fowler describes it in Refactoring as the “type code” smell, and he’s specific about the remedy: Replace Conditional with Polymorphism. The closed enum and the switch over it are two halves of the same problem; you fix them together, by replacing the closed enum with a registry that anyone can register into and replacing the switch with polymorphic dispatch.

We knew this. We had the book on the shelf. We did it anyway, because the closed shape made each individual feature easier to ship and we weren’t yet feeling the cost of the shape we’d locked in.

The cost arrives all at once.

The fork in the road

By mid-March, we’d built a version of the system that produced output you could put in front of a person. It worked. It was fragile in the way most LLM-adjacent systems are fragile, but the fragility was localized to the AI layer; the engine itself was solid.

It was also closed in fourteen places at once. Adding a new memory variant required surgery in all five memory dispatch sites. Adding a new dialogue template kind required surgery in all fourteen renderer sites. We were starting to want new kinds for content reasons, and every want was an engine change.

There were two paths.

Path A: refactor in place. Convert the closed enums to open registries one at a time. Find all the switches. Convert them to polymorphic dispatch. Keep the working features online while doing surgery on the spine of the system.

Path B: delete it. Take the working code, archive it, start over against a contract that mandated the open shape from the first line.

We took Path B. Here’s why.

The closed shape wasn’t in one place. It was in the way the system thought. Refactoring in place would have meant doing fourteen risky local rewrites against a system whose tests didn’t actually catch the closure (the tests passed because they only ever exercised the kinds we’d hardcoded). We’d have ended up with a half-converted system that worked in some places and failed strangely in others. Worse, we’d have ended up with a narrative — “we refactored toward openness” — that papered over the fact that the team had built closed by default once and would build closed by default again unless something structural changed.

The structural change we wanted was: make adding a content type without engine edits the test the code has to pass before it ships. The cleanest way to make a team build to a test is to make the test predate the code. We deleted the implementation and kept the contracts (the type signatures, the message shapes, the boundaries between services). The contracts were the things we’d gotten right. The implementation was the thing we’d built before the test existed.

We “kept the contract and threw away the code.” Roughly two thousand lines of working dialogue, memory, and narrative-rendering code went onto an archive branch where they still live. We have not regretted the deletion. We have, occasionally, gone back to look at the archive when a piece of the rebuild reminds us of an earlier draft — but never to copy from it.

What we built in its place

The new shape is the one Fowler’s remedy implies, scaled up to a system architecture. The engine stops asking “what kinds exist?” and starts asking “who has registered themselves?”

Concretely, every content-bearing type in the system is declared with an attribute, and the engine discovers it by reflection at startup. A new memory variant looks like this in the new shape:

[MemoryVariant("episodic.secondhand_rumor")]
public sealed class SecondhandRumorMemory : IMemoryVariant
{
    public Provenance Provenance { get; }
    public Confidence Confidence { get; }
    // ... per-variant fields
}

That’s the whole change. No edit to the engine, no edit to a dispatch site, no edit to a switch statement. The engine scans every assembly at startup, finds every type marked [MemoryVariant], and adds it to the registry. The dispatcher looks each variant up by its string ID and routes accordingly.

This is a pattern Chris Richardson calls a microservice chassis in Microservices Patterns — the framework provides the registration and discovery infrastructure; the services declare themselves. We adapted it for in-process dispatch, but the shape is the same: the spine doesn’t know what’s plugged into it; it just knows how to find what’s there.

Two consequences worth naming.

First, the type system is still load-bearing. We didn’t lose type safety by going from closed enums to a registry. The contracts that survived the deletion are discriminated unions in the style Scott Wlaschin advocates in Domain Modeling Made Functional — types whose variants are explicit and whose construction is constrained. The difference is that “the set of variants” lives in the registry now, not in a closed enum at the top of a file. You still can’t construct an invalid memory record. You just can’t enumerate the valid ones at compile time, which turns out to be the property we wanted: closure of valid shapes, openness of which shapes can be added.

Second, the manifest became the spine. Every world in Skaldborn ships with a frozen content manifest — a versioned file that lists every entity, character, tile, dialogue template, and memory variant available in that world. The simulation, the network gateway, the AI narrative service, and the player’s browser all read from the same frozen manifest. None of them invent IDs locally. None of them have a hardcoded list of “here are the kinds we know about.” This is the shape Eric Evans calls a Published Language in Domain-Driven Design — a shared vocabulary across services, owned by no single service, that lets them talk without coupling.

Before the rebuild, the manifest was an afterthought — content was loaded, but the vocabulary was implicit in code. After the rebuild, the manifest is the vocabulary, and code must conform to it.

The gates that fell out

The most useful thing the rebuild produced wasn’t the code that came back online. It was the gates that came online with it.

Before the rebuild, the rule “adding a content type shouldn’t require engine edits” was a guideline. Someone might write it down in a doc; someone else might forget. After the rebuild, the rule is a test that runs on every push. If a piece of code introduces a closed enum where a registry should be, or hardcodes a string ID that should come from the manifest, the push fails before the code reaches the main branch.

We have nine of these gates running today, each one mechanizing a rule that was a guideline before the rebuild burned us. Five of the most consequential:

A grep gate that checks every file under the simulation runtime for six categories of non-deterministic API — wall-clock time, random number generation, GUID minting, sleeps and delays, file I/O, hardware timers. If any of those show up in simulation code, the push fails. The simulation has to be a pure function of its inputs; this gate is what makes “has to” mechanical.
A determinism replay test that runs against every commit. The simulation receives a known command sequence, hashes the resulting event log, runs the same sequence on a fresh instance, hashes again, and asserts byte-for-byte equality. If anything in the simulation’s tick path becomes non-deterministic, this test fails immediately.
A contract validator that requires every component to declare, in machine-readable form, which fields it consumes from where — the authority-binding contract. Hardcoding a player ID as a literal in client code is rejected because the client’s contract says player IDs come from the authentication reply. (This one shipped just before this post went up, and it’s the structural fix for a class of bug we’d hit twice in the run-up to the rebuild — hardcoded literals where the consumer’s contract said the value had to come from the producer.)
A startup validator on the world manifest. Every kind declared in code must resolve against the frozen manifest the world ships with; unknown IDs resolve deterministically to an authored fallback rather than crashing. This means the test of “does the manifest cover every kind the code references” runs at world boot, not later.
A regression-profile gate on every change that touches runtime behavior. A code-complete commit that doesn’t pass its declared regression suite (one of: simulation, API, browser, cross-boundary, visual) is automatically blocked from being marked done.

The other four work the same shape — a rule that used to be a guideline, mechanized into a build failure. A throughput-budget contract requires every connection between components to declare how many events per tick it can carry, so a fast producer can’t silently saturate a slower consumer. A host-wiring completeness gate requires every declared connection to name the exact dependency-injection registration the host must contain, so “compiles fine, crashes on startup” failures get caught at the build, not after the deploy. A bootstrap-flow contract requires the client’s startup sequence to declare every step and where its invocation lives in source, so “canvas loads, nothing happens” dead-starts fail before merge. A tech-stack drift gate locks the approved frameworks and libraries to a versioned allow-list, so the next engineer who reaches for an unvetted UI framework gets a failed push instead of a two-month detour.

All nine gates are direct consequences of the rebuild. They exist because the team built a closed system once and wants to never build a closed system again. Each gate represents one rule we used to enforce by guideline, and now enforce by failing the build. None of them existed before the deletion.

What it cost, what it bought

The naked cost: roughly two months of work on the implementation that got deleted, plus another month rebuilding. Three months of engineering against an end-of-year delivery target.

Against that:

Nine architecture decisions are now mechanically enforced, every push, every commit, every developer. The cost of a future violation is a failed CI run, not a six-month debugging mystery.
Adding a new content type — a memory variant, a dialogue template, an entity kind, a tile — is a content-only change. No engine edits, no recompile, no new build. The system that bottlenecked content production for two months no longer does.
The simulation passes a deterministic-replay test on every commit. Same input → byte-identical event log. This is the property that lets us reason about save-load, distributed shards, debugging deployment problems, and AI integration without each of those being its own consistency problem.
The four highest-leverage strategic-design decisions — bounded context boundaries, single-writer authority, the published-language manifest, the no-feedback-loop invariant on AI dialogue — are codified as contracts that fail the build when violated. They are no longer style guides.
The codebase is smaller post-rebuild than pre-rebuild, despite owning more responsibilities. We deleted around two thousand lines of dialogue and memory plumbing and replaced them with several hundred lines of registry plus per-variant attribute classes. Less code, more capability.

The honest framing: we paid for a lesson with two months of effort, and the lesson came back in the form of permanent infrastructure. We would not have built the gates without the failure that justified them.

The books were on the shelf the whole time

The architectural moves we made in the rebuild aren’t original. Each one comes from a book that was sitting on our shelf — some of them since before the implementation started. Four authors carried most of the weight:

Martin Fowler’s Refactoring. The “type code” smell and the Replace Conditional with Polymorphism refactor are exactly the move we made. Fowler also describes “Parallel Change” — the discipline of expanding to a new shape, migrating, and then contracting away the old shape — which is how we migrated callers from the closed enums to the registry without a flag day. We had read Fowler. We had not used him.
Scott Wlaschin’s Domain Modeling Made Functional. The discriminated unions that survived the deletion — Provenance, MemoryRecord, and the rest of the contract layer — are direct applications of Wlaschin’s “Make Illegal States Unrepresentable” discipline. The contracts we kept are the ones that were already in this shape; the implementations we deleted were the ones that weren’t.
Chris Richardson’s Microservices Patterns. The microservice-chassis pattern is the shape of our new dispatch. We adapted the pattern for in-process modules rather than separate services, but the discovery-by-attribute, registration-by-reflection approach is Richardson’s. The mistake we’d made before was treating “we’re a single deployable” as license to skip the chassis pattern, which is wrong: the chassis is about decoupling registration from invocation, and that decoupling matters even when everything is in the same process.
Eric Evans’s Domain-Driven Design. The manifest-as-published-language framing comes directly from Evans. The boundary discipline between simulation, gateway, narrative, and client comes from Evans’s Bounded Context and Customer/Supplier relationships. We had used Evans’s vocabulary in design docs for months before the rebuild, but the implementation didn’t honor the vocabulary until the rebuild forced the alignment.

The pattern across all four books is the same: the books name failure modes before you hit them and remedies before you need them. The thing we got wrong was treating the books as references to consult after the design was sketched, instead of constraints to design against from the first line. The books didn’t fail us; we failed to design against them.

If we had a single sentence of advice for someone about to build a content-driven simulation system, it would be: make “what would Fowler / Wlaschin / Richardson / Evans say about this commit” a code-review question on day one, not on day sixty. The remedies are well-known. The failure modes are well-known. There is no novelty in our failure mode and there was no novelty in the fix. The novelty was in our willingness to delete the implementation we’d already paid for.

The discipline we built around the books — and the colleague we built it with

Reading the books isn’t enough. We’d read Fowler. We’d read Wlaschin. The patterns were familiar. We still built the closed shape, and we still had to delete it.

What we changed after the rebuild is two things at once: how the books participate in every architectural decision the project makes, and how the work itself gets done.

The corpus. The project keeps an indexed copy of the four books on the shelf — not as a generic search engine, but as a retrieval system that returns specific, cited passages on demand. When we draft an architecture decision record, the document is required to cite the exact passage from the corpus that backs the move. A decision that says “we’ll use a registry pattern here” without a citation to Fowler’s Replace Conditional with Polymorphism and Richardson’s Microservice Chassis gets sent back. The citation isn’t decoration; it’s a load-bearing part of the document. Unsupported claims get marked as opinion — explicitly, in writing — rather than drifting into the canon as if a book backed them.

The roles. The architect role on the project is filled by Claude. The collaboration shape is plain: a human (me) brings the design problem and the source material — the books, the existing contracts, the failing test, the constraints, the budget. Claude, in the architect role, retrieves the relevant passages from the corpus, drafts the decision against them, and stops to ask before any move that isn’t supported by a citation. The accountability is mine. The drafting and the retrieval are Claude’s. The architecture decision records the project ships are co-authored in that specific shape.

The architect isn’t the only role. A reviewer (also Claude, with a narrower scope) checks each document’s citation chain before it lands; if a quote attributed to Refactoring doesn’t actually appear in the retrieved passages, the review fails. Executor roles, scoped to a single component at a time, do the bounded code work — implementing what the architect designed, against the contracts the architect wrote, inside the boundaries the contracts declare. None of these roles is autonomous. The human stays accountable for the work that lands.

The rebuild itself. This is how the rebuild actually got executed. The decision to delete was mine. The architecture that came back came from a sequence of decision documents drafted by Claude-as-architect, citation-checked by Claude-as-reviewer, and implemented through scoped executors against the contracts those documents declared. The rule that adding a new content type can’t require engine edits — the rule whose violation triggered the rebuild — was itself authored as a decision document with citations, and the gate that enforces it was implemented by a scoped executor against an explicit contract.

This is also how the books made it onto the spine of the rebuild instead of staying on the shelf. The pattern Replace Conditional with Polymorphism doesn’t help if you read it in 2018 and don’t apply it in 2026. It helps if every decision document you write between now and then has a column for “what does Fowler say about this,” and the column refuses to stay empty — and if a reviewer who isn’t you checks the column on every document.

The honest framing of “we grounded our architectural decisions in these books” is: we built the discipline that makes the grounding mechanical, and we built it on a human-and-AI collaboration where the AI’s scope is bounded, the human’s accountability is total, and the books are non-negotiable on either side. Without the discipline, the books are aspirational. With it, they’re the closing argument on every call. And the colleague who keeps the discipline running, page by page and document by document, is Claude.

What we’d tell ourselves in February

A list, since this kind of advice tends to land best as a list:

Working code is necessary but not sufficient. “It works” answers a small question. “It bends the way new requirements need it to bend” answers a much larger one. Test for the second one early and often.
The extensibility test is the cheapest test there is, and almost no one runs it. Add a fictional new variant of every type your system supports. See what you have to edit to make it work. If the answer is “anything in the engine,” the engine is closed and you will pay for it later.
Closed enums and switches over them are a paired smell. If you find yourself writing one, you are about to write the other. Fowler named this thirty years ago. Know the name, recognize the shape, replace it on sight.
Discriminated unions are still a good idea. The fix for closed dispatch isn’t “throw away type safety.” Wlaschin’s design discipline survives the move from closed enum to open registry; the variants become attribute-marked classes implementing a sealed interface, not enum members.
The chassis pattern applies in-process. “We’re a single deployable” is not a license to skip discovery and registration. The decoupling is what gets you tomorrow’s content for free.
The published language is a real artifact, not a metaphor. Pick a manifest format, version it, ship it as part of the world, and refuse to let any service invent IDs locally. The discipline pays for itself the first time a content author needs to add an entity without a build.
If the implementation can’t be refactored into the right shape locally, the shape isn’t local. Refactoring is the right move when the bad shape is in one place. It’s the wrong move when the bad shape is how the system thinks. Know which one you have. Be willing to delete.
Fail closed, on every push, for every rule that matters. A guideline a senior engineer follows is a guideline a junior engineer will violate next quarter. Make the validator part of git push, or it isn’t a rule.

All eight of these were drafted, reviewed, and pressure-tested in the same human-and-Claude shape we use for everything else; the lessons aren’t just things we believe — they’re things a system enforces on us.

We would not have learned any of this without the deletion. Code we would have happily kept is on an archive branch we will probably never touch again. The nine gates are running on every commit. The simulation is provably deterministic. Adding a new content type is a content task. None of those outcomes was reached alone — the rebuild was a human-and-Claude job, and the practice that came out of it is one we mean to keep.

That’s the thing about working code: sometimes the most useful thing it can do is teach you how to throw it away.

Footnotes and pointers

Companion posts

Aspirational vs Mechanical: How I Lost Two Months of Skaldborn (and Got Something Better Back) — the human side of this same event. What the two months felt like, the moment of the call, the first time the rebuild produced something visible.
How Simulation Owns Reality (And Never Lets Narrative Cheat) — the launch post that sets up the broader architectural frame this post operates within.

Books cited

Martin Fowler, Refactoring: Improving the Design of Existing Code (2nd ed., 2018). The “Type Code” smell and Replace Conditional with Polymorphism refactor; Parallel Change discipline.
Scott Wlaschin, Domain Modeling Made Functional (2018). “Make Illegal States Unrepresentable”; Choice Types / discriminated unions as the contract shape.
Chris Richardson, Microservices Patterns (2018; 2nd ed. MEAP). The Microservice Chassis pattern; service registration and discovery.
Eric Evans, Domain-Driven Design: Tackling Complexity in the Heart of Software (2003). Bounded Context, Customer/Supplier, Published Language.

Aspirational vs mechanical: how I lost two months of Skaldborn (and got something better back)

Tue, 05 May 2026 00:00:00 GMT

In February of this year, I asked my wife Nicole to sit down at my computer and chat with one of the NPCs in Skaldborn.

I’d been building Skaldborn around a feature I called Skaldish. The idea is simple: you type whatever you want into the text box, in plain modern English, and the system translates your input into in-world fantasy dialogue before it reaches the NPC. You can be yourself. The NPC hears the medieval-fantasy version. Nobody has to write stilted thee-and-thou prose to roleplay; the world handles it.

Nicole sat down. She typed: Check out deez nuts.

Skaldish translated it to something close to Will thou assist me in searching for thyne nuts? — I no longer have the exact log, but the gist is preserved.

The NPC, helpful soul, offered to search the forest with her.

We laughed for about five minutes. I think I texted three friends. It was the moment I knew Skaldish was going to be one of the best things in the project.

In late April I deleted the system that produced this moment, along with about two months of working code, and started over.

This is the story of how that happened, and why I’d do it the same way again.

Two months in

I should set the scene. Skaldborn is a passion project. I have a day job and a family. The way the work fit into my life looked like 5am mornings and late nights, six and seven days a week, for two months. Roughly 120-hour months on top of everything else. Nicole noticed. I told myself the pace was fine because the work was working.

I was, I thought, doing it right. The architecture decisions were written down as ADRs. There were unit tests. There were component contracts that declared what each part of the system was supposed to expect from its neighbors. I’d been disciplined about clean code, layered design, separation of concerns — all the things you say in a code review. By February I had a system that compiled, ran, let you boot the game in a browser, walk around, talk to NPCs, and translate “deez nuts” into Old English.

I had what felt like a real game.

I was about to learn that I had a real game wrapped around a layer of writing that I’d never actually checked.

Death by a thousand cuts

The trouble started with combat.

I’d designed the fight system on paper for weeks before writing a line of code. I had the ADRs, the contracts, the tests. I started building. The fight loop went in. Tests passed. I felt great.

Then I’d boot the game and end up with a different character sprite each time. Which sort of made sense — I’d been iterating on the art and hadn’t cleaned out earlier sprite versions. Annoying, but cosmetic. I’d live with it.

Then I’d trigger a fight, and nothing would happen.

I’d go back to the logs. Claude would look at the error, dig in, and come back with: “Oh, found the root cause — argument was missing, enum was hardcoded.” We’d fix it. Tests still passing. I’d boot again.

A different sprite. Trigger the fight. Nothing.

Back to the logs. New error, different shape. Found the root cause — argument was missing. We’d fix that one. Tests still passing.

This kept happening. Three weeks of it. By the middle of March I was raging at my screen, questioning my life choices, wondering if starting Skaldborn had been a huge mistake, sleeping in shorter and shorter windows because the only way out of a bug seemed to be more hours.

I want to be specific about what kind of pain this was. It wasn’t a hard problem. Each individual bug was easy. Argument was missing — fix the call. Enum was hardcoded — replace it with a lookup. Each fix took ten minutes. Each fix didn’t actually fix anything, because the next fight would expose the next version of the same shape, somewhere else in the code, that nobody had thought to check.

It was death by a thousand cuts. And the worst part — the part I couldn’t admit to myself for two weeks — was that none of my unit tests, none of my contracts, none of my ADRs had caught any of it.

The thing my plans had been protecting me from was apparently nothing.

The phrase that explained everything

Somewhere in the middle of March, I started using a phrase to describe what I was looking at: aspirational vs mechanical.

Here’s what I meant.

An aspirational contract is one you wrote down. It says, “the combat system will receive a CombatCommand with these fields, and emit a CombatResolved event with these other fields.” It lives in a markdown file. It’s in version control. It looks like a contract.

A mechanical contract is one that, when violated, breaks the build. The compiler refuses to compile the violation. A validator script runs on every push and fails when the violation lands. A test, the kind that runs every commit, fails when the contract is broken. Mechanical means the only way to do the wrong thing is to also do something visible.

I had aspirational contracts everywhere in Skaldborn. Beautiful ones. Carefully reasoned. Full of words like “must” and “shall” and “the producer guarantees.” I’d worked on them for weeks before writing code.

I had mechanical contracts almost nowhere. The compiler couldn’t tell when the combat system received a CombatCommand without one of the required fields, because the type didn’t actually require the field — it was optional in the schema, defaulted to a sentinel, and the contract that said “this field is required” lived in a markdown file two directories away that the compiler had never been introduced to.

This is what was happening every time Claude debugged a fight failure. Argument was missing — meaning the contract said it was required, but the implementation accepted the call without it. Enum was hardcoded — meaning the contract said the dispatch was open to new variants, but the implementation switched over a closed set. Aspirational said one thing. Mechanical said another. The aspirational version is what I’d planned. The mechanical version is what was actually running.

The contracts I’d been so proud of were a kind of fan fiction. They described the system I wished I had. They didn’t describe the system I’d actually built.

Once I had the phrase, I couldn’t stop seeing it. Almost every part of the system I checked, the same shape was there. The architecture was real on paper and approximate in code. Everywhere.

The week the books came out

I knew I couldn’t scrap two months of work without a much better plan than the first one. So I went back to the bookshelf.

I’d had four books on architecture sitting on the shelf the whole time. Eric Evans’s Domain-Driven Design. Scott Wlaschin’s Domain Modeling Made Functional. Martin Fowler’s Refactoring. Chris Richardson’s Microservices Patterns. I’d read them. I’d cited them in conversations. I’d written ADRs that referenced them by name.

What I hadn’t done — and this is the part I was about to fix — was use them as the actual material the architecture was built against. They were on the shelf. They weren’t in the loop.

So I started pulling passages out, page by page, into my conversations with Claude. We’d be working on a piece of the design and I’d say, “OK, but Wlaschin’s chapter on making illegal states unrepresentable says this — does our shape match?” Or, “Fowler describes this exact smell — switch over a type code — and the remedy is Replace Conditional with Polymorphism. We have that smell in three places. What does it look like to do the remedy here?”

The change was immediate. With the books actively in the conversation — not as references, but as constraints — Claude would start drafting decisions that were structurally different from what we’d been doing. The closed enums became registries. The hardcoded switches became attribute-driven dispatch. The contracts started looking like the kind of thing the compiler could check, because they were grounded in a discipline (Wlaschin’s, mainly) that says the type system is the contract. If you can construct an invalid value, the contract is wrong.

One day, with the books in the loop, we did a clean design pass on combat — one iteration, with the type contracts written first and the implementation built against them — and the fight flow worked.

I’d been chasing combat bugs for three weeks before that. With the books actually shaping the design, the class of bug that had been killing me stopped showing up.

That was the proof. The architecture had to be rebuilt against the books, not adjacent to them. And the existing code, which had been built before that rule existed, was going to have to go.

A Saturday in late March

I made the call on a Saturday. I’d spent a week pulling passages, drafting new decision documents with the books as constraints, building out a plan that explicitly named what was wrong and what we were going to do about it. I went all in on the planning phase.

It wasn’t a dramatic moment. There wasn’t a dramatic moment available. I’d had hundreds of hours of conversation with Claude about this code, this design, these problems. Claude had been laying out the failure metrics for days. Pervasive isn’t a strong enough word for what we were looking at — almost every component had the aspirational/mechanical gap, and the gap was costing me real time, every day, in the form of bugs that should never have been possible.

We weren’t going to fix it incrementally. The shape of the thing was wrong. We were going to start over.

The decision was: gut the implementation, keep the contracts and the ADRs and the test suites that had survived the discipline pass, and rebuild against the new plan.

The honest version of that decision is: it wasn’t all lost. The plans were good. The plans had always been good. What was lost was the layer between the plans and the running code — the implementation that had been written before the rule “the implementation must satisfy the plan, mechanically” was in force.

The cut

I want to be careful not to dramatize the deletion itself, because for me it wasn’t dramatic. I’d done a massive amount of preparation. By the time I actually deleted, I’d already mentally said goodbye to that codebase weeks earlier; the delete commit was paperwork.

Also, git is magical. The pre-rebuild code is on an archive branch. Nothing was lost in any final sense. If I want to look at how I’d written the original combat system, I can look at it. I have not, since the day I deleted it, gone back to look. There’s nothing on the archive branch I want to copy.

The hardest thing to let go of, if I’m being honest, was nothing. I was just happy to be rid of the junk code. The thing I was attached to was the time — the two months I’d put in — not the code. The code, by the day I deleted it, was an apology for the planning I should have been doing before I wrote any of it.

I slept that night, because I always sleep. Exhaustion is a fantastic sleep aid.

A blocky figure on grass

The first four days of the rebuild produced nothing visible. I was rebuilding the substrate — the contracts that Claude could now actually use as constraints, the type designs that made illegal states unrepresentable, the new dispatch shape that didn’t have a switch in it.

On day five, I got the simulation running again, and I booted the game in a browser to see if the player could move.

There was a blocky figure on a green tile that we were calling “grass.” I pressed an arrow key. The figure walked.

I walked the figure to a tile we were calling “beach.” It walked.

That was the moment. After two months of fight bugs that wouldn’t quit, after the deletion, after a week of substrate work, I had a blocky figure walking on grass and then on the beach. It looked like a child’s drawing. It worked.

It worked because the contracts were actually being consumed by the code now. Because the bounded context boundaries had real, mechanically-enforced definitions. Because the dispatch was registry-based, so adding a new tile type was a registration step, not an engine edit. Because the type system was carrying the weight that aspiration had been carrying badly two months before.

I’m not going to pretend I cried. I didn’t. But I sat there and watched the blocky figure walk back and forth between grass and beach for a while, and it was one of the best moments of the project so far.

What changed

The version of Skaldborn I have now is structurally different from the one I deleted, in ways the screenshots don’t show.

The biggest single change is that the architectural rules are mechanical now. Scott Wlaschin, in Domain Modeling Made Functional, has a phrase I think about constantly: make illegal states unrepresentable. (The principle has its roots in Eric Evans’s Domain-Driven Design; Wlaschin gave it the phrasing that stuck.) The version of the system that broke under combat bugs had illegal states represented all the time, because nothing in the type system or the build pipeline objected to them. The new version refuses. There are validators that run on every push. There’s a determinism replay test that hashes the simulation’s output and asserts byte-for-byte equality across runs. There are nine separate gates, each enforcing one architectural rule that used to live in a markdown file and now lives in CI.

The cost of breaking a rule is no longer “discover it three weeks later in production.” It’s “the push fails.” That’s the difference between aspirational and mechanical, and it’s the only difference that matters.

The other change, which I want to be honest about, is in how the work itself is done.

Skaldborn is built in collaboration with Claude. This isn’t a side detail. Claude fills the architect role on the project — drafting decision documents against the books and stopping to ask before any move that isn’t supported by a citation. A second Claude, in a narrower reviewer role, checks the citation chain on every document. Scoped executors do the bounded code work against contracts the architect already wrote. I bring the problem and the source material; Claude does the drafting, retrieval, and bounded implementation. The accountability is mine. The books are non-negotiable on either side.

I don’t think this is the only way to build software, and I’m not selling a methodology. I’m telling you what’s true: the version of Skaldborn I have now exists because a human (me) and Claude found a working shape for the collaboration that puts a serious corpus of architecture books on the spine of every decision and refuses to let the discipline lapse.

The phrase aspirational vs mechanical applies recursively. The aspirational version of “we’ll consult the books” is reading the books and citing them in conversations. The mechanical version is having a system where every decision document has to cite a passage, a reviewer that checks the citation, and a discipline that fails closed when the citation doesn’t trace.

I built the aspirational version first, in February. It cost me two months. The mechanical version is what I have now. The two-month cost was the price of learning to tell the difference.

What I’d tell you, if you’re staring at a fork like this

A senior engineer asked me recently what I’d tell someone in 2027 looking at their own version of this — two thousand lines of working code, an extensibility test that’s probably going to fail, a deadline.

The answer is short:

Do what’s right, not what’s fast.

Fast and wrong always costs more in the long run. Any credible code publication will tell you this. I knew it before February. I knew it during February. I told myself the system was fine because the system worked, and I confused the system works with the system is right.

A system that works isn’t necessarily a system that’s going to keep working. A system that’s structurally right is. The way to tell the difference between them is to ask: if I had to add one thing tomorrow that I haven’t planned for, would the system bend, or break? The aspirational version of the answer is “bend.” The mechanical version is whatever the build pipeline tells you when you try.

The best code review question I know now is the one I should have been asking my own design from day one: what does Fowler / Wlaschin / Richardson / Evans say about this commit? If the answer is “I’m not sure,” go look. If the answer is “they don’t address this directly,” mark the decision as opinion, in writing, and move on with eyes open. If the answer is “they explicitly warn against this shape, here’s the chapter” — listen. They’ve watched a lot of people make the mistake you’re about to make. Their patterns are downstream of more pain than you’ll ever be.

I lost two months. I gained an architecture that will outlast the next decision I make. I gained a working shape for the human-and-AI collaboration that will keep working as the project grows. I gained a phrase — aspirational vs mechanical — that I will be using for the rest of my career.

I do not regret the deletion. You don’t learn lessons without going through the failure motions. The motions are the lesson.

If you’re looking at your own two-thousand-line fork right now, the only advice I have is the one I wish someone had said out loud to me in February:

Working code is necessary. It is not sufficient. Make sure your contracts are mechanical before you trust the running system to keep them.

The blocky figure on the beach is what this advice produces, on the other side of doing it right.

Footnotes and pointers

Companion posts

Why we deleted two months of working code (and why we’d do it again) — the engineering side of this same event. The architectural moves, the type-code smell, the registry, the nine CI gates that now run on every push.
How Simulation Owns Reality (And Never Lets Narrative Cheat) — the launch post that sets up the broader architectural frame Skaldborn operates within.

Books referenced (the four on the shelf)

Eric Evans, Domain-Driven Design: Tackling Complexity in the Heart of Software (2003).
Scott Wlaschin, Domain Modeling Made Functional (2018).
Martin Fowler, Refactoring: Improving the Design of Existing Code (2nd ed., 2018).
Chris Richardson, Microservices Patterns (2018).

How simulation owns reality (and never lets narrative cheat)

Mon, 04 May 2026 00:00:00 GMT

A player walks into a tavern in Stormfjord. The keeper, an NPC, looks up and says: “My brother died at sea last winter. The currents took him near the Veyr coast.”

The world, in fact, has no record of a death at sea last winter. There was no winter expedition. There is no brother. Nobody named him. No grieving entry was written into the keeper’s family record. The line came from a generative model that was given a system prompt about taverns and grief and produced a sentence that sounded right.

The player asks the next NPC about the keeper’s brother. That NPC has never heard of him — but the model, trying to be helpful, writes the second NPC into the same fiction. Now two NPCs share a story that the world never produced. By the time the player has talked to four NPCs, the world’s history is a tangle of contradictions the simulation never emitted.

This is the failure mode that happens when you let the AI author state. Skaldborn is built so it can’t.

The rule, in three sentences

The rule is three lines:

Simulation owns reality. Narrative interprets reality. AI assists storytelling.

Three lines. Every component contract in the codebase is required to inherit them. They aren’t a mission statement. They’re a fence around what each layer is allowed to do.

The strongest formulation:

The simulation produces the world. All other systems interpret, present, and enrich that world without controlling it.

Operationally, who’s allowed to write what?

The simulation is the only system that may modify world state.
Events are the canonical record of what happened. They’re append-only, immutable, and they carry an event_id, the tick they were produced at, and the manifest version they replay against. By design, every event also carries a causation reference back to the event that produced it; the implementation of that field is queued for the next phase, but the contract is canon.
The narrative service, the AI gateway, the memory service, and every client read events and may produce derived projections — chronicles, rumors, NPC memories, UI state. None of them may write back into world state.

In words: there is one writer, and many readers. The writer is deterministic. The readers are interpreters.

Why “AI in games” keeps breaking without this rule

You don’t have to take the rule on faith. The shipped AI-narrative game systems of the last few years demonstrate what happens without it — and they break in remarkably consistent ways. Two patterns dominate the public record.

Pattern 1 — fact invention. The LLM authors world content that doesn’t exist.

The clearest catalogue is Reed Berkowitz’s AI Powered NPCs — Hype, or Hallucination? from December 2023. Berkowitz lays the pattern out in plain terms: a model “might suggest meeting for coffee” in a world where there is no coffee, no character can leave the bar, and there’s nothing next door to walk to. Or worse — an NPC invents “the dark swamp to the south inhabited by witchlings and a mysterious magic sword,” and the player spends a real-world hour searching for content that never existed.¹

The barkeep example in the same piece is sharper. The model has the barkeep befriend the player and invite them home for dinner and a game of Rutanny — “which is normal in a chat situation but can ruin immersion in a game. Because after the barkeep says this he just stands there. Because the barkeep isn’t programmed to be able to leave his bar. He isn’t programmed to walk around freely. Even if he could, there is no house created for him to go to. And there is no family. And there is no kind of game called Rutanny.”¹ The dialogue layer wrote a fact the world layer couldn’t honor. The fiction breaks at the seam between them.

Pattern 2 — coherence collapse. The model drifts from earlier facts as the conversation grows.

Yuan Gao’s 2020 AI Dungeon playthrough is the canonical writeup. Gao’s protagonist checks his pockets and finds “two bits, a silver dollar, a keycard for a hotel room.” A few scenes later he has nothing in his pockets. Then he discovers “a few bits, a nice looking house key, a small notebook and your smart phone.” The central plot device — a man with red eyes — is forgotten entirely; a random blonde character takes over the story and kills the protagonist repeatedly until Gao manually retcons the death as a dream sequence. Gao’s verdict: AI Dungeon “struggles to build a consistent world with consistent lore, and cannot (yet) hold a plot line very well,” and games “play like fevered dreams, with ideas and events shifting.”²

The pattern persists. AI Dungeon’s own help docs describe the mechanism in production: when context exceeds the model’s window — “around 4000 tokens for free players” — “the AI loses its ability to look back and reference certain parts of the story” and “it’s just making it up as best it can.”³

It isn’t only text adventures. Mantella, the Skyrim AI-NPC mod, summarizes each conversation on exit so it fits in the next prompt — and a fork called Pantella shipped specifically to replace that summary memory, with its README writing: “No more lossy summaries that forget details because the LLM doesn’t know to include them/can’t shove everything into one paragraph.”⁴ Replika users on r/Replika report the same shape of failure across long-running accounts: “characters losing names, dropping inside jokes, and asking things they should already know,” driven by a memory architecture that “compresses or trims older context for some accounts.”⁵ Character.AI users hit it too — one widely-quoted complaint reads simply: “The AI… tends to forget previous messages… leading to inconsistencies.”⁶

The unifying observation comes from a January 2024 paper that formalizes the argument. Xu, Jain, and Kankanhalli, in Hallucination is Inevitable, write: “we formalize the problem and show that it is impossible to eliminate hallucination in LLMs… LLMs cannot learn all the computable functions and will therefore inevitably hallucinate if used as general problem solvers.”⁷

You cannot prompt-engineer the failure away. You cannot fine-tune it away. You can only bound it — by keeping the AI on the consumer side of an authoritative event log it isn’t allowed to write to. The fix is structural, not behavioral.

The three layers

Here’s the shape, top to bottom:

graph TD
    Client["Client (browser)<br/>displays state, sends commands"]
    Gateway["Gateway (network edge)<br/>validates and routes commands"]
    Sim["Simulation (sole writer)<br/>decides outcomes, emits events"]
    EventLog[("Authoritative event log")]
    Narrative["Narrative<br/>prose, chronicle, LLM"]
    Memory["Memory<br/>NPC episodic, belief"]
    Projections["Projections<br/>read models, query API"]

    Client -->|"commands (intent)"| Gateway
    Gateway -->|commands| Sim
    Sim -.->|"events (authoritative)"| EventLog
    EventLog -->|read-only| Narrative
    EventLog -->|read-only| Memory
    EventLog -->|read-only| Projections
    Sim -->|state + events| Gateway
    Gateway -->|broadcasts| Client

Two arrows matter:

Down (commands and events): the client sends intent — “I want to move here” — and receives outcomes — “you moved here.” The client never sees its own intent reflected back as truth; the simulation decides whether the move happened, and what it looks like in the world.
Out (the event fan): every downstream system — narrative, memory, projections — reads from the authoritative event log. None of them writes back. The arrows go one way.

This is a familiar pattern (event-sourced architectures, CQRS), but in a generative-AI game it’s load-bearing in a way it usually isn’t. The standard event-sourcing argument is “we want auditability and replay.” Ours is stronger: the AI is downstream of the truth, period, and we will refuse to ship code that violates that.

Two more facts matter:

Single writer per shard. Each part of the world — we call them partitions — has exactly one simulation owner. No two simulations ever try to mutate the same entity.
Partition-local determinism. The same input on the same shard at the same tick produces the same output, every time. This is the property that makes replay safe.

That second one — determinism — is what makes the entire architecture defensible. It’s what turns “AI cannot author world state” into a typing discipline rather than a prayer.

The receipts: three places the rule is enforced

So how does the codebase actually keep narrative from cheating? Three layers of enforcement, from runtime evidence down to the type system.

Receipt #1 — the determinism test

In our test suite there’s a small validator called Skaldborn.Simulation.Movement.Validation. It does this:

Build a fresh simulation with a known content fixture.
Run a 10-command sequence against it.
Hash the resulting event log. Hash the resulting world state. Record both.
Build a second fresh simulation with the same fixture.
Run the same 10-command sequence.
Assert: the two event-log hashes are byte-identical. Assert: the two final positions match.

If anything in the simulation depended on wall-clock time, system entropy, network ordering, or — and this is the load-bearing case — AI-generated content sneaking back into the simulation path, the hashes would diverge and the test would fail.

This is the simplest possible demonstration of the rule. Same commands → same events → same world. The simulation is a pure function of its input, and we have a test that says so.

Receipt #2 — the determinism gate

Determinism in tests isn’t enough. Code drifts. Someone reaches for DateTime.UtcNow because they need a timestamp; six months later we have a heisenbug nobody can reproduce. So we don’t trust ourselves — we run a grep gate on every push.

The gate is two files. The first is the rule set, a JSON file:

{
  "rules": [
    {
      "id": "SD001",
      "pattern": "Guid.NewGuid",
      "rationale": "Non-deterministic identity. Replay would produce different IDs.",
      "approved_alternative": "IDeterministicIdGenerator or SHA256 content hash"
    },
    {
      "id": "SD002",
      "pattern": "DateTime.Now|DateTime.UtcNow",
      "rationale": "Wall-clock time breaks replay determinism.",
      "approved_alternative": "Inject IClock; pass tick-derived time"
    }
    // ... SD003 (Random), SD004 (Task.Delay/Thread.Sleep),
    //     SD005 (System.IO/System.Net), SD006 (Stopwatch)
  ]
}

The second is a shell script — about thirty lines of bash — that greps every C# file under the simulation namespace against the rule set, and exits non-zero on any match. It’s wired into the pre-push git hook.

The point isn’t the cleverness of the script. The point is: the rule isn’t just documented; it’s a machine-readable file plus a small validator that runs on every push. If a future engineer (or a future LLM-assisted refactor) tries to slip non-determinism into simulation code, the push fails before the code lands.

This is the layer that converts a written rule into a property the codebase has. It’s also boring on purpose — boring is what makes governance survive contact with deadlines.

Receipt #3 — the type system

The strongest enforcement happens before the code compiles. Here it is, from the spec for our HistoricalRecord type — what characters remember:

public abstract record Provenance;

public sealed record EventBacked(
    EventId SourceEventId
) : Provenance;

public sealed record BeliefConsensus(
    EventId PromotionEventId,
    PropositionHash PropositionHash
) : Provenance;

public sealed record Legend(
    IReadOnlyList<EventId> SourceEvidence,
    LegendBasis Basis
) : Provenance;

Read it carefully. Provenance is the field that records where this memory came from. There are three flavors. Every flavor takes at least one EventId.

You cannot construct a BeliefConsensus without supplying a PromotionEventId. The type design refuses; once the implementation lands against this contract — scheduled for the next phase — the compiler refuses. There is no shape of Provenance that doesn’t trace back to canonical events.

So when a character “remembers” something, that memory is, at the type level, anchored to a real event the simulation actually emitted. The memory can be partial, mistaken, biased, distorted by faction or fear — those are valid memory states — but it cannot be invented. The type system won’t let it.

The rule is mechanically enforceable — not by prompt engineering, code review, or guidelines, but by a contract the C# compiler will refuse to compile against. The contract is canon today; the implementation is queued for the next phase. The compiler, unlike code review, is unimpressed by deadlines and unmoved by clever arguments.

What actually happens when an LLM tries to cheat

Walk back to the tavern. Imagine the dialogue model produces a plan claiming the keeper’s brother died at sea last winter. Here’s what happens, step by step, in our pipeline:

The plan is structured, not free-text. The model’s output is a JSON object with intent, target_entity_ids, referenced_facts, tone, and a few other fields. The model can’t just emit prose. It has to commit to a specific intent and specific entity references.
Every entity reference is looked up. The validator walks target_entity_ids against canonical world state. The keeper’s brother doesn’t exist as an entity. The plan is rejected.
Every fact reference is looked up. The validator walks referenced_facts against the canonical event log. There’s no event matching “death at sea, winter, Veyr coast” anywhere in the keeper’s family causal chain. The plan is rejected.
No narrative event is emitted. Because the plan failed validation, no narrative.dialogue.response event is written to the authoritative event store. The rendering step never runs. The line never reaches the player.

The model never gets to “say” the false thing. The hallucination doesn’t survive contact with the canonical store. There’s no compensating action, no retcon, no save-file fixup — the false fact never made it past the boundary in the first place.

The dialogue spec is explicit about a corollary:

The whole game keeps working without the model. NPCs don’t stop existing, the world doesn’t pause, fights still resolve, economies still run. What goes away is the prose. The translator goes offline; the world stays.

“AI assists storytelling” — that’s the specific verb. Not participates in storytelling. Not co-authors the world. Assists. When the assistance is unavailable, the thing it was assisting is unaffected. That’s the test.

The honest status: what ships today, what doesn’t

Some of what we’ve described is running in production-shape on main right now. Some isn’t. We want to be straight about which is which, because it’s part of the story of how we got here.

The simulation core — world state, event envelopes with deterministic identity, the tick loop, the manifest loader — runs today. Test fixtures replay deterministically. The Movement Validator hashes match.
The gateway and the browser client — the boundary that enforces “client never writes state, server never trusts client” — run today. The reflection-based event-routing they use is a recent rewrite (more on that in a moment).
The content pipeline — the build-time tooling that produces the world’s content manifest — runs today.
The narrative service, the dialogue pipeline, and the NPC memory pipeline — do not run on main today. The contracts are frozen, the type designs are accepted, but the implementation isn’t there.

Earlier this year we had all three of those services implemented. They worked end-to-end. Player typed, NPC replied, memory got recorded, the chronicle service rendered prose. It looked like a finished thing.

But in February we ran an extensibility test against the implementation. The test was simple: can a content author add a new kind of memory, or a new kind of narrative template, without modifying engine code? The answer was no. The implementation had hardcoded enums for fourteen narrative-template kinds and five memory variants. Every dispatch path was a switch over those enums. Adding a new kind meant editing four files in the engine and shipping a new build.

We had a rule — “content extensibility shouldn’t require engine edits” — and the implementation didn’t satisfy it. Two paths to recover:

Soften the rule.
Delete the implementation.

We deleted the implementation. The contracts survived; the code didn’t. Several thousand lines of working code, and we put it in an archive branch and started over.

The new implementation will use a registry pattern (attribute-driven discovery, no closed enums) so adding a memory variant or a narrative template is a content-only change. It’s queued for a later development phase. In the meantime, main ships with the simulation core, the gateway, the client, and the content pipeline — the parts that own truth. The parts that interpret truth are deliberately offline until we can rebuild them under the rules they were supposed to obey.

This isn’t a recovery story we’re embarrassed about. It’s the story we wanted to tell. The whole point of the architectural frame in this post — simulation owns reality — is that the rule is more important than the implementation. We caught an implementation that didn’t deserve the rule, and we fixed the implementation, not the rule.

A new layer of enforcement (just shipped)

The most recent enforcement layer landed earlier this week.

The pattern we wanted to prevent looks like this:

// In the browser client. Hypothetical. Don't do this.
const playerEntityId = "player-001";  // hardcoded literal
moveCommand({ entity: playerEntityId, x: 4, y: 7 });

That code looks innocent. It’s also wrong. The browser client doesn’t get to decide what the player’s entity ID is; the server owns identity. If the server’s notion of identity ever changes shape (hashed IDs, longer IDs, namespaced IDs), the client’s hardcoded literal is now lying to the simulation.

The fix is a contract called authority bindings, ratified earlier this week and running on every push today as a contract-plus-validator pair. Every component contract in the repo now has to declare, machine-readably, which fields it consumes from where:

{
  "authority_bindings": [
    {
      "binding_id": "player-entity-id",
      "consumed_concept": "player-entity-id",
      "authority_source": {
        "kind": "seam_payload",
        "seam_id": "session-authenticated-reply",
        "field": "character_entity_id"
      },
      "binding_site": {
        "expected_in": "src/skaldborn-client/src/network/auth-handler.ts",
        "fingerprint": "this._state.playerEntityId = msg.character_entity_id"
      },
      "forbidden_literals": ["\"player\"", "\"player-001\""],
      "rationale": "Player entity ID is server-authoritative; the client must read it from the authentication reply, not invent it."
    }
  ]
}

There’s a grep validator that runs on every push. It checks:

Does the file at expected_in actually exist?
Does it contain the fingerprint substring?
Does the codebase contain any of the forbidden_literals outside test fixtures?

If any of those checks fails, the push fails. Static analyzers (Roslyn for C#, ESLint for TypeScript) are queued to take the same checks deeper, into expression analysis.

This contract is what we built to prevent a specific class of bug from being possible to ship. A bug landed a few weeks ago where the client was holding onto a literal placeholder ID when the server authentication reply changed shape. The simulation kept running correctly. The client kept showing pixels. But every move command from the client was silently dropped because the entity ID didn’t match anything the simulation knew about. The authority of the simulation was preserved — that’s why nothing crashed — but the client’s lie was invisible until a player tried to walk and nothing happened.

The authority-bindings contract is the rule that says: every authoritative ID in a component has one source, declared, in writing, with a fingerprint check. You can’t hardcode it locally. You can’t keep a backup copy “just in case.” There is one root, one binding, one source-of-truth.

What’s next

Some of this is familiar shape: event log, projections, single-writer per partition, deterministic replay. We didn’t invent it. What’s unusual is how tight the rule runs — all the way through the AI — and the boring infrastructure to keep it tight under deadline pressure. The non-obvious bit is the LLM-offline-correctness invariant. A lot of “AI in games” architectures are honest event-sourced underneath but have an asterisk: except for the AI bits. No asterisk here. The simulation is the simulation, with or without the model. The model is a translator. Translators are nice to have. They are not load-bearing.

This is the first post in the launch arc. Up next, two paired companions on the early-2026 rebuild: Aspirational vs Mechanical (story) and Why we deleted two months of working code (engineering). After those, the launch arc continues with deterministic event sourcing at the partition level, then a tour of the sixteen simulation systems, with a shorter post on the manifest model in between.

If you want to follow along, subscribe via the form at the bottom of any page — one short email when the next post lands. If you want to argue, write to devlog@skaldborn.com.

Everything else is the boring engineering of making it true.

Reed Berkowitz (“Rabbit Rabbit”), AI Powered NPCs — Hype, or Hallucination? Curiouser Institute on Medium, December 9, 2023. medium.com/curiouserinstitute/ai-powered-npcs-hype-or-hallucination-11ddfc530e33 ↩ ↩²
Yuan Gao (Meseta), I attempt to play a coherent story in AI Dungeon: Attempt 1, a noire-future-fantasy/mystery. Medium, December 6, 2020. meseta.medium.com ↩
AI Dungeon Help Center, Why does the AI forget or mix things up? help.aidungeon.com ↩
Pantella (Pathos14489/Pantella, GitHub) — fork of Mantella with a ChromaDB-based memory replacement for the original’s summary-based memory. README quote verbatim from the ChromaDB Memory Manager section. github.com/Pathos14489/Pantella ↩
RoboRhythms, Why Replika Memory Suddenly Stops Working for So Many Users. roborhythms.com/replika-memory-broken-fix ↩
Lark Birdy, Negative Feedback on LLM-Powered Storytelling & Roleplay Apps, Cuckoo Network blog, April 17, 2025. Quote verbatim from the “Technical Limitations in Storytelling Bots — Context/Memory Limits” section. cuckoo.network ↩
Ziwei Xu, Sanjay Jain, Mohan Kankanhalli, Hallucination is Inevitable: An Innate Limitation of Large Language Models, arXiv:2401.11817 (submitted January 22, 2024; revised February 13, 2025). arxiv.org/abs/2401.11817 ↩

Skaldborn devlog

The colleague who hallucinates (and the fences that keep it honest)

The thing that saved me was a human glance

You don’t make it honest. You make it checkable.

What an AI colleague actually is on this project

The fence around the writer

The fence around the investigator

How the fences come together: the loop

What it feels like to work this way

What I’d tell myself in February

How to let an agent write your code without giving it the keys

Why this is hard

The one mechanism everything hangs on

Piece 1 — the launcher

Piece 2 — the fence around the writer

Piece 3 — the fence around the investigator

Wiring it together

The loop that drives it

Gotchas worth knowing before you build this

How Skaldborn consumes this

What’s next

Skaldborn's art content pipeline

The thirty-second version

The stack

The hero artifact: the recipe

The pipeline, top to bottom

I am the art director

Bounding the random

The saga

What we validate (and what we don’t)

War stories

What it’s like to add an asset

Receipts

What we’d tell ourselves in February

What’s next

Set up ComfyUI for your own content pipeline (on a 4 GB consumer GPU)

Table of contents

Why this is hard on a 4 GB card

1. Prerequisites

2. Clone skald-forge and install ComfyUI

3. Install the FaceDetailer custom nodes

4. Configure your CivitAI token

5. Download the production model stack

6. Launch ComfyUI

7. Build the production workflow

8. From concept to sprite: PixelLab

Get an API key and set the token

Preprocess your concept

Submit the rotation job

Poll the background job

Download the rotations

Bound your spend

Pure-shell smoke test

9. Tuning for more VRAM

10. Troubleshooting

How Skaldborn consumes this

What’s next

Why we deleted two months of working code (and why we'd do it again)

The test

What “closed” looked like in our code

The fork in the road

What we built in its place

The gates that fell out

What it cost, what it bought

The books were on the shelf the whole time

The discipline we built around the books — and the colleague we built it with

What we’d tell ourselves in February

Footnotes and pointers

Companion posts

Books cited

Aspirational vs mechanical: how I lost two months of Skaldborn (and got something better back)

Two months in

Death by a thousand cuts

The phrase that explained everything

The week the books came out

A Saturday in late March

The cut

A blocky figure on grass

What changed

What I’d tell you, if you’re staring at a fork like this