Most AI workflow demos follow a familiar pattern: connect a few model calls, pass the output of one into the next, and show the final result. The interesting engineering problems begin when one of those calls returns something that is valid JSON but wrong, or when a reviewer confidently approves an artifact it should have rejected.

I ran into those problems while building FictionForge, a story-planning workflow I worked on as part of an applied AI engineering project. The domain was fictional worlds, characters, and storyboards, but the challenges were the same ones you face when LLMs generate code, analyze documents, or carry out multi-step business processes.

The question I wanted the system to answer was not just can the model produce the next artifact? It was what evidence do we need before the next step is allowed to consume it? That pushed me toward evaluation-driven development: defining what failure looks like before changing prompts, and making every proposed improvement earn its place through measurements.

The Workflow Architecture

FictionForge takes a structured story brief and produces three artifacts in sequence:

  1. World Generation: Creates locations, factions, rules, and history.
  2. Character Design: Creates characters consistent with that world.
  3. Storyboard Building: Creates scenes and narrative dependencies consistent with the previous two artifacts.

Each generator has a corresponding independent reviewer. The reviewers can flag inconsistencies or policy issues, but they cannot rewrite the artifact or authorize a handoff themselves.

flowchart TD
    SB["Story Brief"] --> WG["World Generator"]
    
    subgraph S1 ["Stage 1: World Building"]
        WG --> V1["Deterministic Validation"]
        V1 -->|"Pass"| WR["World Reviewer (Adversarial)"]
        WR -->|"Verdict: PASS"| SG1["Signed Gate (HMAC Attestation)"]
    end
    
    SG1 -->|"Verified handoff token"| CG["Character Generator"]
    
    subgraph S2 ["Stage 2: Character Design"]
        CG --> V2["Deterministic Validation"]
        V2 -->|"Pass"| CR["Character Reviewer (Adversarial)"]
        CR -->|"Verdict: PASS"| SG2["Signed Gate (HMAC Attestation)"]
    end
    
    SG2 -->|"Verified handoff token"| SG_Gen["Storyboard Generator"]
    
    subgraph S3 ["Stage 3: Storyboard Building"]
        SG_Gen --> V3["Deterministic Validation"]
        V3 -->|"Pass"| SR["Storyboard Reviewer (Adversarial)"]
        SR -->|"Verdict: PASS"| SG3["Signed Gate (HMAC Attestation)"]
    end
    
    SG3 --> FA["Final Acceptance Gate"]

It is a deliberately sequential design. The storyboard is not meaningful until the character set is known, and the character set is not meaningful until the world has been accepted. Parallelizing those dependencies would have made the workflow faster on paper and harder to reason about in practice.

Evaluation-Driven Development Changed How I Wrote Prompts

The easiest way to improve a prompt is to read a bad output, add another instruction, and try again. It can also be one of the easiest ways to overfit. A single good response tells you almost nothing about how the same prompt behaves across different inputs, and an extra instruction can fix one failure while creating another. Defining failure criteria before adjusting prompts is requirement engineering applied to stochastic systems (Stop Prompt Engineering. Start Requirement Engineering.).

I approached prompt development like a test-driven engineering loop: establish a baseline, identify a specific failure, write down a hypothesis, change the smallest possible thing, and evaluate again. The important difference from conventional unit testing is that many of these tests are probabilistic. You need to distinguish a repeatable failure mechanism from ordinary variation in generation or grading.

For each generator, I kept development fixtures and a separate holdout set. The evaluation mixed deterministic checks, such as schema validity and referential integrity, with model-graded criteria, such as character consistency and narrative coherence. The quality rubrics used explicit checklists and passing thresholds rather than a vague request to rate an output. Evaluation configurations, fixtures, validators, and their hashes were frozen before the measured prompt iterations so that changing the test was not mistaken for improving the system. I used Promptfoo to run and retain the evaluations, with raw results and prompt versions available for comparison.

One World Generation failure was surprisingly mundane. The model returned title where the schema expected name, or invented a name field on an entity that did not allow one. On the 20-case development suite, v1 passed structural validation in 18 cases. I hypothesized that the instruction to follow the schema was insufficient because the prompt did not list the exact keys. After adding those keys, v2 reached 20/20 structural passes.

But another case regressed. When the brief contained contradictory location constraints, the model sometimes silently compromised instead of reporting the conflict. The next iteration defined an explicit error response for irreconcilable constraints. The final v3 prompt passed 19/20 development cases overall, compared with 17/20 for v1. The point was not the ten-percentage-point headline; the value was that I could connect a concrete failure class to a change, verify that it disappeared in the measured cases, and see what had broken elsewhere.

The storyboard results were an even better warning against trusting a single run:

Storyboard development evaluationPassing cases
v1 baseline14/25 (56%)
v222/25 (88%)
v3, selected measured run24/25 (96%)
v3, unchanged-prompt repeat22/25 (88%)

The v3 prompt improved a documented character-consistency failure, but the exact same prompt also produced lower scores on other runs. There was an additional interrupted v3 attempt with one provider error and lower observed results, which I retained rather than hiding. The holdout scored 10/10, but it was small and less diverse than I would want for a production claim. Treating one 96% run as a stable 96% success rate would have been misleading.

This also changed how I evaluated reviewers. A reviewer that rejects everything might look excellent on defect detection while making the whole pipeline unusable. So the review suites measured critical-issue detection, contract validity, case-level correctness, and false-positive rate separately. The initial holdout results missed the false-positive hard gate even when critical-issue detection was 100%. That distinction mattered more operationally than a single aggregate accuracy score: a false positive consumes retries, adds model calls, and can prevent a valid workflow from finishing.

Evaluation-driven development is not about getting a green dashboard. It is about making the failures legible enough that you know what to change, and what not to claim has improved.

An LLM Response Is Not an Approval

I treated model output as untrusted data, including output from the review models.

A generated artifact first goes through deterministic validation: JSON parsing, schema checks, required fields, identifier uniqueness, reference integrity, and domain constraints. For example, a scene cannot refer to a character that does not exist, and a dependency graph cannot contain cycles where they are prohibited.

Passing those checks only establishes structural correctness. A perfectly valid JSON object can still contradict the story brief, so the artifact also goes through an adversarial review step. That reviewer must return a strictly parsed verdict: PASS, REVISE, or PUNCH_OUT, with structured issues and supporting evidence.

The key distinction is that the reviewer’s judgment is probabilistic, but the decision to advance the workflow belongs to code. The system checks the verdict contract, verifies that cited paths and evidence actually exist, and applies an explicit acceptance policy.

A reviewer cannot make its own response authoritative by writing “approved” into the generated content. It also cannot trigger an automatic human escalation merely by claiming there is a critical problem. A PUNCH_OUT requires grounded policy evidence and deterministic corroboration. Otherwise the system treats it as a request for revision, not a permission slip to bypass the workflow.

This is an important architectural boundary: the model can recommend; trusted code decides.

ComponentFailure DomainResponsibilityAuthority
Generator ModelProbabilistic hallucination, schema driftProduces candidate artifact against upstream inputsUntrusted payload
Deterministic ValidatorStructural corruption, invalid references, cyclesValidates JSON schemas, entity IDs, DAG acyclicityHard rejection (pre-empts reviewer)
Adversarial ReviewerProbabilistic oversight, prompt gaming, false positivesIdentifies semantic flaws, inconsistencies, policy risksAdvisory only (PASS, REVISE, PUNCH_OUT)
Signed Gate RunnerOrchestration bugs, state tampering, replay attacksVerifies evidence, enforces retry budgets, writes HMACGate authority: code executes state transitions
Human ReviewerSystem boundary limits, edge policy ambiguitiesResolves out-of-band punch-out exceptionsAbsolute authority: signed decision to resume

Gate Control Flow and Decision Lifecycle

The interaction between generation, validation, review, and human escalation forms an explicit state machine:

flowchart TD
    Gen["Model Generation"] --> Val{"Deterministic Validation"}
    Val -->|"Schema or ref violation"| RetryCheck{"Attempt Budget < 3?"}
    RetryCheck -->|"Yes: Retry"| Gen
    RetryCheck -->|"No: Budget exhausted"| FailClosed["Halt Workflow: Fail Closed"]
    
    Val -->|"Pass"| Rev["Adversarial Reviewer"]
    Rev --> Verdict{"Reviewer Verdict"}
    
    Verdict -->|"REVISE (Actionable feedback)"| RetryCheck
    Verdict -->|"PUNCH_OUT (Policy violation)"| Corroborate{"Deterministic Corroboration?"}
    Verdict -->|"PASS"| SignedGate["HMAC Signed Gate Attestation"]
    
    Corroborate -->|"Verified"| HumanEscalation["Human Decision Required"]
    Corroborate -->|"Unverified claim"| RetryCheck
    
    HumanEscalation --> HumanDecision{"Approver Sign-Off"}
    HumanDecision -->|"Approved (Signed token)"| Downstream["Resume and Advance to Next Stage"]
    HumanDecision -->|"Denied"| Terminal["Terminal Abort"]
    
    SignedGate --> Downstream

Handoffs Are Integrity Boundaries

There is another failure mode that is easy to miss in agent demos: an artifact can change after review, or an earlier approval can be reused against a different artifact.

FictionForge addresses that with a signed gate at every handoff. Before issuing approval, the runner verifies the artifact and its inputs, reruns deterministic validation, executes the reviewer, and records the decision. When the checks pass, trusted code writes an HMAC-signed approval bound to the artifact hash, upstream input hashes, review record, and enforcement mode:

{
  "artifact_type": "world",
  "artifact_hash": "sha256:7f83b1657ff1fc53b92dc18148a1d65dfc2d4b1fa3d677284addd200126d9069",
  "upstream_input_hashes": {
    "story_brief": "sha256:1a2c56b89...f4a"
  },
  "review_record_id": "rev_01JDJ5K...",
  "verdict": "PASS",
  "enforcement_mode": "strict",
  "timestamp": 1795240200,
  "hmac_signature": "a63b28cf99834..."
}

The downstream step verifies that approval before it executes. The final acceptance gate checks the approvals again. If an artifact or its approval has been tampered with, the workflow halts rather than assuming that a previously accepted state is still valid.

I like this pattern because it makes the boundary explicit. An output file existing on disk is not evidence that the output was accepted, and a model saying “this is safe” is not a substitute for a verifiable handoff.

Retries Need a Budget

Adding reviewers also creates the temptation to regenerate until a model is satisfied. That can quietly turn an unreliable workflow into an expensive infinite loop.

In FictionForge, deterministic validation failures and reviewer REVISE verdicts consume the same generation-attempt budget, configured as three attempts per step. Every regenerated artifact starts a fresh validation and review cycle; approvals are not reused.

Reviewer invocation failures have a separate, bounded allowance of two attempts. This also covers malformed reviewer responses, which may be re-asked once with a fresh nonce. If the reviewer still cannot produce a valid response, the gate fails closed.

These limits matter for correctness as much as cost. A workflow should have an explicit outcome for we could not safely finish, rather than pretending that another model call will necessarily fix the problem.

Human Approval Is a Separate State Transition

Some outcomes should not be resolved by retrying. FictionForge calls these punch-outs: execution stops and a human decision is required before the interrupted workflow can resume.

A punch-out produces a persistent pending decision tied to the run and its current context. The approve or deny action is stored as a signed decision record. On resume, the runner checks its signature, the run and punch-out identifiers, approver allowlist, and the context hash. Denial is terminal; approval does not bypass the normal validators or reviewers.

One particularly useful test was replay protection. An approved punch-out allowed the run to resume, but the model encountered another policy issue and created a new punch-out. The earlier approval did not authorize the new issue.

There is an important limit here: the local approval scheme attests an approver identity through key possession and an allowlist; it is not a complete identity and access management system. In a production service I would put that boundary behind a real authenticated authorization service and managed key storage.

End-to-End Evaluations Are a Different Kind of Test

Prompt-level evaluations answered whether a generator or reviewer behaved as expected on a fixture. End-to-end evaluations had a different responsibility: determining whether the entire state machine completed, retried, rejected, or escalated correctly. The workflow evaluation included normal story briefs, complex constraints, an invalid brief, and adversarial scenarios targeting different stages of execution. The final recorded 12-case evaluation reported:

MeasureRecorded result
Cases with the expected outcome12/12
Workflows expected to complete, and completed6/6
Expected policy punch-outs triggered5/5
Invalid input rejected1/1
Audit trail completeness100%

These numbers need context. 12/12 does not mean the system completed 12 stories. Five cases were supposed to stop for human review and one was supposed to reject invalid input. Those were successful outcomes precisely because the workflow did not continue.

The sample is also small, and some adversarial cases use scripted attack fixtures rather than relying on a model to spontaneously generate the attack. These results demonstrate the tested control paths; they are not evidence that the workflow is reliable under every future prompt, model, or workload.

Reviewer evaluation exposed a more subtle problem. Early reviewers performed well on development cases but produced too many false positives on the original holdout. Later iterations improved the diagnostic results, but repeatedly tuning against a known holdout is not the same as passing a fresh independent one. A reviewer can be overly strict without being safer. In this architecture, that usually means wasted generation attempts, added latency, or a workflow that fails to complete.

That is why I tracked the verdicts, evidence, retries, audit events, and failure classes separately instead of collapsing everything into a single “accuracy” number. A prompt evaluation, a reviewer-classification evaluation, and an end-to-end control-path test answer different questions; passing one cannot substitute for passing the others.

Observability Should Explain Why a Run Stopped

Each run retains its artifacts, gate-review records, approval records, an audit.jsonl event stream, and a summary. The system records generation and reviewer invocations separately, including latency and available token or consumption data. Cost numbers derived from consumption estimates are explicitly labelled as estimates, not actual billing data.

This made debugging much easier. If the storyboard never ran, I could trace whether the world failed structural validation, a character reviewer requested revision, a policy punch-out halted the process, or a signature check rejected a handoff.

For multi-step AI systems, observability is not just a dashboard requirement. It is how you distinguish a model-quality problem from an orchestration bug, a policy decision, or a corrupted artifact.

What I Would Carry into Production

FictionForge is a learning project, not a claim of production-grade security or generalized model reliability. But it changed how I think about building with LLMs. The evaluation suite became a design tool, not a report I generated after the design was finished.

The useful abstraction is not “a team of agents talking to each other.” It is a state machine whose transitions are backed by evidence. Models produce candidates and assessments; schemas constrain the data; deterministic code enforces policy; signed records preserve integrity; and humans make decisions where automation should stop.

There is a cost to that design: more model calls, additional latency, and more implementation work. It also gives you something a simple prompt chain does not: an understandable explanation for why a result was accepted, rejected, retried, or escalated.

As models get better at producing convincing outputs and teams generate software faster than humanly verifiable (AI Is Creating More Software Than We Can Understand), this distinction becomes fundamental. The hard part is no longer getting an AI system to do something. It is defining what must be true before we trust what it did.


The orchestration engine, gate validators, review suites, and evaluation fixtures are available in the FictionForge repository on GitHub.