Agent output moves downstream to a merge, a deploy, or a customer, and the question is whether it was good enough. Evals are automated quality checks that score output against rubrics, patterns, or schemas. They run after a node completes and before human review, so every output is measured against your criteria.

In short

Each eval runs a check (LLM judge, regex, schema, or custom function) against a pipeline node's output and returns a pass or fail.
Use them when output must meet a quality bar before moving downstream.
Skip them when the output is trivial, read-only, or already covered by an automated gate that catches the same failure modes.
Failure behavior is configurable: warn logs and continues, block stops the run.

The problem it solves

Say an agent writes code and opens a pull request. It passes the structural checks - the diff is well-formed, the branch name is valid, the commit messages follow convention - then ships a change that introduces a race condition, a test that always passes, or a response that answers but misses the customer's concern. Schema validation confirms shape, not substance, and a regex catches patterns, not intent. You need a check that scores output against a quality standard you define. Without evals, the only quality gate before the next step is a person reading everything, which does not scale.

How it works

Evals are post-node steps: when a node finishes, its output passes through every configured eval before the pipeline moves on. Each eval scores the output independently.

LLM judge

An LLM judge sends the output to a separate model with a rubric prompt; the model returns a score from 0 to 1 and reasoning. Configure the model backend (independent of the agent's own), rubric text, and pass threshold. Use it when quality is subjective: tone, completeness, persuasiveness, or anything a regex cannot capture.

Regex

A regex eval applies a pattern to the output text; a match passes, with no numeric threshold. Use it to confirm the presence or absence of a pattern: a classification label, a required footer, or a hardcoded credential.

JSON schema

A JSON schema eval validates output against a formal schema, stricter than the agent's own. Use it for semantic validation beyond shape: a non-empty list, a resolving URL, or a field that matches a format constraint the base schema does not enforce.

Custom function

A custom function calls a Python function registered via the modulo.evals entry-point group, which receives the output and returns a score from 0 to 1. Use it for logic that does not fit the other types: cross-referencing external data, running a deterministic algorithm, or a compound condition.

Configuration

Every eval has a pass_threshold (0 to 1) and a failure_behaviour. Set the threshold to the minimum passing score, and the behavior to warn to log and continue or block to stop the run with an eval_failed terminal state:

{
  "id": "quality-check",
  "name": "Output quality",
  "type": "llm_judge",
  "config": {
    "model_backend_id": "review-model",
    "rubric": "Score the response on completeness (0-1), accuracy (0-1), and clarity (0-1). Weight equally."
  },
  "pass_threshold": 0.8,
  "failure_behaviour": "block"
}

Conditional HITL gating

A HITL gate can reference an eval in its condition field, creating a gate that pauses only when the eval score falls below a threshold. Output that scores well continues without review; output that does not pauses so a reviewer sees the output, the score, and the reasoning, routing only borderline cases to a person.

Guardrails as input-side evals

Guardrails are the input-side counterpart to evals: they check input before a run is created, using the same EvalDefinition structure with type guardrail. Their detection is deterministic (regex or JSON schema only) and runs at the ingestion edge. A guardrail can observe, warn, block, or redact content. A block is terminal: the run is never dispatched, never retried, and has no HITL gate.

Journey example: before and after

Before

Priya's team uses an agent to generate release notes from merged pull requests: it reads the diff, summarizes, and posts to a changelog. One week it summarizes a breaking database migration as "minor improvements," and the team publishes it. Customers upgrade, the migration breaks their integrations, and the team spends the week fielding tickets. Nothing measured the output before publish: the schema check confirmed well-formed JSON, not accuracy.

After

Priya adds two evals: an LLM judge scoring completeness and accuracy against the diff, and a regex checking for "BREAKING CHANGE" when the diff touches migration files. Both run at block, so publish runs only when both pass. The judge catches misleading summaries, the regex catches missed breaking changes, and support tickets for that category drop to zero.

When to use it vs. when not to

Use evals when output quality matters and a person would catch the failure by reading: incomplete answers, inaccurate summaries, content-policy violations, or responses that miss the point. Use them to automate that judgment and escalate only borderline cases through HITL gating. An eval on every node adds latency and cost for trivial output or output a structural check already covers; when a schema check confirms the shape and a regex catches the failure mode, that is enough. Reserve evals for steps where quality variation is real.

Where it fits in a pipeline

Evals sit between node execution and HITL gates. Automated gates catch patterns, evals measure quality, and checkpoints catch the judgment calls that need a person, so a pipeline needs less manual oversight as confidence grows. For the gate that pauses for human review, see How HITL gates stop bad agent output before it ships. For the broader governance model, see The principles of Farna Control.

Go deeper

Read the Evals docs for the configuration reference, or the related docs on HITL, schemas, and agents. Take the product tour to see evals in action, or use the contact page to talk about putting evals in your pipeline.