Farna Control
Read-only demoInstall

Evals

Evals are automated quality checks on agent output. They run as a post-node step, after node completion and before any human-in-the-loop (HITL) gate check. Each eval definition specifies a type, configuration, pass threshold, and failure behavior.

Eval definition

Each eval definition has:

Eval definition
Field Description
id Unique identifier
name Human-readable name
eval_type model_judge, regex, json_schema, custom_function, guardrail, or human_set
config_json Type-specific configuration
pass_threshold Optional numeric threshold between 0 and 1
suite_id Optional suite that groups related evals

A node-level policy gate references an eval definition and sets what a failure does: warn logs the result and the run continues, while block stops the node before its output is used downstream.

Eval types

Eval types are described in this section and under Guardrails, which is a separate section because it runs on input rather than output.

Large language model (LLM) judge

Sends agent output to an LLM with a rubric prompt. Returns a score from 0 to 1 and reasoning. Uses a separate, independently configured model backend, not the agent’s own backend.

Regex

Applies a regex pattern to the output. Pass means a match was found. No threshold.

JSON Schema

Validates output against a JSON Schema definition. Stricter than the agent’s output schema, for catching well-typed but semantically invalid outputs.

Custom function

Calls a Python function registered via the modulo.evals entry-point group. Returns a score from 0 to 1.

Human-authored set

Runs a registered, versioned eval set written by a person: a fixed list of deterministic assertions that check correctness rules rather than output shape. Select the set with config_json.set_name; an optional version is advisory, because one version of a name is active at a time. The eval passes only if every assertion passes, and its detail names the assertions that failed. A missing set_name or an unregistered set name fails the eval.

Because the result is not model-mediated, output content cannot talk the eval into a pass, which is the gap that model_judge leaves open. The shipped demo_classification set (v1) checks a support-ticket classifier, including a business rule such as “a technical outage is always high priority” that regex and JSON Schema checks cannot express. An operator registers a new set in code and references it by name.

Eval result record

Each eval run stores a result per run per node:

Eval result record
Field Description
eval_id The eval definition that ran
run_id The run
node_id The node whose output was evaluated
score Numeric score, nullable for non-scored types
passed Whether the eval passed
detail Human-readable detail
evaluated_at Timestamp

Results are displayed in the run inspection UI. A block failure is written to the AuditEvent log.

Conditional HITL gating

A HITL gate’s hitl_config can carry an eval_condition that references an eval definition by name. The condition fires the gate when the eval result matches the comparison:

Pipeline config 
{
  "id": "55555555-5555-4555-8555-555555555555",
  "node_type": "hitl",
  "position": { "x": 680, "y": 200 },
  "output_schema_id": "66666666-6666-4666-8666-666666666666",
  "hitl_config": {
    "label": "Quality review",
    "description": "The quality eval scored below 0.8, so a human must review the output.",
    "claim_expiry_minutes": 15,
    "human_only": true,
    "eval_condition": {
      "eval_name": "quality-check",
      "threshold": 0.8,
      "operator": "lt"
    }
  }
}

If the condition is true (score < 0.8), the pipeline pauses for human review. If false, execution continues without pausing. The supported operators are lt, gt, lte, gte, eq, and neq.

Guardrails

Guardrails are the input-side analogue of evals: deterministic data-safety checks at the ingestion boundary.

  • A guardrail is an EvalDefinition with eval_type="guardrail"
  • Detection is deterministic and pure (regex or json_schema only)
  • Interception runs at run-creation, before the input payload is persisted

Actions:

Guardrails
Action Behavior
observe Log only
warn Warn and continue
block Terminal eval_failed (error code eval_blocked)
redact Redact the offending content

A block is terminal: the run is never dispatched, never retried, and has no HITL gate.

Policy-as-code

Guardrail configuration is versioned, hashed YAML. You propose a change, review the diff, and apply it through a pull request. Each run’s snapshot pins the guardrail configuration it ran under, with drift detection.

Next: Library