Farna Control
Read-only demoInstall

Variant A/B Testing

A/B testing answers one question: which pipeline configuration produces the better result for the same input? Farna Control answers it by running variants of the same pipeline and keeping the per-run record that makes a comparison possible.

Reading a comparison

The comparison draws on the runs recorded against each group and covers:

  • Evaluation scores per variant (see Evals)
  • Token cost per run and per variant (see Cost Management)
  • Human decisions at human-in-the-loop (HITL) gates, with reviewer and timestamp
  • Output differences between variants for the same input
  • Prompt version hashes, so a result can be traced to the prompt that produced it

Comparing a batch

To compare every variant on the same input at once, start a batch run and read its comparison. POST /api/v1/variant-groups/{id}/batch-run fires one run per variant against the same payload, and GET /api/v1/variant-groups/batches/{batch_id}/compare returns each run’s status, eval pass rate, cost, tokens, and the frozen snapshot and override diff. See Variant Groups for the endpoints.

Comparing fairly

Scores are only comparable when each variant exercised the same checks. Check coverage gaps before drawing a conclusion: a variant that did not trigger a given eval has one fewer way to fail, so its scores are not directly comparable to a variant that did. See Variant Execution for how gaps are reported.