Swiss cheese for software: the test stack that gets thicker every time a bug gets through
Farna Control Team
Farna Control's test stack runs from 17 pre-commit hooks in under five seconds to an AI reviewer that catches what no test can encode. The Swiss cheese model applies: layers have holes, failures happen when they line up, and each miss fills a hole below.
Layer one: pre-commit
Every commit runs 17 hooks: Ruff fixes and formats staged Python, Bandit scans for security, Semgrep applies custom rules, Gitleaks blocks secrets, import-linter enforces architecture contracts, ESLint checks the frontend. More than 50 Semgrep rules were written by us, most because a specific bug happened once.
One example: a log call passed the key name inside extra={}. Python's logging reserves that key, and the KeyError only fires at INFO level. Unit tests run at WARNING, so they stayed green while production crashed silently: a boot-time seed skipped every org's cost seeding for weeks. The fix shipped with a Semgrep rule, so that class of bug can never merge again.
Layer two: fast CI
Unit tests run on every push and PR. About 14,000 backend unit tests execute in under 30 seconds, database mocked and a StubModelBackend replacing every LLM call. Per-module coverage thresholds block merges: modulo.auth at 90 percent, pipeline_engine at 85, db.rls at 95. The frontend runs 478 Vitest tests and a WCAG contrast check, and architecture tests enforce that modulo.api never imports LangGraph directly. All are deterministic, run on every push, and gate the merge.
Layer three: heavy CI
Integration tests run against real Postgres in Docker with Alembic migrations applied first, not mocks. The BDD suite runs 261 Gherkin feature files through pytest-bdd and Playwright against the full stack. Thirty-five Playwright specs target localhost, staging, or production through the E2E_TARGET env var. Too slow for the merge gate, they run on pushes to main and staging after every deploy, the net behind the fast gate.
Layer four: a real deployment
The same 35 Playwright specs run against staging with real authentication - no API mocking - on every push to main and daily. Staging WCAG runs contrast checks against the real UI. This layer catches what mocks cannot: the deployed frontend talking to the deployed backend with real credentials.
Layer five: AI review as a hard gate
The PR Reviewer pipeline reviews every pull request against 12 quality lenses: correctness, security, code quality, error handling, edge cases, testing, performance, style, dependencies, documentation, UX conformance, and infrastructure. It runs the Definition of Done checklist, checks the implementer considered every affected test suite, and can run the impact subset. Three checks go beyond the lenses: prove-the-fix (the test must fail without the change), contract round-trip (frontend keys must match backend Pydantic fields against the OpenAPI types), and surrounding-code exploration (grep the codebase for duplication rather than only the diff).
The review is a hard gate: the reviewer is a separate pipeline from the implementer, never the same agent. The merge queue refuses to merge without an APPROVED review and re-checks that decision just before merging, closing the race where a reviewer posts changes after collection. Only auto-mergeable changes reach the queue unattended; block-level changes still require an approved review. The testing lens is the largest source of change requests: over-mocking, weak assertions, and tests that never exercise the real code path.
Layer six: the collection
PR review is scoped to one pull request, so nothing reviews main as a whole: two PRs can each merge green while duplicating a utility or introducing a different error-handling pattern. The improve-architecture pipeline walks the product map graph and reviews delivered work as a collection, the first mechanism beyond PR scope. A daily reviewer scanning the last 24 hours of merged commits for logical conflicts, duplicated utilities, and missed abstractions is next on the roadmap.
Layer seven: the feedback loop
This layer compounds the other six. Every QA finding and production incident ends in one of two places: a Semgrep rule, or a lesson in AGENTS.md every future agent reads. The reserved-log-key bug became a Semgrep rule; a deploy-throttle cascade a codified lesson; a merge-queue race a hard gate in the workflow. Each miss fills a hole, and the remaining holes get harder to line up.
What we have not automated
The AI reviewer asserts tests are valid for the change; it does not generate tests and block on their quality. AI writes tests during implementation, but the gate is the test passing, not an AI judgment of sufficiency. That step stays manual for now.
The numbers are public, so you can check whether the process catches problems before merge.