Configuration

Configuration lives at .zoto/eval-system/config.yml — the only supported path. The file is validated against templates/schema/config.schema.json.

Migration

Earlier scaffolding wrote to .zoto-eval-system/config.json. That path is no longer supported; the plugin reads .zoto/eval-system/config.yml exclusively. Run /z-eval-init in any repository that still has the legacy file to drop the new template into place; the old file can then be deleted.

The init template

/z-eval-init drops a fully-commented YAML skeleton into .zoto/eval-system/config.yml. Every key is commented out alongside the internal default; uncomment the keys you want to override.

# ─────────────────────────────  discovery  ───────────────────────────────────
# evalsDir: evals
# skillsRoots:
#   - .cursor/skills
#   - skills
#   - plugins/*/skills
# discoveryTargets:
#   - skill
#   - command
#   - agent
#   - hook
# ignore: []

# ──────────────────────────────  backends  ───────────────────────────────────
# static:
#   framework: pytest          # pytest | vitest | jest
# llm:
#   runtime: tsx               # tsx | node
#   strategy: declarative      # fallback default when analyser classification unavailable (declarative | code)
#   codeFramework: vitest      # vitest | jest — applies to all code-strategy targets
#   model:
#     id: composer-2.5           # composer-2.5 | claude-opus-4-8[] | sonnet

# ────────────────────────────────  judge  ────────────────────────────────────
# judgeModel: claude-opus-4-8[]

# ────────────────────────────  manual checklists  ────────────────────────────
# manualChecklists:
#   enabled: false

# ────────────────────────────  extra automation  ─────────────────────────────
# additionalAutomation: []     # any of: vitest, jest

# ───────────────────────────────  analyser  ──────────────────────────────────
# analyser:
#   concurrency: 4
#   maxCallsPerInvocation: 50

# ───────────────────────────  run retention (gc)  ────────────────────────────
# runs:
#   retention: 30

# ────────────────────────────────  update  ───────────────────────────────────
# update:
#   criticalChangeRules:
#     addedTargetWithoutCoverage: true
#     removedTargetWithActiveCases: true
#     skillFrontmatterChange: true
#     publicSurfaceChange: true
#     promptTemplateChange: true
#   preserveUserAuthoredCases: true       # hard-coded contract — do not change
#   writeMetaMarker: true                 # hard-coded contract — do not change

Field reference

Host layout

Field	Default	Purpose
`hostLayout`	`plugin`	`plugin` = lean (bridge to installed plugin). `ejected` = self-contained vendored runtime under `.zoto/eval-system/`. Patched automatically by `eval:stamp-host-layout` and `eval:un-eject`.

Discovery

Field	Default	Purpose
`evalsDir`	`evals`	Where static tests, the `_llm/` declarative runner, and `evals/llm/` code-strategy tests are stamped.
`skillsRoots[]`	`[".cursor/skills", "skills", "plugins/*/skills"]`	Glob roots the discoverer walks to find `SKILL.md` files.
`discoveryTargets[]`	`[skill, command, agent, hook]`	Which kinds of artefacts to scaffold evals for. Also valid: `cli`, `lib`.
`ignore[]`	`[]`	Glob patterns to skip during discovery.

Static backend

Field	Default	Purpose
`static.framework`	`pytest`	Static test harness the stamper targets: `pytest` \| `vitest` \| `jest`.

LLM backend

Field	Default	Purpose
`llm.runtime`	`tsx`	`tsx` or `node` — runner runtime for the LLM backend.
`llm.strategy`	`declarative`	Fallback default when analyser classification is unavailable — not the per-target choice. Hybrid scaffolds coexist: non-interactive targets get declarative JSON (`evals.json` + `_llm/runner.ts`); interactive targets get code-strategy `evals/llm/test_*.test.ts`. Per-target backend is chosen at stamp time from analyser `requiresInteraction`.
`llm.codeFramework`	`vitest`	Test runner for all code-strategy targets: `vitest` or `jest`. Must equal `static.framework` when both are TS-based. Changing strategy defaults or reclassification triggers `eval:cleanup-stale` to remove wrong-backend artefacts per target.
`llm.model.id`	`composer-2.5`	Default LLM model — `composer-2.5` \| `claude-opus-4-8[]` \| `sonnet`. CLI `--model` and `ZOTO_EVAL_MODEL` env var win in that order.

Judge

Field	Default	Purpose
`judgeModel`	`claude-opus-4-8[]`	Model used by `/z-eval-judge` for adversarial scoring.

Extras

Field	Default	Purpose
`manualChecklists.enabled`	`false`	Stamp `USER_EVAL_CHECKLISTS.md` on create.
`additionalAutomation[]`	`[]`	Optional extras such as `vitest` / `jest` stamped alongside the baseline static backend. Not legacy `bats` (removed in eval-system v2).
`analyser.concurrency`	`4`	Max parallel analyser calls. Range: 1–32.
`analyser.maxCallsPerInvocation`	`50`	Hard cost cap per command run. Cache hits do not count.
`runs.retention`	`30`	Maximum number of run folders kept under `{evalsDir}/_runs/`.

Update rules

Field	Default	Purpose
`update.criticalChangeRules.addedTargetWithoutCoverage`	`true`	New uncovered surface area is critical.
`update.criticalChangeRules.removedTargetWithActiveCases`	`true`	Dangling generated cases must not pass vacuously.
`update.criticalChangeRules.skillFrontmatterChange`	`true`	Triggering / retrieval depend on skill frontmatter.
`update.criticalChangeRules.publicSurfaceChange`	`true`	The thing being tested behaves differently.
`update.criticalChangeRules.promptTemplateChange`	`true`	Generated prompts cite the template.
`update.preserveUserAuthoredCases`	`true` (hard-coded)	Setting this to `false` is a validation error.
`update.writeMetaMarker`	`true` (hard-coded)	Setting this to `false` is a validation error.

Environment variables

/z-eval-create stamps a .env.example placeholder at the repo root containing CURSOR_API_KEY= and a commented ZOTO_EVAL_MODEL=. It is never overwritten if one already exists. The runner imports dotenv/config at startup, so values in .env flow into process.env automatically.

Variable	Purpose
`CURSOR_API_KEY`	Required for the LLM backend. Without it, `--full` exits cleanly.
`ZOTO_EVAL_MODEL`	Overrides `llm.model.id`. CLI `--model` wins over both.

cp .env.example .env       # then edit .env locally; .env is gitignored
pnpm install               # picks up the dotenv devDep
pnpm run eval:full         # CURSOR_API_KEY is now sourced from .env

Never commit .env. The default repo .gitignore already excludes .env* while allowing .env.example.

Manifest layout

The manifest pair under .zoto/eval-system/ records the discovery snapshot and an append-only history.

.zoto/eval-system/manifest.yml — current state. Validated against templates/schema/manifest.schema.json.
.zoto/eval-system/manifest.history.yml — append-only list of full snapshots, keyed by git_ref and updated_at.