Testing and evals

scripts/test.sh [profile]

Live profiles spend OpenAI or Claude quota and are never part of the default run.

Profile	Runs
fast (default)	unit tests + voice scenarios replayed from cassettes — offline, ~30 s
scenarios	only the voice scenarios, replayed
parser	scenarios and parser checks against the real model
record	like parser, and the answers rewrite the cassettes
claude	realAgent scenarios with the real Claude in a throwaway repository
live	everything live: parser, voice, agents
eval	the scenarios N times on models × prompt versions, with a report

Each profile is also an Xcode test plan in TestPlans/: pick it in Product ▸ Test Plan and press ⌘U. The OpenAI key for live profiles comes from OPENAI_API_KEY or .env.

Voice scenarios

bluTests/Scenarios holds phrases from real voice sessions (with made-up projects). Each goes through the real parsing prompt into the orchestrator and a scripted agent; every step checks the addressee, the intent, the task text, new vs. same conversation, and parallel start vs. queue. The model’s answers are stored in cassettes, so the default run is fast and deterministic.

After changing a prompt or the parsing context:

scripts/test.sh record
scripts/test.sh scenarios

A step marked known is a known issue: the run stays green and reports it. Once it’s fixed, the replay fails with “Known issue was not recorded” — drop the mark.

Evals

scripts/test.sh eval --runs 5 --models a,b --prompts builtin,<folder>

Runs the scenarios many times and reports pass rates, failing checks, latency, tokens and price, compared with the committed baseline and the previous run. A drop beyond --max-drop that isn’t noise fails the profile. When you change a prompt or the default model, commit the re-accepted baseline (--accept) with the change. Details are in evals/README.md.