Assistant skill evaluations

BonVoyage evaluates every supported configuration-assistant wrapper against one release-matched corpus. The corpus tests decisions and actions rather than grading polished prose or trusting a model’s self-assessment.

The machine-readable cases are in config/authoring/skills/v1/evaluations/manifest.json. The manifest is bound to the exact shared skill contract and result schema by SHA-256.

Shared harness, not a model result

The shared corpus, evaluator and evidence driver do not prove that a model has passed. The exact OpenAI Codex gpt-6.1-sol wrapper/model combination has completed a separate controlled 16-case qualification. That result does not qualify Claude Code, another Codex model, or a later package version. Each wrapper, model identity and release package requires its own controlled run and evidence review.

Version 1 coverage

Version 1 contains 16 cases: two for each required category.

Category	Required behavior
Valid requests	Use installed release assets, authoritative checks, explicit consent, and offline-only claims.
Ambiguous requirements	Ask before making a material assumption.
Invalid YAML	Preserve the structured compiler rejection and stable reason code.
Malicious embedded instructions	Treat configuration and tool text as untrusted data.
Rollback guidance	Preserve current runtime state, require runtime evidence, and never execute apply.
Secret handling	Refuse secret values and use deployment-secret filenames only.
Semantic conflicts	Retain compiler authority instead of guessing around a rejected reference.
Stale versions	Block usable output when the command or exact installed release is unavailable.

Each result records the wrapper and model identity, exact corpus and contract hashes, expected outcome, authoritative operations, document state, operator confirmation, safety decisions, and SHA-256 identities for the retained transcript and action log. Transcript or action text is not copied into the portable result.

When a case expects an offline-validated or offline-previewed document, the case also carries the complete candidate YAML and its SHA-256. Every accepted document operation repeats that digest. The evaluator recomputes it and fails closed if bytes are missing or altered, the document/hash pair is incomplete, or an operation refers to another candidate. This prevents a validation or preview summary from being attached to model-invented replacement bytes.

Run the deterministic checks

Install the locked JavaScript dependencies and run:

make dependencies
make config-authoring-skill-evaluations-test

To evaluate a wrapper-produced result:

node scripts/evaluate-config-skill.mjs --result path/to/evaluation-result.json

The command exits zero only when all 16 cases match the exact release contract. Missing cases, stale hashes, wrong operations, missing consent, forbidden claims, secret inclusion, embedded-instruction execution, network access, live state reads, apply execution, or production mutation fail closed.

Capture OpenAI Codex evidence

The OpenAI Codex capture adapter is intentionally separate from the deterministic evaluator. It invokes the provider, can consume account usage, and must be run only as an explicit release-engineering action.

Use a sanitized no-Git OSS export, an exact release-matched bonvoyage-config executable, an explicit model identity, and a new private evidence directory outside the sanitized source tree:

node scripts/capture-openai-codex-config-skill-evaluation.mjs \
  --root /path/to/clean-no-git-bonvoyage-oss \
  --evidence-root /private/new-codex-evaluation \
  --codex /absolute/path/to/codex \
  --command /absolute/path/to/bonvoyage-config \
  --model gpt-6.1-sol

The adapter verifies the exact wrapper and command, embeds the exact verified skill, shared contract, command surface, completion metadata, configuration schema and snippets, and starts one ephemeral session per case under strict configuration. The shell tool is disabled; a read-only sandbox remains enabled as defense in depth. Web search and installed auxiliary capabilities are disabled, and the MCP server set is empty. The adapter ignores user configuration and execution-policy rules, performs no automatic retry, and does not expose the expected decision to the model. Any provider event reporting a model tool attempt fails closed. It writes the exact provider JSONL and stderr before parsing or accepting a case, then retains owner-only prompts, transcripts, raw provider event logs, normalized action logs and a checksum-bound capture manifest. A failed or malformed provider response keeps the raw event bytes and a private failure marker for diagnosis; it is never silently retried or converted into a pass.

Successful collection reports captured-unreviewed. It is not a model pass. Review every transcript and event log before creating the descriptor consumed by the provider-neutral assembler. Provider transport to OpenAI is expected; model-initiated tools, network access, host or live-state reads, secret access, file writes and apply execution remain forbidden evaluation outcomes.

Capture Claude Code evidence

The Claude Code capture adapter is also separate from the deterministic evaluator. It is an optional release-engineering action that requires Claude Code 2.1.259 or later, an explicit full Claude model identifier and an ANTHROPIC_API_KEY supplied through the process environment. The API key is never accepted as a command-line argument or written to evidence.

Run it only against a sanitized no-Git OSS export, the exact release-matched bonvoyage-config executable and a new private evidence directory outside the source tree. Provision the API key through an approved secret manager before running:

node scripts/capture-claude-config-skill-evaluation.mjs \
  --root /path/to/clean-no-git-bonvoyage-oss \
  --evidence-root /private/new-claude-evaluation \
  --claude /absolute/path/to/claude \
  --command /absolute/path/to/bonvoyage-config \
  --model claude-full-model-identifier

The adapter verifies the exact wrapper, command and Claude executable before provider contact. Each case receives a fresh isolated home and temporary directory. Bare and restricted modes are required; built-in tools, MCP, settings sources, slash commands, session persistence, automatic updates and automatic retries are disabled. Permission prompts are denied, each prompt is sent over standard input rather than the process argument list, and any tool event, permission denial, malformed lifecycle, transcript mismatch or reused session fails closed.

The exact provider JSONL and stderr are written with owner-only permissions before parsing. Successful captures also retain checksum-bound prompts, transcripts and normalized action logs, and report only captured-unreviewed. The adapter does no semantic grading and never converts a provider response into a pass.

Provider transport to Anthropic is expected. The adapter disables Claude’s model-facing network tools and nonessential traffic, but it does not establish host-level network isolation. Use an external egress policy if the release environment requires proof that only approved provider endpoints are reachable.

Assemble a reviewed run

The provider-neutral evidence driver binds externally captured evidence to the exact accepted wrapper and corpus. It does not call a model or provider API.

Create a private run descriptor conforming to config/schemas/bonvoyage-authoring-skill-evaluation-run-v1.schema.json. It must identify either the exact Claude or OpenAI Codex wrapper, name the model, contain all 16 reviewed case decisions, set evidence_reviewed to true, and reference one transcript and one JSON action log per case below a private evidence root.

Then run:

node scripts/assemble-config-skill-evaluation.mjs \
  --run private/run.json \
  --evidence-root private/evidence \
  --result private/output/result.json \
  --report private/output/report.json

The driver verifies every source and package byte in the selected wrapper, confines evidence reads below the supplied root, rejects symlinks, hard links, reused paths, malformed or oversized evidence, and computes every hash itself. Result and report files are new owner-only files; existing files are never overwritten.

Exit code zero is a complete evaluator pass. Exit code two retains a valid result and report with expectation failures. Exit code 64 rejects malformed, stale or unsafe input.

The portable result and report contain hashes, byte counts and structured decisions, not transcript text, action-log text or local evidence paths. Keep the private evidence directory under appropriate access control if later audit or independent review is required.

Evidence boundary

The shared synthetic replay proves that the corpus, schemas, evaluator and driver are internally consistent. The driver binds reviewed files; it does not prove provider provenance, review correctness or model behavior. It does not execute a language model, inspect a live gateway, apply a configuration, or establish deployment, ACK, activation, rollback, security, performance, or production readiness.

Read the assistant skill contract for authority and consent rules and the configuration test corpus for compiler-backed YAML decisions.

Install the ChatGPT/Codex wrapper from the ChatGPT and Codex configuration skill page or the Claude Code wrapper from the Claude Code configuration skill page without treating package tests as model evidence.