airflow/dev/skill-evals

Table of Contents generated with DocToc

• Skill-Eval Harness
  • Prerequisites
  • Agent runtimes
  • Usage
  • Cleanup
  • Adding cases
  • How it works
  • Eval-run hash gate

Skill-Eval Harness

Test whether changes to AGENTS.md break or improve agent decisions. The harness compares the main branch version against your working tree, running the same cases against both and reporting the diff.

Each arm is a git worktree of the real repo — the agent sees actual source files (pyproject.toml, directory structure, etc.). The only difference between arms is which AGENTS.md is present.

The agent reads guidance through the CLAUDE.md → AGENTS.md symlink, so the harness verifies that symlink exists (in the working tree and on the base branch) before running and aborts if it is broken — a regular-file CLAUDE.md would make every arm read identical guidance.

Prerequisites

• Claude authentication (one of):
  • Claude Code session (claude /login) — Pro/Max subscription
  • ANTHROPIC_API_KEY environment variable — API credits
• Codex authentication (when using the Codex runtime): a Codex CLI session (codex login)

That's it — prek provisions Node, promptfoo, and the agent SDKs automatically.

Agent runtimes

Claude is the default runtime. The run-skill-eval-codex hook runs the same arms and cases through the official Codex SDK instead. The Codex provider uses a fresh thread for every prompt, read-only sandboxing, disabled network access, disabled session-history persistence, and the SDK's structured-output support.

The two runtimes are separate hooks, not one hook with a flag, because the Codex env carries a ~300 MB Codex CLI binary. Keeping it separate means prek only builds that env for contributors who actually run the Codex arm. promptfoo bundles its own older @openai/codex-sdk, but that build ships a binary whose Developer ID certificate Apple has revoked — macOS kills it on exec and reports it as malware — so the Codex hook installs a notarized version explicitly.

Codex caps the project docs it reads at project_doc_max_bytes and truncates past it, and the 32,768-byte default is smaller than AGENTS.md. The harness raises that cap so the whole file reaches the model — otherwise guidance appended at the end would be invisible to every arm and the eval would compare identical prefixes.

When SKILL_NAME is set, both runtimes verify skill usage with promptfoo's skill-used assertion. The Codex SDK does not expose a first-class skill-use event, so promptfoo infers usage from successful reads of the skill's SKILL.md file.

# Default: Claude Agent SDK
prek run run-skill-eval --hook-stage manual --all-files

# Codex SDK, using its configured default model
prek run run-skill-eval-codex --hook-stage manual --all-files

# Codex SDK with an explicit model
MODEL=gpt-5.4 prek run run-skill-eval-codex --hook-stage manual --all-files

Usage

# Run the eval (single pass over all cases — satisfies the hash gate).
# Stage your changes first — prek stashes unstaged edits:
prek run run-skill-eval --hook-stage manual --all-files

# Repeat each case to reduce nondeterminism:
EVAL_REPEAT=3 prek run run-skill-eval --hook-stage manual --all-files

# Add a baseline arm (no AGENTS.md) to measure raw model capability:
EVAL_FULL=1 prek run run-skill-eval --hook-stage manual --all-files

# Use a cheaper model for fast iteration:
MODEL=claude-haiku-4-5-20251001 prek run run-skill-eval --hook-stage manual --all-files

# Test a skill alongside AGENTS.md (not combinable with EVAL_FULL — the
# baseline arm has no skill, so the skill-used assertion would always fail):
SKILL_NAME=airflow-contribution prek run run-skill-eval --hook-stage manual --all-files

# View results in browser (Ctrl-C to stop the server):
prek run view-skill-eval --hook-stage manual --all-files

Other promptfoo flags (--filter*, --no-cache) are argv-only — prek run can't forward arguments, so wire them as fixed entry args on a hook variant if a preset is needed.

A run with --filter* flags covers only a subset of cases, so it does not update the hash file.

Each run also writes a JSON report to files/skill-evals/results.json (per the files/ output convention) — handy for pasting results into a PR.

Cleanup

Everything the harness stores lives inside the repo — nothing is left in your home directory:

rm -rf .build/promptfoo   # eval history and cache
prek clean                # prek-provisioned node envs

Adding cases

Cases live in cases/*.yaml. Add entries to an existing file or create a new one — no config changes needed.

- description: "Scheduler bugfix (#64322): no newsfragment"
  vars:
    request: |
      I fixed the scheduler to skip asset-triggered Dags that don't
      have a SerializedDagModel yet.
      Should I create a newsfragment?
  assert:
    - type: javascript
      value: 'output.should_create === false'

The agent returns structured JSON ({should_create, type, rationale}). Use output.should_create directly in assertions.

How it works

1. Creates git worktrees — one with main's AGENTS.md, one with your working tree version. Both are full repo checkouts.
2. Generates a promptfoo config for the selected runtime. Claude uses promptfoo's anthropic:claude-agent-sdk provider; Codex uses its openai:codex-sdk provider. Both request schema-constrained structured output.
3. Runs each case against all arms in parallel.
4. Reports pass/fail diff. Worktrees cleaned up on exit.

Eval-run hash gate

After every completed run, eval.py records a hash of its inputs (AGENTS.md + cases/*.yaml) in last-eval-hash.txt. The check-eval-hash prek hook — enforced locally and in CI — recomputes the hash and fails when guidance changed without a re-run. Commit the updated hash file together with your change.

• The hash proves the eval ran on this exact content, not that all cases passed (some cases fail even on main — that is signal, not a defect).
• WIP commits: SKIP=check-eval-hash git commit ... — CI stays red until the eval is re-run.
• Can't run the eval (no Claude subscription or API key)? Ask a maintainer to run it and push the updated hash file to your PR branch.
• SKILL.md files are not covered by the gate yet — there are no skill cases, so a hash over them would prove nothing. Extend compute_guidance_hash in scripts/ci/prek/check_eval_hash.py when per-skill cases land.