Remove the implicit 500-byte tool-result cutoff so relevance judges can assess the same evidence recorded for the agent. Cover boundary lengths, tail and middle evidence, UTF-8, and large outputs.
Closes#4770
Assisted-By: docker-agent
Signed-off-by: melmennaoui <mohamed.elmennaoui@docker.com>
Write fake container runtime scripts before t.Parallel and symlink
them from each subtest's temp dir, so forked children don't inherit
a writable executable fd and hit ETXTBSY on Linux.
Assisted-By: cagent
Recording/replay now scrubs evaluator credentials and matches on
canonical request bodies, with tests covering the gateway, direct,
and replay paths.
Assisted-By: docker-agent
Create the shared shell fixture before the parent parallel barrier and use per-case symlinks and data files. Preserve digest pinning and fallback assertions without concurrent writes to executable fixture inodes.
Signed-off-by: Arnaud Héritier <aheritier@users.noreply.github.com>
Flavors are now forwarded to eval containers in order, and tool-mode
structured output is treated as the agent's real final answer instead
of the output tool call itself or an earlier rejected attempt.
Assisted-By: docker-agent
Use crypto/rand-backed stdlib uuid.NewV4().String() for random UUIDs and stdlib parsing instead of github.com/google/uuid, wherever the stdlib can express the behavior.
github.com/google/uuid is kept only in pkg/model/provider/base/opencode.go for uuid.NewSHA1, since the stdlib has no UUIDv5 support; the namespace and resulting IDs are unchanged.
pkg/userid/userid.go keeps a compatibility helper that preserves every legacy persisted-ID form the Google parser accepted, without rewriting or rotating existing IDs. pkg/evaluation/eval.go keeps reading the first 4 UUID bytes as big-endian to preserve existing decimal evaluation names.
Added tests covering v4 bit layout, legacy persisted-ID forms, the evaluation name format, and the OpenCode UUIDv5 fallback.
Signed-off-by: David Gageot <david.gageot@docker.com>
Add per-evaluation usage accounting: evaluations are persisted as session
items, surfaced in the /cost dialog under a dedicated "By Evaluator" section,
and feed into budget enforcement. Unknown pricing renders as "unknown" rather
than a misleading $0.
Assisted-By: Claude
withVersion wrote the global version.Version from a t.Parallel() test
while sibling parallel tests read it through DefaultAgentImage(), which
the race detector flagged. Extract the pure defaultAgentImageFor helper
and table-test it with explicit versions instead.
runDockerAgentInContainer read stderrData after cmd.Wait() while a
hand-rolled goroutine was still writing it; Wait does not wait for our
own goroutine. Let exec.Cmd own the copy via a bytes.Buffer so Wait
(bounded by WaitDelay) guarantees the data is complete before we read it.
Every session created against a local workspace now records the absolute
workspace root that owns it, so generated files can later be resolved
against the owning workspace instead of whatever cwd the viewer happens
to run from:
- delegated/nested sub-sessions inherit parent.WorkingDir
- serve mcp / a2a / chatserver capture the server workspace once at
startup instead of reading os.Getwd per request (or not at all)
- embeddedchat conversations and `docker-agent new` capture the
configured root or the creation-time cwd
- --working-dir writes the absolutized path back into RuntimeConfig so a
relative flag value cannot leak into persisted provenance
- compaction runs inherit the compacted session's workspace
Remote/headless surfaces (remote backend template, API templates without
working_dir, ACP without client cwd, evaluation container transcripts)
intentionally keep an empty WorkingDir and now document that choice.
session.ResolveWorkingDir adds a bounded, cycle-safe, validated
parent-chain fallback for old persisted sub-sessions with an empty
WorkingDir; it rejects relative/malformed roots and never falls back to
the process cwd. It is the primitive the generated-media resolver will
build on.
RunOutputConfig.AgentImage previously used omitempty on the *resolved*
image (ResolvedAgentImage), which is empty precisely when
--agent-image none was passed to skip binary injection. That made an
explicit opt-out indistinguishable from a run predating this field.
Drop omitempty so the field is always present, with an empty string
now unambiguously meaning injection was skipped.
Also clarifies that binary injection applies to every eval run, not
just --base-image ones, and adds save_test.go coverage for all three
--agent-image outcomes.
Previously the eval harness always injected the docker-agent binary from
the rolling docker/docker-agent:edge tag, so `docker agent eval` tested
against a moving main-HEAD binary regardless of the host CLI's own
version. Default the injected image to docker/docker-agent:<version>,
matching this CLI's release version, falling back to :edge for dev/main
builds where version.Version isn't a valid semver.
Add --agent-image to override the default explicitly, including a
"none" value that skips binary injection entirely and trusts whatever
/docker-agent is already present in the base image.
Fixes part 2 of #4175.
- config.Load is now YAML-only. HCL support becomes a source decorator,
hcl.NewSource, applied by the resolver; embedders wrap sources that may
carry HCL themselves.
- The OCI source moves to pkg/config/ocisource (ocisource.New).
- Resolve, ResolveSources, ResolveAlias and BuiltinAgentNames move to
pkg/config/sources together with the embedded built-in agents. This is
the package that knows every source type.
- teamloader no longer hardwires the resolver for external sub-agents:
WithSourceResolver supplies it, and loaderdefaults.Opts wires
sources.Resolve so CLI behaviour is unchanged.
pkg/config drops hcl, go-cty, levenshtein, textseg, wordwrap,
klauspost/compress and image-spec, and only reaches
go-containerregistry/pkg/name for reference parsing.
Extend the baseline regression gate to cover assertion pass rates and
surface pass@k/pass^k consistency metrics from --repeat runs.
Also removes the verify script dead code (verify field, VerifyCheck type,
verify.go/verify_test.go) — the assertion system covers verifiable cases
via the captured event stream, and verify cannot work without container
lifecycle changes that aren't warranted.
Assertion rate gates:
- AssertionRate added to Metrics, derived from Summary totals
- Compare() gates on assertion rate drops (same semantics as size
and relevance)
- computeSummary() accumulates assertion counts
- printSummary() displays the Assertions metric line
Repeat metrics (informational):
- PassK, HatK, RepeatK added to Metrics
- Compare() reports pass@k and pass^k as informational deltas
Verify cleanup:
- Removed EvalCriteria.Verify, VerifyCheck, verify.go, verify_test.go
- Schema's DisallowUnknownFields ensures clean future re-addition
Closes#4087
Add a code-based assertion evaluator and integrate it into the eval
pipeline so assertions declared in the eval JSON are executed after the
agent run, alongside the existing size/tool-call/relevance checks.
New files:
- assertions.go: runAssertions() evaluates 8 assertion types
(contains, not_contains, equals, starts_with, ends_with, regex,
cost_threshold, tool_called) against the agent response, cost,
and tool-call list.
- assertions_test.go: comprehensive tests for all assertion types.
Modified files:
- types.go: AssertionResults/AssertionsPassed/AssertionsTotal fields
on Result; checkResults() reports assertion successes and failures.
- eval.go: runSingleEval() runs assertions after relevance checks.
- save.go: populateEvalResult() populates AssertionsCheck on the
session's EvalResult.
- progress.go: tracks and displays assertion failure counts in the
live progress bar.
- eval_test.go: checkResults tests for assertion pass/fail/zero.
Closes#4086
When --repeat k is greater than 1, compute and display two
consistency metrics after the standard summary:
- pass@k: fraction of unique evals that passed at least once across
k repetitions. Measures whether the agent can produce the correct
answer at all.
- pass^k: fraction of unique evals that passed every repetition.
Measures determinism / reliability.
Results are grouped by InputPath (the source eval JSON file) so
repeated runs of the same eval are correctly paired.
Changes:
- types.go: RepeatMetrics struct, new field on Summary
- scoring.go: computeRepeatMetrics(), updated printSummary
- eval.go: wire computeRepeatMetrics into Evaluate()
- eval_test.go: tests for all-pass, flaky, never-pass, mixed,
and printSummary rendering
Closes#4085
Replace the minimal judge prompt with a structured evaluation prompt
that produces more consistent and fair relevance judgments:
- Rubric: explicit pass/fail definitions (fully satisfies vs. partial)
- Chain of thought: step-by-step reasoning before the verdict
- Anti-bias rules: ignore length, politeness, formatting, and effort
unless the criterion explicitly requires them; evaluate substance
over style; binary criterion — no partial credit
The structured-output schema (pass/fail + reason) is unchanged.
Closes#4084
Add runVerifyScript() to the evaluation package. It executes a shell
script via `docker exec` on the eval container after the agent run
completes, enabling filesystem-level and side-effect checks that
cannot be expressed as text assertions on the response.
- Zero exit code → pass; non-zero → fail
- 60-second timeout prevents runaway scripts
- stdout + stderr captured, capped at 8 KB
- Empty script is a no-op pass (verify is optional)
- Fake-runtime test helper avoids a real Docker daemon
Closes#4083
Tests now write a run with SaveRunSessionsJSON and read it back through
LoadBaseline, so the round trip the command actually performs is covered
rather than one that no caller uses.
Adds the fail-closed cases (foreign baseline, empty run), pins that
resultPassed agrees with checkResults for every expectation kind, and pins the
specific divergence the old definition hid: a tool score dropping below 1.0
flips a printed PASS to FAIL and must gate even inside the tolerance.
Three defects, each of which made the gate unusable or untrustworthy.
--baseline could not read anything `docker agent eval` produces. LoadBaseline
expected an EvalRun; the command writes a RunOutput via SaveRunSessionsJSON,
and their Duration fields are typed differently (string vs time.Duration), so
the load failed outright. SaveRunJSON, which the description named as the
producer, has no non-test callers. Baselines are now read from RunOutput —
its summary carries the aggregate rates and sessions[].eval_result.passed
carries per-eval pass/fail — so "no new format" is finally true.
The gate failed open. Any JSON object loaded successfully, and an empty
baseline yields all-zero metrics: every rate is skipped by its has-flag guard,
the failure rate by its own guard, and the run reports "No regression" while
every evaluation in it fails. The same held when the current run had no
evaluations, e.g. an --only pattern matching nothing. Both are now rejected, as
is a baseline sharing no metric and no evaluation with the run — a comparison
that can only pass is worse than no comparison.
resultPassed contradicted the project's own definition of pass. checkResults
requires a tool-call score of 1.0; resultPassed required only
>= ToolCallsExpected, so an evaluation whose printed status flipped from PASS
to FAIL was recorded as no change and the aggregate movement was then absorbed
by the tolerance. It now delegates to checkResults, so there is one notion of
"passed" in the package and the gate cannot drift from the product.
Also: a run that did not complete no longer prints a baseline verdict at all.
Its missing evaluations register as absent and do not gate, so it printed
"✅ No regression against baseline" and then exited non-zero.
And --regression-tolerance above 1 is rejected at startup: no rate can move
that far, so it silently disabled the aggregate gate.
pkg/session mixed the in-memory store with the file-backed constructor,
so every embedder of pkg/runtime linked modernc.org/sqlite (large and
slow to compile) even when using purely in-memory sessions.
session.NewSQLiteSessionStore(ctx, path) becomes sqlitestore.New(ctx,
path); behavior (open, migrate, backup+retry recovery) is unchanged.
SQLiteSessionStore, NewSQLiteSessionStoreFromDB and the migrations stay
in pkg/session - they only use database/sql and don't drag the driver.
BREAKING CHANGE: callers of session.NewSQLiteSessionStore must import
github.com/docker/docker-agent/pkg/session/sqlitestore and call New.
On context cancellation, exec.CommandContext's default Cancel SIGKILLs
the `docker run` CLI, which is never proxied to the container — the
container keeps running daemon-side and --rm never fires. This broke
when the containerTracker was removed (75654ed12).
Override cmd.Cancel to send SIGINT instead, letting the docker CLI
proxy the signal to the container so it stops and --rm removal fires.
WaitDelay force-kills the CLI as a backstop.
Assisted-By: Claude
Only forward dedicated single-secret model API keys into the eval
container. Exclude non-secret detection/mode flags (GOOGLE_GENAI_USE_VERTEXAI),
multi-variable AWS/Bedrock credentials, and the broad GITHUB_TOKEN, none of
which should leak into a privileged container. Sort output for determinism
and add test coverage.
Thread context through sqlite/session/RAG/memory DB setup so constructors and migrations no longer mint fresh roots. Fake proxy cleanup and askpass connection handling now use context providers derived from their startup context. LocalRuntime stores a context provider for tool-change refresh callbacks.
The remaining suppressions are intentionally detached event hooks/MCP supervisor callbacks or compatibility fallbacks.
RAG toolsets, the eval judge, and the js/wasm build all broke with
'unknown provider type' because the default registry is now empty.
Thread the resolved registry through RuntimeConfig and repopulate
the wasm-safe provider set directly in factory_js.go.
Assisted-By: Claude
melmennaoui
Djordje Lukic
David Gageot
Arnaud Héritier
Arnaud Héritier
Arnaud Héritier
Dwin Gharibi
Sayt-0