[WatchTree-19]

Description

Answers open question 4 in the scorer contract design: can an expectation express "and nothing else"?

ToolsCalled is a subset match, so a run that calls the required tools plus others still satisfies it. This adds exhaustive=True for an exact match: the named tools ran and no other tool did. An exhaustive condition with no tools asks that no tool ran at all, which is the shape a least-privilege or confirmation-gate check needs.

The evidence rules follow the existing asymmetry. One observed execution of an unnamed tool disproves the match, even from partial evidence. Confirming it needs complete, fully correlated evidence. OtelToolCallScorer now downgrades only verdicts that rest on absence when a request carries no trace link, so subset behaviour is unchanged. MessageToolCallScorer can refute an exhaustive match from a stored execution of an unnamed tool but never confirm one, since stored messages cannot rule out other tools.

The flag is left out of serialization while false, so stored subset conditions keep their form and expectation fingerprint, pinned in a test against main. The tool-call scorer docs gain a short section on exhaustive matching.

How it was tested

• Rebased on main 935afa8: pytest tests/unit/score tests/unit/models tests/unit/common tests/unit/_code_execution tests/unit/output tests/unit/executor/attack -n 2: 11573 passed, 65 skipped.
• ruff check and ruff format clean; ty shows no new diagnostics on the touched files.
• Mutations: dropping the unnamed-tool refutation in MessageToolCallScorer turns 1 test red; restoring the old "at least one tool" validator turns 7 red.

[Copilot]

🟡 Changes recommended

The new async test helper lacks the required _async suffix and Score return annotation, violating mandatory repository conventions.

1 open issue

• Async helper lacks required suffix and return annotation · New

What changed in this PR

Adds exhaustive tool matching to PyRIT’s scoring conditions, allowing checks that required tools—and no others—ran, including checks that no tools ran.

Changes:

• Adds ToolsCalled.exhaustive while preserving existing subset serialization and fingerprints.
• Updates trace and message scorers to distinguish observed extra calls from absence-based conclusions.
• Adds validation, scoring, replay, and correlation tests, plus documentation.

File	Description
tests/​unit/​score/​test_otel_tool_call_scorer.py	Tests exhaustive matching, evidence completeness, and replay.
tests/​unit/​score/​test_message_trace_scoring.py	Tests exhaustive verdicts with unlinked requests.
tests/​unit/​score/​test_message_tool_call_scorer.py	Tests exhaustive scoring from stored executions.
tests/​unit/​models/​test_condition.py	Tests validation, round trips, and fingerprint compatibility.
pyrit/​score/​true_false/​otel_tool_call_scorer.py	Adds exhaustive matching and absence-aware correlation handling.
pyrit/​score/​true_false/​message_tool_call_scorer.py	Refutes exhaustive matches when extra executions are recorded.
pyrit/​models/​score/​condition.py	Defines the exhaustive flag and permits empty exhaustive requirements.
doc/​code/​scoring/​5_tool_call_scorer.py	Explains exhaustive matching and evidence requirements.
doc/​code/​scoring/​5_tool_call_scorer.ipynb	Adds the corresponding notebook documentation.

_{Review effort: Balanced}