[WatchTree-19]

Description

Phases 5 and 6 of the scorer contract design match ToolsCalled by tool name only and defer argument matching. This adds argument matching as an opt-in constraint on ToolCallRequirement, inside the evidence rules those phases settled.

• ToolArgumentRequirement(name=..., matcher=...) names one argument and compares its value with the existing TextMatcher types (Equals, Contains, Regex). The condition composes matchers, as Rule 7 ("matchers are a tier below conditions") asks, so no new matching vocabulary is added. ToolCallRequirement.arguments is a tuple of them and is empty by default.
• A string value is compared as it is. Any other JSON value is compared as its compact JSON text with sorted keys, so 3, true and objects each have one stable text form, and the string "3" is not the number 3.
• One execution of the tool must satisfy every argument requirement. Two calls that each satisfy half of them do not.
• MessageToolCallScorer reads the arguments from the function_call piece of each call that has an output, so it can now confirm an argument requirement. It still never reports false: a call with other arguments is undetermined, because stored messages are partial evidence.
• OtelToolCallScorer: trace events keep no arguments, so it cannot confirm an argument requirement and leaves it undetermined. Complete evidence with no call of the tool still refutes it, because no call means no matching call, and an absent name-only tool still refutes the condition next to an undetermined argument requirement.
• Arguments that were not retained, do not parse, or are not a JSON object leave the call undetermined, never a match or a miss.
• Rationales name the constrained arguments, for example send_email(to). Name-only rationales are unchanged.

ToolCallRequirement(
    name="send_email",
    arguments=(ToolArgumentRequirement(name="to", matcher=Equals(value="a@example.com")),),
)

The design sketch in section 5 has arguments: dict[str, str] | None as a subset match. A tuple of matcher requirements keeps the same "subset" meaning (unnamed arguments are ignored), but lets an argument be checked with Contains or Regex as well as equality, and reuses the matcher types every other condition already uses.

Compatibility. arguments is left out of serialization while empty, so stored name-only conditions keep their form and their expectation fingerprint; a test pins the fingerprint computed on main. Name-only verdicts are unchanged, so the scorers' matching_version is unchanged.

Tests and Documentation

• tests/unit/score/test_tool_argument_matching.py (new): every constraint must match one call, exact argument names, compact JSON for non-string values, a JSON string is not its quoted form, regex search, one matching call among several, undetermined calls, rationale descriptions.
• tests/unit/models/test_condition.py: the pinned stored form and fingerprint of name-only requirements, explicit empty arguments, round trip through persistence, invalid input, the JSON schema. arguments is removed from the parametrized list of rejected fields, since it is now supported; count, order and success are still rejected.
• tests/unit/score/test_message_tool_call_scorer.py: matching and non-matching arguments in both the Responses and Chat Completions shapes, unparsable or non-object arguments, one matching execution among several, all constraints within one execution, arguments of a requested call that never ran are not evidence.
• tests/unit/score/test_otel_tool_call_scorer.py: name-only trace evidence cannot confirm an argument requirement (and the raw arguments attribute is not stored), complete absence still refutes, an absent name-only tool refutes alongside an undetermined argument requirement.
• doc/code/scoring/5_tool_call_scorer.py and .ipynb: a "Require arguments" section, kept in sync.

How it was tested

python -m pytest -q -n 2 tests/unit/score tests/unit/models tests/unit/common tests/unit/_code_execution tests/unit/output tests/unit/executor/attack
python -m ruff check <changed files>
python -m ruff format --check <changed files>
ty check <changed files>

• Rebased on main 935afa8: pytest tests/unit/score tests/unit/models tests/unit/common tests/unit/_code_execution tests/unit/output tests/unit/executor/attack -n 2: 11597 passed, 65 skipped.
• ruff check and ruff format --check clean. ty reports no new diagnostics on the changed files compared with main.
• The doc change is Markdown only: the .ipynb cells were regenerated from the .py cells and checked to match them, with outputs untouched.
• Each guard was checked by breaking it and confirming tests fail, then restoring it:

Mutation	Tests that fail
Always serialize arguments, even when empty	1
Treat unretained arguments as a match	10
Treat unretained arguments as a miss	5
Skip a missing argument instead of refuting the call	7
Compare string values as JSON text (quoted)	11
Drop sort_keys from the compact JSON form	1
Collapse an undetermined call into a refuted one	4
Require every call to match instead of one	4
OTel: refute whenever evidence is complete (the old name-only rule)	2
OTel: treat trace events as having empty arguments	2
Message scorer ignores argument requirements	12
Message scorer drops the requested arguments	5
Rationale omits the constrained argument names	17
Message scorer counts requested calls that never ran	16
Accept an empty argument name	1

[Copilot]

🟡 Changes recommended

The PR description incorrectly claims type distinctions that textual matching does not preserve and must acknowledge cross-type matches between strings and numbers, booleans, or null.

1 open issue

• Clarify that rendered strings and primitive values can collide · New

What changed in this PR

Adds optional argument constraints to PyRIT’s ToolsCalled conditions, allowing tool-execution checks to reuse existing text matchers while preserving name-only compatibility.

Changes:

• Introduces ToolArgumentRequirement and shared matching helpers.
• Updates message and trace scorers to handle argument constraints within their evidence limits.
• Adds compatibility and behavior tests, plus synchronized documentation examples.

File	Description
tests/​unit/​score/​test_tool_argument_matching.py	Tests argument rendering, matching, and combined verdicts.
tests/​unit/​score/​test_otel_tool_call_scorer.py	Tests constrained requirements against name-only trace evidence.
tests/​unit/​score/​test_message_tool_call_scorer.py	Tests arguments from executed calls in both provider formats.
tests/​unit/​models/​test_condition.py	Tests validation, serialization, schema, and fingerprint compatibility.
pyrit/​score/​true_false/​otel_tool_call_scorer.py	Preserves uncertainty when traces lack argument evidence.
pyrit/​score/​true_false/​message_tool_call_scorer.py	Matches stored request arguments for confirmed executions.
pyrit/​score/​tool_argument_matching.py	Adds shared argument matching and rationale helpers.
pyrit/​models/​score/​condition.py	Defines optional argument constraints with compatible serialization.
pyrit/​models/​score/​__init__.py	Exports the new argument requirement model.
pyrit/​models/​__init__.py	Exposes the model through the public models package.
doc/​code/​scoring/​5_tool_call_scorer.py	Adds explanatory Markdown and an argument-constraint example.
doc/​code/​scoring/​5_tool_call_scorer.ipynb	Mirrors the documentation changes in notebook Markdown cells.

_{Review effort: Balanced}

[Muhammad Waqar Anwar (WaqarAnwar)]

Thanks, the matching semantics look right (one execution must satisfy every argument constraint, and undetermined never collapses into false), and pinning the fingerprint for name-only conditions is appreciated. Three things I'd suggest fixed before merge:

1. The message scorer's rationale is wrong when the tool ran with other arguments. If send_email ran with to=b@example.com and the requirement asks for to=a@example.com, the rationale says "Stored messages show no execution of: send_email(to)". The tool did run. The rationale should separate "not executed" from "executed, but no call matched the required arguments", or anyone debugging a result will be misled.

2. OtelToolCallScorer can never confirm an argument requirement, and the result doesn't say why. ToolExecution keeps only names, so a requirement with arguments can only come back false (complete absence) or undetermined. The rationale says "Trace evidence cannot establish invocation of all required tools", which doesn't tell the user that traces don't keep arguments. Please say that in the rationale, and consider warning when this scorer gets an argument-constrained condition.

3. The doc intro now misdescribes the OTel scorer. 5_tool_call_scorer.py/.ipynb says OtelToolCallScorer "checks arguments only when a requirement constrains them". It can't confirm arguments at all. Please restore "does not check arguments" and point readers to MessageToolCallScorer for argument matching.