[WatchTree-19]

Description

Phase 5 of the scorer contract proposal shipped name-only ToolsCalled matching over trace evidence (#2725) and deferred ordering conditions. This adds them as a separate leaf, following section 5: one declared condition type per leaf (Rule 4), with presence and order combined in the scorer tree rather than inside one scorer (Rule 5).

• ToolsCalledInOrder is a new condition (condition_type: "tools_called_in_order") holding at least two distinct exact tool names. It registers through the existing condition registry, so expectations persist and round-trip like the other conditions.
• OtelToolCallOrderScorer matches it. It reads the same ObservationSource[TraceScorable] and the same stored ToolEventsObservationPayload as OtelToolCallScorer, so no payload, model, or serialized form changes. Message scorables resolve trace links exactly as OtelToolCallScorer does.

What counts as an order: a listed tool comes before the next one only when one of its execution spans ended strictly before a span of the next tool started, by more than an optional clock_skew_tolerance. Other tools and repeated attempts may run in between. Overlapping spans, unfinished spans, and spans with equal or near-equal timestamps establish no order in either direction, because the evidence does not say which came first.

Verdicts follow the existing evidence rules:

• True when an order is observed, including from partial evidence, as with presence.
• False only with complete coverage, when every way of picking one attempt per tool is contradicted beyond the tolerance, or a listed tool never ran.
• Undetermined otherwise: overlap, partial coverage, error or unavailable acquisition, and, inherited from OtelToolCallScorer, requests in the conversation that carry no trace link.

The tolerance is part of the scorer identifier. Stored observations replay offline against new orders without calling the trace client. Ordering uses only what span intervals establish, so it does not rely on argument or result capture.

Tests and Documentation

• tests/unit/score/test_otel_tool_call_order_scorer.py (34 tests): the interval rules (sequential, reversed, concurrent, equal timestamps, unfinished spans, repeats and interleaving), the earliest-finishing and earliest-starting choices across repeated attempts, clock tolerance in the matcher and through the scorer, error and unavailable acquisition, identifier contents, negative tolerance, condition routing, offline replay, unlinked requests, and a composite that routes ToolsCalled and ToolsCalledInOrder to separate leaves.
• tests/unit/models/test_condition.py: round trip through the registry and persisted expectations, order sensitivity, and the two-distinct-names rule.
• doc/code/scoring/5_tool_call_scorer.py and .ipynb: a section that scores a sequential trace in both orders and an overlapping trace. doc/code/framework.md mentions the new scorer.

How it was tested

python -m pytest -q tests/unit/score/test_otel_tool_call_order_scorer.py tests/unit/models/test_condition.py
python -m pytest -q -n 4 tests/unit/score tests/unit/models
python -m ruff check <touched files>; python -m ruff format --check <touched files>
ty check --python <venv> <touched files>
jupytext --to ipynb doc/code/scoring/5_tool_call_scorer.py, then jupyter nbconvert --execute

Rebased on main 935afa8: pytest tests/unit/score tests/unit/models tests/unit/common tests/unit/_code_execution tests/unit/output tests/unit/executor/attack -n 2: 11578 passed, 65 skipped. Ruff and format are clean; ty reports nothing on the touched pyrit/ and test files. The notebook was executed and its outputs match the .py cells; the .py and .ipynb cell sources are identical.

Mutations, each applied alone and then restored, with the number of failing tests:

Mutation	Failing
Equal timestamps count as an order (< to <=)	2
Clock tolerance ignored	3
Negative returned on partial coverage	3
Overlap returns false (no possible-order check)	9
First finished attempt used instead of the earliest finishing	1
Possible-order search uses the latest start	1
Unfinished attempts allowed to precede	1
Error and unavailable acquisition not guarded	2
Unlinked-request downgrade removed	1
Tolerance missing from the identifier	1
One-tool order condition accepted	1

One further mutation, treating an unfinished span as ending at its start inside the precedence check, leaves every test green. It is unreachable: the definite search filters unfinished spans first, and the possible-order search runs only on complete coverage, which cannot contain unfinished spans.

[Copilot]

🟡 Changes recommended

The ordering search loses constraints from earlier tools, causing an impossible three-tool sequence to return undetermined instead of being rejected.

1 open issue

• Preserve ordering constraints across the full selected tool prefix · New

What changed in this PR

Extends PyRIT’s trace-backed tool scoring from presence checks to relative execution order, building on #2725 while reusing stored evidence and offline replay.

Changes:

• Adds ToolsCalledInOrder, a registered condition requiring at least two distinct tool names.
• Adds OtelToolCallOrderScorer with interval-based matching and configurable clock-skew tolerance.
• Adds validation and scorer tests, plus runnable ordering examples.

File	Description
tests/​unit/​score/​test_otel_tool_call_order_scorer.py	Tests ordering, tolerance, replay, and condition routing.
tests/​unit/​models/​test_condition.py	Tests condition validation and serialization.
pyrit/​score/​true_false/​otel_tool_call_scorer.py	Implements interval matching and the order scorer.
pyrit/​score/​__init__.py	Exports the new scorer lazily.
pyrit/​models/​score/​condition.py	Defines the ordering condition and validation.
pyrit/​models/​score/​__init__.py	Exports the condition from scoring models.
pyrit/​models/​__init__.py	Exposes the condition through the public models API.
doc/​code/​scoring/​5_tool_call_scorer.py	Adds sequential, reversed, and overlapping-span examples.
doc/​code/​scoring/​5_tool_call_scorer.ipynb	Adds matching notebook examples and outputs.
doc/​code/​framework.md	Describes order scoring alongside presence scoring.

_{Review effort: Balanced}