[WatchTree-19]

Description

Phase 5 of the scorer contract proposal separated retrieval (TraceClient), acquisition (OtelTraceSource) and evaluation (OtelToolCallScorer), shipped the local InMemoryTraceClient, and deferred remote adapters. This adds one: OtlpJsonTraceClient, a TraceClient that fetches each requested trace by ID from an HTTP endpoint returning OTLP/JSON.

client = OtlpJsonTraceClient(
    url_template="http://tempo:3200/api/traces/{trace_id}",
    headers={"Authorization": "Bearer ..."},
)
scorer = OtelToolCallScorer(source=OtelTraceSource(trace_client=client))

It is generic rather than a vendor SDK: OTLP/JSON is the OpenTelemetry wire format, and the client uses httpx, which PyRIT already depends on. No dependency is added. It reads spans under resourceSpans (or the older batches key), at the top level or inside a result or trace object, with hex identities as the OTLP/JSON specification requires or base64 identities as some backends emit. Attribute values convert as InMemoryTraceExporter converts them, bytes included. Timestamps are converted from integer nanoseconds exactly and truncated, never rounded, so conversion cannot reorder two spans. Callers can pass their own httpx.AsyncClient for proxies, certificates or tests; otherwise each query opens and closes one through get_httpx_client.

The verdict rules stay as they are; the client only reports what it saw:

• A remote backend cannot say when every span of a trace has arrived, so coverage is never complete (remote_completion_unknown). An observed tool call stands; a missing one is undetermined, never false.
• A trace the backend answers with 404 is unavailable. If every requested trace is missing, the result is available=False, so the observation is UNAVAILABLE.
• A connection error, timeout, non-success status (401, 403, 429, 5xx), malformed body, span of another trace, invalid identity or timestamp, or a repeated span identity with different evidence marks that trace failed. If nothing was retrieved, the client raises TraceAcquisitionError, which OtelTraceSource already maps to ERROR. If another requested trace was retrieved, its spans are kept and coverage is partial, so a positive found there is not thrown away. A failed trace contributes no spans at all, rather than its readable remainder.
• TraceQuery.limit is applied, with query_limit_exceeded when it truncates. The source's own timeout still applies on top of the per-request timeout.

The identifier records the endpoint without its query string, fragment or headers, which may hold credentials, and URLs with embedded credentials are rejected.

Tests and Documentation

• tests/unit/score/test_otlp_json_trace_client.py (41 tests). Every request goes through httpx.MockTransport or a patched client factory; no test opens a network connection. Covered: span and attribute conversion, every accepted layout and identity encoding, unfinished spans, truncation of nanosecond timestamps, incomplete coverage, a remote positive and an undetermined absence through OtelToolCallScorer, missing traces as unavailable, each failure kind as an acquisition error, one failed trace among several as partial evidence, duplicate spans, the query limit, a slow backend hitting the source timeout, headers and per-request timeout, client ownership, the identifier, and configuration validation.
• tests/unit/common/test_lazy_package_imports.py: the new export is added to the existing export-parity and no-SDK-import checks.
• doc/code/scoring/5_tool_call_scorer.py and .ipynb: a section that reads a trace through an in-process transport standing in for a backend. doc/code/framework.md mentions the client.

How it was tested

python -m pytest -q tests/unit/score/test_otlp_json_trace_client.py tests/unit/common/test_lazy_package_imports.py
python -m pytest -q -n 4 tests/unit/score tests/unit/models tests/unit/common/test_lazy_package_imports.py
python -m ruff check <touched files>; python -m ruff format --check <touched files>
ty check --python <venv> <touched files>
jupytext --to ipynb doc/code/scoring/5_tool_call_scorer.py, then jupyter nbconvert --execute
python -m build_scripts.validate_docs

Rebased on main 935afa8: pytest tests/unit/score tests/unit/models tests/unit/common tests/unit/_code_execution tests/unit/output tests/unit/executor/attack -n 2: 11582 passed, 65 skipped. Ruff, format, ty and docs validation are clean. The notebook was executed and its outputs match the .py cells.

The response shapes were checked against fixtures built from the OTLP/JSON format, not against a running backend.

Mutations, each applied alone and then restored, with the number of failing tests (22 of 22 caught):

Mutation	Failing
Coverage reported complete	6
404 treated as an empty retrieved trace	2
Total failure reported as unavailable instead of raising	15
One failed trace discards the other traces' evidence	3
Failed trace missing from coverage reasons	4
Error status treated as success	6
Transport errors not caught	2
Span of another trace accepted	1
Conflicting duplicate span accepted	1
Query limit not applied	1
Identifier keeps the query string	1
Credentials in the URL accepted	1
Headers not sent	1
Timestamps rounded instead of truncated	2
Unset end time parsed as a time	1
Base64 identities rejected	3
result/trace wrappers ignored	2
batches key ignored	1
Error status mapped to unset	1
Integer attributes left as strings	1
Unknown status code accepted	1
Per-request timeout not passed	1

[Copilot]

🟡 Changes recommended

Malformed OTLP timestamps can abort scoring or produce false-positive tool-call verdicts, while exception logging can expose private trace data.

3 open issues

• Sanitize parsing errors before logging · New
• Validate timestamp order before datetime truncation · New
• Reject timestamps outside the OTLP unsigned 64-bit range · New

What changed in this PR

Adds OtlpJsonTraceClient, a remote trace-retrieval adapter for PyRIT’s existing observation and tool-call scoring pipeline, without adding dependencies.

Changes:

• Fetches and converts OTLP/JSON traces over HTTP, with partial-coverage reporting and configurable client ownership.
• Exposes the adapter through lazy public exports and adds mocked retrieval and scoring tests.
• Documents remote trace acquisition with a runnable example.

File	Description
tests/​unit/​score/​test_otlp_json_trace_client.py	Tests parsing, failures, coverage, scoring, and client ownership.
tests/​unit/​common/​test_lazy_package_imports.py	Extends export-parity and no-SDK-import checks.
pyrit/​score/​observation/​otlp_json_trace_client.py	Implements HTTP retrieval and OTLP/JSON conversion.
pyrit/​score/​observation/​__init__.py	Adds the observation-package lazy export.
pyrit/​score/​__init__.py	Adds the scoring-package lazy export.
doc/​code/​scoring/​5_tool_call_scorer.py	Adds a remote-backend example using an in-process transport.
doc/​code/​scoring/​5_tool_call_scorer.ipynb	Adds the matching notebook example and outputs.
doc/​code/​framework.md	Describes the adapter’s role and incomplete-coverage semantics.

_{Review effort: Balanced}