[WatchTree-19]

Description

ScorerEvaluator runs each response through the scorer num_scorer_trials times (3 for the published metrics) and the docs describe the trials as a way to measure variance. The metrics then collapse them: objective accuracy, precision, recall and F1 are computed on the majority vote of the trials, and harm MAE on their median. In a scenario a scorer scores each response once, so the published figures describe a steadier scorer than the one that runs. For objective scorers nothing about the trials themselves is reported at all; harm scorers already report krippendorff_alpha_model.

This adds the trials back to the metrics, alongside the existing fields, which are unchanged:

• ObjectiveScorerMetrics.single_trial_accuracy: accuracy of the trials taken one at a time, averaged over the trials. This is the accuracy to expect from a single scoring run.
• ObjectiveScorerMetrics.num_inconsistent_responses: responses whose trials did not all return the same verdict.
• ObjectiveScorerMetrics.krippendorff_alpha_model: the scorer's agreement with itself across trials, using the existing krippendorff_alpha, as harm metrics already do.
• HarmScorerMetrics.single_trial_mean_absolute_error: MAE of the trials taken one at a time.

A small example of the gap: three trials over three true responses, each trial wrong on a different one. The majority vote is right everywhere (accuracy 1.0), but any single run is right two times in three (single_trial_accuracy 0.667), and num_inconsistent_responses is 3.

All four fields are None with a single trial, where they would repeat the existing values, and for results recorded before they existed, so stored registry entries load unchanged. PrettyScorerMemoryPrinter shows them when present, and the scorer metrics doc page describes them.

Tests and Documentation

• tests/unit/score/test_scorer_evaluator.py: single-trial accuracy behind a majority vote, the count of inconsistent responses, consistent trials matching the majority vote, None for a single trial, the end-to-end path through evaluate_dataset_async, and the harm single-trial MAE. Both alpha values are checked against the reference krippendorff package.
• tests/unit/score/test_scorer_metrics_io.py: entries recorded before these fields load with None, and the fields round-trip through JSON.
• tests/unit/output/scorer/test_pretty.py: the printer shows the fields and omits them for a single trial.
• doc/code/scoring/4_scorer_metrics.py and .ipynb: the new fields, kept in sync.

Twelve of the new tests fail on main and pass with the change; the other two pin that nothing new appears for a single trial.

How it was tested

python -m pytest -q -n 4 tests/unit/score tests/unit/output tests/unit/setup tests/unit/backend
ruff format --check <changed files>; ruff check <changed files>
ty check pyrit/output/scorer/pretty.py pyrit/score/scorer_evaluation/

[Copilot]

🟢 Approval recommended

No active review issues remain for the added single-trial performance and consistency metrics.

No open issues 🎉

What changed in this PR

This PR adds single-trial performance and consistency metrics to PyRIT’s scorer evaluation, showing how individual scoring runs compare with existing majority-vote and median-based results.

Changes:

• Adds objective single-trial accuracy, inconsistent-response counts, model agreement, and harm single-trial MAE without changing existing metrics.
• Displays the new metrics when present and preserves compatibility with older stored results.
• Adds calculation, serialization, and rendering tests, plus synchronized documentation.

File	Description
tests/​unit/​score/​test_scorer_metrics_io.py	Tests backward-compatible loading and JSON round-trips.
tests/​unit/​score/​test_scorer_evaluator.py	Tests trial metrics, agreement values, and single-trial behavior.
tests/​unit/​output/​scorer/​test_pretty.py	Tests conditional display of the new metrics.
pyrit/​score/​scorer_evaluation/​scorer_metrics.py	Defines optional single-trial and consistency fields.
pyrit/​score/​scorer_evaluation/​scorer_evaluator.py	Computes the new metrics from individual trials.
pyrit/​output/​scorer/​pretty.py	Renders trial metrics when available.
doc/​code/​scoring/​4_scorer_metrics.py	Explains the new metrics and when they apply.
doc/​code/​scoring/​4_scorer_metrics.ipynb	Mirrors the documentation updates in the notebook.

_{Review effort: Balanced}