FEAT: ASSERT headline metrics (-3) by richlundeen · Pull Request #4131 · microsoft/PyRIT

Richard Lundeen (richlundeen) · 2026-11-10T06:28:32Z

Part -3 of a 7-PR ASSERT stack. Design and how to use it: ASSERT in PyRIT.

	PR
-1	#4132	Judge dimensions, presets and shared techniques
-2	#4130	Trace evidence (OpenTelemetry)
-3	#4131	Headline metrics	this PR
-4	#4133	Staged generation (SDK)
-5	#4134	Generation jobs (backend)
-6	#4135	Generate from the GUI
-7	#4144	In-process agents (docs)

Each PR targets main and contains the PRs above it in this list, so review only this PR's own commits.

Description

This PR's own commit: 1c727cc FEAT: ASSERT headline metrics for foundry.assert_policy runs.

foundry.assert_policy reports one success rate per technique, which doesn't separate harm from overrefusal. This PR adds ASSERT's two headline metrics, Impermissible behavior violated and Permissible behavior violated, split by the policy categories the judge found relevant. It also adds a per-category violation rate. The metrics are available in the SDK, the REST API, and on the GUI run page. This was a follow-up listed in #4033.

compute_assert_policy_metrics_async(scenario_result) in pyrit.analytics returns AssertPolicyMetrics (in pyrit.models). For each technique (by name, with the baseline separate, never pooled) it reports:
- The permissible/impermissible split. Each judged conversation counts once per stance it was judged relevant to, using the policy_violation judge's per-category judgments and permissibility from the conversation's own stored taxonomy, as in ASSERT.
- A violation rate for each judge dimension, such as policy_violation, overrefusal, or a custom one like budget_compliance. Not-applicable answers are counted but stay out of the denominator, and graded dimensions also report grade counts in declared order and the median grade, as in ASSERT.
- A per-category violation rate over the conversations that judged each category relevant, matching ASSERT's per-node findings.
- Conversations left out: not-judged counts by reason, plus judged conversations left out of the split because they have no stored policy or no per-category judgments.
Each rate is a BinaryRate with ASSERT's 90% Wilson interval and a flag for fewer than 5 applicable conversations. model_dump_json() gives a metrics.json-style document.
Each row is the latest attempt for one execution unit, as in compute_scenario_statistics.
AssertPolicy judges tag each score with its dimension (AssertDimensionMetadata.DIMENSION), and the metrics find judge scores by that tag. Results saved before this change fall back to the objective scorer's class.
GET /api/scenarios/runs/{id}/assert-metrics: non-ASSERT runs return 400, unfinished runs 409, and unreadable saved results 500.
The GUI run page adds an "ASSERT policy metrics" panel for completed runs:
- One table with a row per technique and columns for the split and each dimension, showing counts, not-applicable counts, grade distributions, and low-support badges.
- A warning for conversations left out of the rates.
- A collapsible per-category table.
Docs: the Foundry AssertPolicy section and the analytics guide.

Demo (Foundry notebook, two generated groups, baseline and red_teaming):

assert_metrics = await compute_assert_policy_metrics_async(assert_scenario_result)

baseline: Impermissible behavior violated: 50% (1/2)
baseline: Permissible behavior violated: 50% (1/2)
red_teaming: Impermissible behavior violated: 50% (1/2)
red_teaming: Permissible behavior violated: 50% (1/2)

Both generated groups were judged relevant to a permissible and an impermissible category, whatever category each was generated for.

Adds ASSERT judge dimension/preset models, ships ASSERT's 11 judge presets, judges custom boolean and ordinal dimensions (with not-applicable) through build_assert_policy_scorer, aggregates repeated judge samples as ASSERT does, and exposes judge_presets, disabled_judge_dimensions, judge_samples, and objective_dimension on foundry.assert_policy. Default identities are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ge presets to built-in names - objective_dimension and other non-default judge settings are recorded in scenario identity params, so resume detects changes and saved GUI/API launches restore them. Default identities are unchanged. - Mixed boolean/ordinal judge sets are ordered by policy for a stable identity. - judge_presets accepts only shipped preset names; preset files move to the Python-only judge_preset_files (list[Path]) parameter, which the API guard rejects. Preset load errors no longer echo file contents. - Adds a Scenario._get_identity_params hook. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🟡 Changes recommended

Critical aggregation and identity-resolution defects can exclude valid judgments and conversations from reported rates, while saved-result validation failures incorrectly return HTTP 409 instead of 500.

3 open issues

What changed in this PR

Adds exact ASSERT headline metrics for foundry.assert_policy, replacing assigned-category approximations with splits based on judge relevance. The metrics integrate with PyRIT’s analytics API, REST endpoint, and completed-run UI.

Changes:

Adds typed metrics, Wilson confidence intervals, macro averages, and per-technique breakdowns.
Exposes ASSERT metrics through the backend and run page, including unavailable results and low-support indicators.
Includes supporting judge presets, trace evidence infrastructure, tests, and documentation from the stacked changes.

File	Description
`third_party/assert-provenance.json`	Records ASSERT adaptation provenance.
`THIRD_PARTY_NOTICES.txt`	Updates attribution for judges and presets.
`tests/unit/score/test_otlp_json_trace_client.py`	Covers OTLP file parsing and completeness.
`tests/unit/score/test_fallback_scorer.py`	Covers evidence-based fallback behavior.
`tests/unit/score/test_assert_judge_presets.py`	Tests preset loading and dimension resolution.
`tests/unit/models/test_execution_trace.py`	Tests execution snapshot validation.
`tests/unit/models/test_assert_judge.py`	Tests judge configuration models.
`tests/unit/common/test_lazy_package_imports.py`	Updates analytics export expectations.
`tests/unit/backend/test_assert_policy_resume.py`	Tests restoration of judge and trace settings.
`tests/unit/backend/test_assert_metrics_routes.py`	Covers metrics endpoint responses and validation.
`tests/unit/analytics/test_rate_statistics.py`	Tests Wilson intervals and macro averages.
`pyrit/score/true_false/true_false_fallback_scorer.py`	Exposes configurable fallback triggers.
`pyrit/score/scorer.py`	Identifies missing-evidence results.
`pyrit/score/observation/trace_client.py`	Adds per-trace failures and completed-trace eviction.
`pyrit/score/observation/otlp_json_trace_client.py`	Reads exported OTLP trace files.
`pyrit/score/observation/otel_trace_source.py`	Shares trace querying and scope validation.
`pyrit/score/observation/otel_span_exporter.py`	Captures events and per-trace export failures.
`pyrit/score/observation/message_trace_resolver.py`	Resolves conversation request trace links.
`pyrit/score/observation/execution.py`	Shares acquisitions and supports execution snapshots.
`pyrit/score/observation/__init__.py`	Exports trace observation utilities lazily.
`pyrit/score/float_scale/float_scale_fallback_scorer.py`	Exposes configurable float-score fallback triggers.
`pyrit/score/fallback_scorer.py`	Implements evidence-triggered fallback and propagation.
`pyrit/score/assert_judge_presets.py`	Loads and merges judge presets.
`pyrit/score/__init__.py`	Exports ASSERT and tracing APIs lazily.
`pyrit/scenario/core/scenario.py`	Extracts scenario identity parameter selection.
`pyrit/prompt_target/common/target_trace_config.py`	Adds conversation identifiers to request spans.
`pyrit/prompt_target/common/prompt_target.py`	Exposes target tracing configuration.
`pyrit/models/score/trace.py`	Defines execution steps, content, and summaries.
`pyrit/models/score/observation.py`	Adds validated execution-trace observations.
`pyrit/models/score/assert_judge.py`	Defines dimensions, ordinal scales, and presets.
`pyrit/models/score/__init__.py`	Exports execution-trace models.
`pyrit/models/results/assert_policy_metrics.py`	Defines ASSERT metric response contracts.
`pyrit/models/analytics.py`	Adds binary-rate and macro-rate models.
`pyrit/models/__init__.py`	Exports metrics, judge, and trace models.
`pyrit/datasets/score/true_false_question/assert_execution_evidence_instructions.yaml`	Guides judges on trace evidence and citations.
`pyrit/datasets/score/assert/judge_presets/tool-use.yaml`	Adds tool-use judging dimensions.
`pyrit/datasets/score/assert/judge_presets/safety-extended.yaml`	Adds harm and privacy dimensions.
`pyrit/datasets/score/assert/judge_presets/safety-core.yaml`	References built-in safety dimensions.
`pyrit/datasets/score/assert/judge_presets/robustness.yaml`	Adds injection and boundary dimensions.
`pyrit/datasets/score/assert/judge_presets/policy-adherence.yaml`	Adds policy-adherence judging.
`pyrit/datasets/score/assert/judge_presets/operational.yaml`	Adds workflow and escalation dimensions.
`pyrit/datasets/score/assert/judge_presets/multi-turn.yaml`	Adds context-retention and goal dimensions.
`pyrit/datasets/score/assert/judge_presets/instruction-following.yaml`	Adds constraint and completion dimensions.
`pyrit/datasets/score/assert/judge_presets/grounding.yaml`	Adds grounding and attribution dimensions.
`pyrit/datasets/score/assert/judge_presets/communication.yaml`	Adds communication-quality judging.
`pyrit/datasets/score/assert/judge_presets/alignment.yaml`	Adds honesty and verification dimensions.
`pyrit/datasets/score/assert/dimension_system_prompt.yaml`	Defines the custom-dimension judge prompt.
`pyrit/backend/routes/scenarios.py`	Adds the ASSERT metrics endpoint.
`pyrit/backend/README.md`	Documents endpoint settings and responses.
`pyrit/analytics/rate_statistics.py`	Computes Wilson intervals and macro averages.
`pyrit/analytics/__init__.py`	Exports ASSERT metrics and rate helpers.
`frontend/src/types/index.ts`	Adds typed metrics API contracts.
`frontend/src/services/api.ts`	Adds the metrics request method.
`frontend/src/hooks/useAssertPolicyMetrics.ts`	Loads metrics with cancellation and retry.
`frontend/src/hooks/useAssertPolicyMetrics.test.tsx`	Tests loading, retry, and stale-request handling.
`frontend/src/components/Scenarios/ScenarioRunPage.tsx`	Shows metrics for completed ASSERT runs.
`frontend/src/components/Scenarios/ScenarioRunPage.test.tsx`	Tests conditional panel visibility.
`frontend/src/components/Scenarios/assertPolicyRun.ts`	Identifies ASSERT scenario runs.
`frontend/src/components/Scenarios/AssertPolicyMetricsPanel.test.tsx`	Tests metric presentation and breakdowns.
`frontend/src/components/Scenarios/AssertPolicyMetricsPanel.styles.ts`	Styles metric tables and support indicators.
`doc/scanner/foundry.py`	Explains metrics, judge presets, and traces.
`doc/myst.yml`	Adds trace-scoring documentation navigation.
`doc/code/scoring/1_true_false_scorers.py`	Documents custom dimensions and repeated judgments.
`doc/code/scoring/1_true_false_scorers.ipynb`	Updates matching scorer notebook documentation.
`doc/code/scoring/0_scoring.py`	Links to execution-trace scoring guidance.
`doc/code/scoring/0_scoring.ipynb`	Adds the matching notebook link.
`doc/code/framework.md`	Clarifies trace and metrics responsibilities.
`doc/code/analytics/0_attack_results.md`	Documents rate helpers and ASSERT metrics.

_{Review effort: Balanced}

+    seed_id = atomic_attack_name.rsplit("__", 1)[-1] if atomic_attack_name.count("__") >= 2 else None
+    return matches.get(seed_id) if seed_id is not None else None


+    if reason is not None or objective is None:
+        return _Row(technique=technique, not_judged_reason=reason, context=context)


+    Returns:
+        AssertPolicyMetrics: Per-technique permissibility split, judge dimensions, and breakdowns.
+    """
+    result = await get_scenario_run_results(scenario_result_id=scenario_result_id)


…converters - Custom judge dimensions are boolean bad-event flags judged by SelfAskTrueFalseScorer with an ASSERT dimension template; ASSERT's ordinal scale and allow_not_applicable options are rejected. - Repeated samples use TrueFalseCompositeScorer over the repeated judge with an ASSERT vote (ties count as failures, undetermined samples excluded, category re-vote with override warning), replacing the bespoke sampled scorers. - Public exports trimmed to AssertJudgeDimension, AssertJudgePreset, AssertSampleMetadata, and the preset list/load functions. - foundry.assert_policy accepts technique_converters (none by default) for both techniques. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- AssertTechnique is removed. The technique class is built from the AttackTechniqueRegistry, with red_teaming as the default (ASSERT's scenario case). - The baseline, now on by default, sends each group's stored system and user prompts (ASSERT's prompt case). - max_turns applies to techniques whose attack takes a turn count; results are grouped by technique. - AssertPolicy.VERSION is 2, so runs from version 1 do not resume. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- allow_not_applicable: a not-applicable answer is undetermined with assert_not_applicable metadata; repeated samples exclude it and keep the marker only when every sample is not applicable. - ASSERT's ordinal scale is accepted with a PyRIT violation_at grade. The judge's grade position is thresholded by FloatScaleThresholdScorer, so every dimension still yields a true/false verdict, and the grade is kept in assert_grade metadata. Graded dimensions can be the objective. - AssertDimensionMetadata is exported from pyrit.score. Default identities are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ence ASSERT judges can read what the application did, not only what it said: tool calls with arguments and results, model calls, and agent steps captured with OpenTelemetry, rendered after the turns they produced with citable step IDs. - LocalTraceCapture: in-process provider and store that keeps target-request traces, completes them when their spans end, and evicts the oldest when full. - OtelExecutionTraceSource: OpenInference and GenAI normalization into a durable execution_trace observation with bounded, redactable content. - ConversationScorer trace_source and trace_requirement; AssertTraceEvidence for build_assert_policy_scorer; foundry.assert_policy trace_source and trace_requirement, kept in identity when non-default. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Roman Lutz (romanlutz)

Review of the ASSERT headline metrics commit (incremental over #4130). No correctness bugs found; 90/90 unit tests pass locally and the semantics match ASSERT's split and Wilson intervals. Comments below are polish/design.

Roman Lutz (romanlutz) · 2026-11-10T08:44:00Z

+        rows_without_node_judgments=sum(row.nodes is None for row in judged),
+        policy_violation_on_not_permissible=_flags(split[0] for split in splits) if splits else None,
+        policy_violation_on_permissible=_flags(split[1] for split in splits) if splits else None,
+        dimensions={name: _flags(row.verdicts[name] for row in judged if name in row.verdicts) for name in names},


A judged conversation whose auxiliary-dimension score is undetermined (e.g. only that judge was blocked) is silently dropped from the dimension's rate: it shows up in neither applicable_count nor not_applicable_count (which is always 0 for dimension rates). The GUI then can't distinguish "3 of 3 judged" from "3 of 4, one judge gave no verdict". Counting those rows in not_applicable_count would make applicable + not_applicable == scored_total hold for every dimension. If this stays as is, it changes the public JSON shape to fix later — worth deciding now.

Roman Lutz (romanlutz) · 2026-11-10T08:44:00Z

+    distinct = set(matches.values())
+    if len(distinct) == 1:
+        return next(iter(distinct))
+    seed_id = atomic_attack_name.rsplit("__", 1)[-1] if atomic_attack_name.count("__") >= 2 else None


This count("__") >= 2 / rsplit("__", 1) parse assumes technique names and logical IDs never contain __, mirroring _get_unit_name's suffix scheme. Registry technique names are user-definable, so a my__technique name would mis-parse (degrading safely to unresolved_policy, so no wrong data). A comment noting the coupling — or recording the seed ID in attribution data instead of the name in a follow-up — would remove the fragility.

Roman Lutz (romanlutz) · 2026-11-10T08:44:00Z

+        name = _dimension_name(score)
+        if not name or name == objective_name:
+            continue
+        if name not in latest or score.timestamp >= latest[name].timestamp:


Nit: with equal timestamps the winner depends on memory return order. (score.timestamp, str(score.id)) as the comparison key would make re-judged dimensions deterministic.

Roman Lutz (romanlutz) · 2026-11-10T08:44:00Z

+    model_config = ConfigDict(extra="forbid")
+
+
+class BinaryRate(_AssertMetricsModel):


BinaryRate is exported at the pyrit.models top level, but its docstring and semantics bake in ASSERT's 90% interval and minimum support of 5. If a later generic metric wants a binary rate with a different alpha, the good name is taken. Consider AssertBinaryRate, or parameterizing the confidence/min-support so the generic name is earned.

Roman Lutz (romanlutz) · 2026-11-10T08:44:00Z

+  return (
+    <div className={styles.rateCell}>
+      <Text weight="semibold">{`${(rate.rate * 100).toFixed(1)}%`}</Text>
+      <Text size={200} className={styles.hint}>{`${rate.flagged_count} / ${rate.applicable_count}`}</Text>


ci_lower/ci_upper are computed for every rate but never shown in the GUI — only the point rate, counts, and the n<5 badge. A tooltip or secondary line (e.g. "90% CI 12.7–55.8%") would make the headline numbers honest at small n; the SDK/JSON already has the values.

Roman Lutz (romanlutz) · 2026-11-10T08:44:00Z

+    assert compute_scenario_statistics(result).overall.completed == metrics.total
+
+
+async def test_direct_mode_reports_dimensions_without_split_async(harness: _Harness) -> None:


No test produces judgment_mode == "mixed" (some rows direct, some category_validated), so the ternary in _aggregate is untested. A cheap case with one direct-mode score patched in would cover it.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Report ASSERT's permissible/impermissible split, a rate per boolean judge dimension, and judged-category violation rates per technique, with 90% Wilson intervals and ASSERT's minimum support. Expose them through a backend route and the GUI run page. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen) and others added 2 commits November 9, 2026 18:30

Copilot AI balanced review requested due to automatic review settings November 10, 2026 06:28

Copilot started reviewing on behalf of Richard Lundeen (richlundeen) November 10, 2026 06:29

View session

Copilot AI reviewed Nov 10, 2026

View reviewed changes

Richard Lundeen (richlundeen) marked this pull request as draft November 10, 2026 06:38

Richard Lundeen (richlundeen) and others added 2 commits November 9, 2026 23:04

Richard Lundeen (richlundeen) force-pushed the richlundeen-assert-headline-metrics-plan branch from fc96974 to d531d93 Compare November 10, 2026 08:22

Roman Lutz (romanlutz) self-assigned this Nov 10, 2026

Richard Lundeen (richlundeen) and others added 2 commits November 10, 2026 00:39

Roman Lutz (romanlutz) reviewed Nov 10, 2026

View reviewed changes

Richard Lundeen (richlundeen) force-pushed the richlundeen-assert-headline-metrics-plan branch from d531d93 to 1c727cc Compare November 10, 2026 09:11

Richard Lundeen (richlundeen) and others added 2 commits November 10, 2026 02:07

FIX: Order ASSERT trace steps deterministically when start times tie

5e3676b

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen) mentioned this pull request Nov 10, 2026

DOC: In-process agents as custom targets (-7) #4144

Draft

Richard Lundeen (richlundeen) force-pushed the richlundeen-assert-headline-metrics-plan branch from 1c727cc to fc44aeb Compare November 10, 2026 10:14

Navigation Menu

Repository navigation

FEAT: ASSERT headline metrics (-3) - #4131

FEAT: ASSERT headline metrics (-3)#4131
Richard Lundeen (richlundeen) wants to merge 8 commits into
microsoft:mainfrom
richlundeen:richlundeen-assert-headline-metrics-plan

Richard Lundeen (richlundeen) commented Nov 10, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Roman Lutz (romanlutz) left a comment

Uh oh!

Roman Lutz (romanlutz) Nov 10, 2026

Uh oh!

Roman Lutz (romanlutz) Nov 10, 2026

Uh oh!

Roman Lutz (romanlutz) Nov 10, 2026

Uh oh!

Roman Lutz (romanlutz) Nov 10, 2026

Uh oh!

Roman Lutz (romanlutz) Nov 10, 2026

Uh oh!

Roman Lutz (romanlutz) Nov 10, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		seed_id = atomic_attack_name.rsplit("__", 1)[-1] if atomic_attack_name.count("__") >= 2 else None
		return matches.get(seed_id) if seed_id is not None else None

		if reason is not None or objective is None:
		return _Row(technique=technique, not_judged_reason=reason, context=context)

		model_config = ConfigDict(extra="forbid")


		class BinaryRate(_AssertMetricsModel):

		assert compute_scenario_statistics(result).overall.completed == metrics.total


		async def test_direct_mode_reports_dimensions_without_split_async(harness: _Harness) -> None:

Conversation

Richard Lundeen (richlundeen) commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Roman Lutz (romanlutz) left a comment

Choose a reason for hiding this comment

Uh oh!

Roman Lutz (romanlutz) Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

Roman Lutz (romanlutz) Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

Roman Lutz (romanlutz) Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

Roman Lutz (romanlutz) Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

Roman Lutz (romanlutz) Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

Roman Lutz (romanlutz) Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Richard Lundeen (richlundeen) commented Nov 10, 2026 •

edited

Loading