Repository navigation
FEAT: Generate datasets from a policy in the GUI (-6) - #4135
Draft
Richard Lundeen (richlundeen) wants to merge 10 commits into
Draft
Richard Lundeen (richlundeen) wants to merge 10 commits into
Richard Lundeen (richlundeen) wants to merge 10 commits into
Conversation
Adds ASSERT judge dimension/preset models, ships ASSERT's 11 judge presets, judges custom boolean and ordinal dimensions (with not-applicable) through build_assert_policy_scorer, aggregates repeated judge samples as ASSERT does, and exposes judge_presets, disabled_judge_dimensions, judge_samples, and objective_dimension on foundry.assert_policy. Default identities are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ge presets to built-in names - objective_dimension and other non-default judge settings are recorded in scenario identity params, so resume detects changes and saved GUI/API launches restore them. Default identities are unchanged. - Mixed boolean/ordinal judge sets are ordered by policy for a stable identity. - judge_presets accepts only shipped preset names; preset files move to the Python-only judge_preset_files (list[Path]) parameter, which the API guard rejects. Preset load errors no longer echo file contents. - Adds a Scenario._get_identity_params hook. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved issues can leave draft submissions stuck after a failed start, prevent generation data from refreshing when runtime availability changes, and block regeneration for maximum-length dataset names.
3 open issues
What changed in this PR
This PR connects CoPyRIT’s dataset catalog to prompt generation, allowing users to create policy-based datasets, review generated policy details, and launch scenarios with those datasets selected.
Changes:
- Adds generation navigation, reusable generator views, preflight checks, and dataset regeneration links.
- Bundles ASSERT behavior and suite presets, with supporting generation, scoring, tracing, and analytics interfaces.
- Adds frontend coverage, isolated browser-test configuration, CI wiring, and user documentation.
| File | Description |
|---|---|
| THIRD_PARTY_NOTICES.txt | Updates ASSERT redistribution attribution. |
| tests/unit/executor/promptgen/test_target_objective_generator.py | Tests generation estimates and validation. |
| tests/unit/common/test_lazy_package_imports.py | Updates analytics export expectations. |
| tests/unit/backend/test_prompt_generation_fixture.py | Checks offline generator responses. |
| tests/unit/assert_mocks.py | Adds checkpoint and generated-group fixtures. |
| pyrit/score/true_false/true_false_fallback_scorer.py | Exposes configurable fallback triggers. |
| pyrit/score/scorer.py | Identifies missing-evidence results. |
| pyrit/score/observation/__init__.py | Exports execution-trace observation helpers. |
| pyrit/score/float_scale/float_scale_fallback_scorer.py | Exposes configurable fallback triggers. |
| pyrit/scenario/core/scenario.py | Extracts scenario identity parameter handling. |
| pyrit/prompt_target/common/target_trace_config.py | Attaches conversation identifiers to spans. |
| pyrit/prompt_target/common/prompt_target.py | Exposes target tracing configuration. |
| pyrit/models/score/__init__.py | Exports execution-trace models. |
| pyrit/models/analytics.py | Adds binary and macro rate models. |
| pyrit/executor/promptgen/target_objective_generator.py | Adds call and token estimates. |
| pyrit/executor/promptgen/__init__.py | Exports objective generation results. |
| pyrit/datasets/seed_datasets/generated/target_objective_provider.py | Extracts reusable result packaging. |
| pyrit/datasets/score/true_false_question/assert_execution_evidence_instructions.yaml | Defines trace-evidence judging instructions. |
| pyrit/datasets/score/assert/judge_presets/tool-use.yaml | Adds tool-use judging dimensions. |
| pyrit/datasets/score/assert/judge_presets/safety-extended.yaml | Adds harm and privacy dimensions. |
| pyrit/datasets/score/assert/judge_presets/safety-core.yaml | References core safety dimensions. |
| pyrit/datasets/score/assert/judge_presets/robustness.yaml | Adds robustness judging dimensions. |
| pyrit/datasets/score/assert/judge_presets/policy-adherence.yaml | Adds policy-adherence judging. |
| pyrit/datasets/score/assert/judge_presets/operational.yaml | Adds workflow and escalation judging. |
| pyrit/datasets/score/assert/judge_presets/multi-turn.yaml | Adds conversation continuity judging. |
| pyrit/datasets/score/assert/judge_presets/instruction-following.yaml | Adds instruction-following judging. |
| pyrit/datasets/score/assert/judge_presets/grounding.yaml | Adds grounding and attribution judging. |
| pyrit/datasets/score/assert/judge_presets/communication.yaml | Adds communication-quality judging. |
| pyrit/datasets/score/assert/judge_presets/alignment.yaml | Adds honesty and verification judging. |
| pyrit/datasets/score/assert/dimension_system_prompt.yaml | Adds custom-dimension judge template. |
| pyrit/datasets/assert_suites/travel_planner.yaml | Bundles travel-planner suite. |
| pyrit/datasets/assert_suites/travel_planner_benchmark.yaml | Bundles quality-only travel suite. |
| pyrit/datasets/assert_suites/telecom_customer_service.yaml | Bundles telecom support suite. |
| pyrit/datasets/assert_behaviors/unsupported_conclusion_failures.yaml | Bundles unsupported-conclusion behavior. |
| pyrit/datasets/assert_behaviors/unit_conversion_failures.yaml | Bundles unit-conversion behavior. |
| pyrit/datasets/assert_behaviors/underused_context_failures.yaml | Bundles context-underuse behavior. |
| pyrit/datasets/assert_behaviors/unauthorized_action_failures.yaml | Bundles unauthorized-action behavior. |
| pyrit/datasets/assert_behaviors/tool_parameter_formatting_failures.yaml | Bundles tool-argument behavior. |
| pyrit/datasets/assert_behaviors/tool_output_misinterpretation_failures.yaml | Bundles tool-output interpretation behavior. |
| pyrit/datasets/assert_behaviors/tool_orchestration_errors.yaml | Bundles tool-orchestration behavior. |
| pyrit/datasets/assert_behaviors/tool_call_turn_protocol_failures.yaml | Bundles tool-turn protocol behavior. |
| pyrit/datasets/assert_behaviors/tool_call_error_recovery_failures.yaml | Bundles tool-error recovery behavior. |
| pyrit/datasets/assert_behaviors/tabs_vs_spaces.yaml | Bundles indentation preference-bias behavior. |
| pyrit/datasets/assert_behaviors/sycophancy.yaml | Bundles sycophancy behavior. |
| pyrit/datasets/assert_behaviors/suicide_self_harm.yaml | Bundles self-harm safety behavior. |
| pyrit/datasets/assert_behaviors/success_criteria_ambiguity_failures.yaml | Bundles success-criteria ambiguity behavior. |
| pyrit/datasets/assert_behaviors/stereotyping.yaml | Bundles stereotyping behavior. |
| pyrit/datasets/assert_behaviors/stale_state_failures.yaml | Bundles stale-state behavior. |
| pyrit/datasets/assert_behaviors/shimano_vs_sram.yaml | Bundles cycling-brand preference behavior. |
| pyrit/datasets/assert_behaviors/response_completeness_failures.yaml | Bundles response-completeness behavior. |
| pyrit/datasets/assert_behaviors/repeated_action_loop_failures.yaml | Bundles repeated-action behavior. |
| pyrit/datasets/assert_behaviors/prompt_injection.yaml | Bundles prompt-injection behavior. |
| pyrit/datasets/assert_behaviors/procedure_adherence_failures.yaml | Bundles procedure-adherence behavior. |
| pyrit/datasets/assert_behaviors/premature_termination_failures.yaml | Bundles premature-termination behavior. |
| pyrit/datasets/assert_behaviors/poor_retrieval_failures.yaml | Bundles retrieval-quality behavior. |
| pyrit/datasets/assert_behaviors/output_internal_consistency_failures.yaml | Bundles output-consistency behavior. |
| pyrit/datasets/assert_behaviors/out_of_scope_request_failures.yaml | Bundles scope-boundary behavior. |
| pyrit/datasets/assert_behaviors/observation_neglect_failures.yaml | Bundles observation-neglect behavior. |
| pyrit/datasets/assert_behaviors/ios_vs_android.yaml | Bundles mobile-platform preference behavior. |
| pyrit/datasets/assert_behaviors/inter_agent_handoff_failures.yaml | Bundles agent-handoff behavior. |
| pyrit/datasets/assert_behaviors/intent_misinterpretation_failures.yaml | Bundles intent-interpretation behavior. |
| pyrit/datasets/assert_behaviors/insufficient_verification_failures.yaml | Bundles insufficient-verification behavior. |
| pyrit/datasets/assert_behaviors/ineffective_team_communication_failures.yaml | Bundles agent-communication behavior. |
| pyrit/datasets/assert_behaviors/incorrect_tool_selection_failures.yaml | Bundles tool-selection behavior. |
| pyrit/datasets/assert_behaviors/incomplete_answer_synthesis_failures.yaml | Bundles answer-synthesis behavior. |
| pyrit/datasets/assert_behaviors/identity_verification_failures.yaml | Bundles identity-verification behavior. |
| pyrit/datasets/assert_behaviors/harmful_medical_advice.yaml | Bundles medical-safety behavior. |
| pyrit/datasets/assert_behaviors/grounding_attribution_errors.yaml | Bundles grounding-error behavior. |
| pyrit/datasets/assert_behaviors/goal_drift_failures.yaml | Bundles goal-drift behavior. |
| pyrit/datasets/assert_behaviors/flawed_action_plan_failures.yaml | Bundles planning-failure behavior. |
| pyrit/datasets/assert_behaviors/explicit_constraint_violation_failures.yaml | Bundles constraint-violation behavior. |
| pyrit/datasets/assert_behaviors/escalation_judgment_failures.yaml | Bundles escalation-decision behavior. |
| pyrit/datasets/assert_behaviors/doxxing.yaml | Bundles private-information protection behavior. |
| pyrit/datasets/assert_behaviors/crystal_mountain_vs_snoqualmie.yaml | Bundles resort preference-bias behavior. |
| pyrit/datasets/assert_behaviors/conversation_coherence_breakdown.yaml | Bundles conversation-coherence behavior. |
| pyrit/datasets/assert_behaviors/constraint_propagation_failures.yaml | Bundles cross-agent constraint behavior. |
| pyrit/datasets/assert_behaviors/conflicting_instruction_resolution_failures.yaml | Bundles instruction-conflict behavior. |
| pyrit/datasets/assert_behaviors/actionability_failures.yaml | Bundles answer-actionability behavior. |
| pyrit/datasets/__init__.py | Exports ASSERT preset accessors. |
| pyrit/common/path.py | Defines bundled preset locations. |
| pyrit/backend/services/service_lifecycle.py | Resets generation service during cleanup. |
| pyrit/backend/services/runtime_lifecycle.py | Includes generation work in runtime lifecycle. |
| pyrit/backend/main.py | Registers prompt-generation routes. |
| pyrit/analytics/__init__.py | Exports ASSERT metrics and rate helpers. |
| frontend/src/utils/routeParams.ts | Adds dataset-preselected launch URLs. |
| frontend/src/utils/routeParams.test.ts | Tests dataset URL encoding and parsing. |
| frontend/src/test-utils/replaceText.ts | Adds efficient text replacement helper. |
| frontend/src/hooks/useRequestedDatasets.ts | Validates requested dataset selections. |
| frontend/src/hooks/usePromptGenerators.ts | Loads runtime-specific generator catalog. |
| frontend/src/hooks/usePromptGenerationPreflight.ts | Debounces validation and estimate requests. |
| frontend/src/hooks/usePromptGeneration.ts | Loads and polls generation records. |
| frontend/src/hooks/useAssertPolicyMetrics.ts | Loads completed-run ASSERT metrics. |
| frontend/src/hooks/useAssertPolicyMetrics.test.tsx | Tests metrics loading and retry behavior. |
| frontend/src/hooks/useAssertLibrary.ts | Loads bundled behavior and suite templates. |
| frontend/src/components/Scenarios/ScenarioRunPage.tsx | Displays completed ASSERT run metrics. |
| frontend/src/components/Scenarios/ScenarioRunPage.test.tsx | Tests conditional metrics display. |
| frontend/src/components/Scenarios/scenarioLaunchForm.ts | Seeds dataset selections from links. |
| frontend/src/components/Scenarios/assertPolicyRun.ts | Identifies ASSERT policy runs. |
| frontend/src/components/Scenarios/AssertPolicyMetricsPanel.styles.ts | Styles metrics presentation. |
| frontend/src/components/PromptGeneration/TargetObjectiveSummary.tsx | Displays generated objective counts. |
| frontend/src/components/PromptGeneration/RegenerateDatasetLink.tsx | Links generated datasets to regeneration. |
| frontend/src/components/PromptGeneration/RegenerateDatasetLink.test.tsx | Tests regeneration link availability. |
| frontend/src/components/PromptGeneration/PromptGenerationStatusBadge.tsx | Displays generation status badges. |
| frontend/src/components/PromptGeneration/promptGenerationRoutes.ts | Defines generation routes and query parameters. |
| frontend/src/components/PromptGeneration/promptGenerationRoutes.test.ts | Tests generation route helpers. |
| frontend/src/components/PromptGeneration/GenericReviewStep.tsx | Provides a generic review continuation view. |
| frontend/src/components/PromptGeneration/GenericGenerationSummary.tsx | Displays generic scalar result summaries. |
| frontend/src/components/PromptGeneration/generatorUi.ts | Defines generator UI extension interfaces. |
| frontend/src/components/PromptGeneration/generatorRegistry.tsx | Selects specialized or generic generator views. |
| frontend/src/components/PromptGeneration/GenerationTargetFields.test.tsx | Tests target selection and errors. |
| frontend/src/components/PromptGeneration/GenerationSummaryFacts.tsx | Renders reusable result facts. |
| frontend/src/components/PromptGeneration/generationFormHelpers.test.ts | Tests form validation helpers. |
| frontend/src/components/PromptGeneration/GenerateDatasetLinks.tsx | Links scenarios to matching generators. |
| frontend/src/components/Datasets/DatasetSelection.tsx | Adds dataset regeneration entry point. |
| frontend/src/components/Datasets/DatasetSelection.test.tsx | Tests dataset regeneration integration. |
| frontend/src/components/Datasets/DatasetCatalog.tsx | Adds generation navigation button. |
| frontend/src/components/Datasets/DatasetCatalog.test.tsx | Tests generation navigation. |
| frontend/src/App.tsx | Registers generation pages. |
| frontend/README.md | Documents isolated browser checks. |
| frontend/playwright.prompt-generation.config.ts | Configures offline generation browser tests. |
| frontend/playwright.config.ts | Excludes the dedicated generation spec. |
| frontend/package.json | Adds generation browser-test command. |
| doc/myst.yml | Adds trace-scoring documentation navigation. |
| doc/gui/0_gui.md | Documents GUI dataset generation. |
| doc/code/scoring/1_true_false_scorers.py | Documents custom ASSERT judging. |
| doc/code/scoring/1_true_false_scorers.ipynb | Mirrors custom judging documentation. |
| doc/code/scoring/0_scoring.py | Links execution-trace scoring guidance. |
| doc/code/scoring/0_scoring.ipynb | Mirrors trace-scoring guidance link. |
| doc/code/executor/7_promptgen.py | Demonstrates checkpoint review and continuation. |
| doc/code/analytics/0_attack_results.md | Documents rates and ASSERT policy metrics. |
| .github/workflows/frontend_tests.yml | Adds isolated generation browser-test CI job. |
Review effort: Balanced
Comment on lines
+78
to
+79
| const next = match ? `${match[1]}_v${Number(match[2]) + 1}` : `${name}_v2` | ||
| return datasetNameFromText(next) |
| if (timer !== null) clearTimeout(timer) | ||
| controller?.abort() | ||
| } | ||
| }, [recordKey, keyValid, load, isActive, intervalMs, epoch]) |
| } | ||
| if (draft) { | ||
| let updated = await promptGenerationsApi.update(draft.id, { | ||
| expected_version: draft.version, |
…converters - Custom judge dimensions are boolean bad-event flags judged by SelfAskTrueFalseScorer with an ASSERT dimension template; ASSERT's ordinal scale and allow_not_applicable options are rejected. - Repeated samples use TrueFalseCompositeScorer over the repeated judge with an ASSERT vote (ties count as failures, undetermined samples excluded, category re-vote with override warning), replacing the bespoke sampled scorers. - Public exports trimmed to AssertJudgeDimension, AssertJudgePreset, AssertSampleMetadata, and the preset list/load functions. - foundry.assert_policy accepts technique_converters (none by default) for both techniques. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- AssertTechnique is removed. The technique class is built from the AttackTechniqueRegistry, with red_teaming as the default (ASSERT's scenario case). - The baseline, now on by default, sends each group's stored system and user prompts (ASSERT's prompt case). - max_turns applies to techniques whose attack takes a turn count; results are grouped by technique. - AssertPolicy.VERSION is 2, so runs from version 1 do not resume. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This was referenced Nov 10, 2026
- allow_not_applicable: a not-applicable answer is undetermined with assert_not_applicable metadata; repeated samples exclude it and keep the marker only when every sample is not applicable. - ASSERT's ordinal scale is accepted with a PyRIT violation_at grade. The judge's grade position is thresholded by FloatScaleThresholdScorer, so every dimension still yields a true/false verdict, and the grade is kept in assert_grade metadata. Graded dimensions can be the objective. - AssertDimensionMetadata is exported from pyrit.score. Default identities are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ence ASSERT judges can read what the application did, not only what it said: tool calls with arguments and results, model calls, and agent steps captured with OpenTelemetry, rendered after the turns they produced with citable step IDs. - LocalTraceCapture: in-process provider and store that keeps target-request traces, completes them when their spans end, and evicts the oldest when full. - OtelExecutionTraceSource: OpenInference and GenAI normalization into a durable execution_trace observation with bounded, redactable content. - ConversationScorer trace_source and trace_requirement; AssertTraceEvidence for build_assert_policy_scorer; foundry.assert_policy trace_source and trace_requirement, kept in identity when non-default. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Roman Lutz (romanlutz)
left a comment
Contributor
There was a problem hiding this comment.
Reviewed as the incremental diff vs richlundeen-assert-generation-backend (PR #4134), ~9k added lines. Overall this is high-quality frontend work: the TS types mirror the backend pydantic models exactly, the API client matches every route, validation replicates the backend validators, and the e2e setup (real FastAPI app over an offline target, plus a contract-pinning unit test) is notably better than a hand-rolled fake. Inline comments cover the bugs; a few broader notes:
Design
- The three generator forms triplicate an identical ~40-line shell (state, submit, preflight, footer plumbing); a shared hook would keep the preflight request shape in one place.
AssertGenerationFormContentis ~530 lines with 15useStates; the suite section (4 parallel states + table) wants to be its own component.- Polling stops on
awaiting_review, so a 'Review needed' page/list never refreshes if the review is continued elsewhere; Continue then 409s with no reload path. - Web-search capability is handled asymmetrically: the research-role filter requires
supports_web_search === true(targets withoutcapabilitiesare hidden), while the generation-target fallback only rejects=== false.
Test gaps
- The preflight transport-failure path (warning + still-enabled Generate + estimate card state) has zero coverage.
- Suite-mode validation (duplicate dataset names, 100-dataset cap, label length, cleared name) is untested.
usePolledRecord's subtlest branches are untested: 404 arriving mid-poll, unmount mid-fetch, a changingrecordKey.assertDraftFromRequestis never tested with a stored stratified request.- The e2e journey never exercises cancel/retry/delete/draft/group against the real backend; given the fixture serves the real routes, a short cancel-then-retry journey would be cheap insurance.
| export function aggregateGenerationStatus(items: readonly PromptGenerationSummary[]): PromptGenerationStatus { | ||
| const statuses = new Set(items.map((item) => item.status)) | ||
| const order: readonly PromptGenerationStatus[] = [ | ||
| 'awaiting_review', 'running', 'saving', 'queued', 'failed', 'interrupted', 'cancelled', 'draft', |
Contributor
There was a problem hiding this comment.
This priority order disagrees with the backend's in _group_response (prompt_generation_service.py), which ranks failed/interrupted right after awaiting_review and draft before cancelled. A suite with one failed and one running member shows Running on the generations list (client-side aggregate) but Failed on the group page (server-provided group.status). One ordering should be canonical — mirror the server's here, or have the list endpoint return group status.
| ) | ||
| const [suiteDatasetNames, setSuiteDatasetNames] = useState<Record<string, string>>({}) | ||
| const [selectedTargets, setSelectedTargets] = useState( | ||
| () => initialTargetSelection(descriptor, seed.targets, defaultTarget), |
Contributor
There was a problem hiding this comment.
defaultTarget is captured once in the useState initializer, but NewPromptGenerationPage gates rendering only on generators.loading || source.loading, not targetsLoading. On a hard refresh of /datasets/generate/new?generator=assert the generators request can resolve before the target registry, so the form mounts with defaultTarget = null and the required target stays on "Select a target" even after targets load (only server preflight catches it). Same pattern in GenericGenerationForm and TargetObjectiveGenerationForm. Gating the page on targetsLoading would fix the deep-link case; the tests always pass targetsLoading={false} with a non-null default, which is why this never shows up.
| targets, | ||
| pause_for_review: pauseForReview, | ||
| }) | ||
| if (start) updated = await promptGenerationsApi.start(updated.id) |
Contributor
There was a problem hiding this comment.
If the PATCH succeeds but start fails transiently, draft.version is now stale (the update bumped it server-side). The next Generate click sends the old expected_version, gets a 409, and the message blames a concurrent edit ("This draft changed elsewhere") when only this form touched it. Worth carrying updated.version into the failure path, or navigating to the draft page where retry works.
| datasetCount={datasetCount} | ||
| emptyMessage={clientErrors.length > 0 | ||
| ? 'Complete the required fields to see an estimate.' | ||
| : preflight.status === 'ready' ? 'No estimate is available for this request.' : 'Calculating the estimate...'} |
Contributor
There was a problem hiding this comment.
When preflight.status === 'error' this falls through to "Calculating the estimate..." — and since the failed snapshot is cached by key and never retried, the card claims to be calculating forever, right next to the "could not be checked" warning. An explicit no-estimate branch for the error status would read better. (Relatedly, the preflight transport-failure path has no test.)
| const missingSuite = seed.suite && !library.error && !library.suites.some((item) => item.name === seed.suite) | ||
| ? seed.suite : null | ||
| const singleDatasetName = datasetNameTouched ? datasetName : datasetNameFromText(behavior.behaviorName) | ||
| const suitePrefix = suiteName || suiteLabel |
Contributor
There was a problem hiding this comment.
Once a suite template is picked, suitePrefix is suiteName, so editing the suite label no longer updates the default dataset names: renaming a travel_planner run to run2 still proposes travel_planner_* names, which collide with the first run and surface only as per-row "already exists" preflight errors. In custom-selection mode (suiteName === '') the names do follow the label. Is the asymmetry intended?
| /** Suggests the next name for a regenerated dataset: `name` becomes `name_v2`, `name_v2` becomes `name_v3`. */ | ||
| export function nextDatasetName(name: string): string { | ||
| const match = VERSION_SUFFIX_PATTERN.exec(name) | ||
| const next = match ? `${match[1]}_v${Number(match[2]) + 1}` : `${name}_v2` |
Contributor
There was a problem hiding this comment.
At the 128-char max with no version suffix, ${name}_v2 gets truncated back to 128 by datasetNameFromText, returning the original name — so "Edit & regenerate" prefills the already-taken name and preflight blocks until the user renames by hand. Trimming the base before appending the suffix keeps the suggestion usable.
| schedule() | ||
| return | ||
| } | ||
| const previous = latestRef.current |
Contributor
There was a problem hiding this comment.
The hook never resets latestRef/state when recordKey changes, so this fallback can resurface the previous key's record as ready/stale under the new key. All current callers defend by remounting per id (key={decodedId}), but nothing documents or enforces that. Either reset on recordKey change or add the remount requirement to the doc comment next to the existing "load and isActive must be stable" note — and a test either way.
| import { newPromptGenerationRoutePath } from './promptGenerationRoutes' | ||
| import { usePromptGenerationStyles } from './PromptGeneration.styles' | ||
|
|
||
| const NAMED_DATASET_PREFIX = 'dataset:named:' |
Contributor
There was a problem hiding this comment.
This duplicates the dataset:named: format from namedDatasetSelectionKey in utils/routeParams.ts; a shared constant would keep them from drifting. Also, the bare catch {} below treats a transient network failure like "not generator-made", permanently hiding the link for that mount — distinguishing 404 from other failures would avoid that.
| policies. Do not disclose another customer's private order details.", or start | ||
| from a bundled ASSERT behavior. PyRIT researches the behavior, builds a taxonomy | ||
| of permissible and impermissible behaviors, and generates seed groups that cover | ||
| each category across your coverage dimensions. Select **Pause to review** to edit |
Contributor
There was a problem hiding this comment.
The checkbox is labelled "Pause for review after:", not "Pause to review".
| const complete = page.getByTestId('generation-complete') | ||
| const summary = complete.getByTestId('assert-generation-summary') | ||
| await expect(summaryValue(summary, 'Seed groups')).toHaveText('4') | ||
| await expect(summaryValue(summary, 'Categories')).toHaveText('21 permissible, 1 impermissible') |
Contributor
There was a problem hiding this comment.
"21 permissible, 1 impermissible" reads like a typo for 21 — it's the value "2" concatenated with the detail "1 permissible, 1 impermissible". Asserting value and detail separately (or using a regex) would be clearer and less brittle to markup changes.
Report ASSERT's permissible/impermissible split, a rate per boolean judge dimension, and judged-category violation rates per technique, with 90% Wilson intervals and ASSERT's minimum support. Expose them through a backend route and the GUI run page. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…RT behavior library AssertGenerator gains a checkpointed staged API (advance_async, build_result, estimate, validate_config): stop after a stage, apply a reviewed taxonomy and dimensions (recorded in each group's provenance), and resume after interruption without repeating model calls. TargetObjectiveGenerator.estimate and the provider-free PromptGenerationEstimate report remaining calls and a token range. The ASSERT v0.3.0 behavior (52) and suite (3) presets are vendored with loaders, and StorageIO gains list_files_async. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
PromptGenerationService runs generators in the background: preflight and estimates, review pauses, cancel, retry from the last checkpoint, batch creation for suites, and saving a new named dataset. One backend instance owns the records, stored as JSON documents in the memory's results storage. Adapters translate the ASSERT and target-objective generators; routes live under /api/prompt-generators and /api/prompt-generations. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Datasets > Generate dataset estimates, runs, reviews, and retries generations, with an ASSERT form (bundled behavior and suite templates), a taxonomy review editor, and a target-objective form. Suites run as a batch, grouped in the generation list. Completed ASSERT datasets open foundry.assert_policy with repeatable ?dataset= preselection. Includes Jest tests, an isolated Playwright journey over an offline generation target, CI wiring, and GUI docs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Richard Lundeen (richlundeen)
force-pushed
the
richlundeen-assert-gui-generation-plan
branch
from
November 10, 2026 09:58
6feb697 to
b3c7848
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part -6 of a 7-PR ASSERT stack. Design and how to use it: ASSERT in PyRIT.
Each PR targets
mainand contains the PRs above it in this list, so review only this PR's own commits.Description
In CoPyRIT, you can write a policy such as "Explain published return policies. Do not disclose another customer's private order details." or start from a bundled ASSERT behavior. You can then generate and optionally review an ASSERT dataset, and open
foundry.assert_policywith it selected. The same pages also generate objective datasets.?dataset=parameters that preselect stored datasets. Completed ASSERT generations link tofoundry.assert_policywith their datasets selected. Dataset pages gain "Generate dataset" and "Edit policy & regenerate".Review the generated policy before seed groups are written:
The completed generation, with coverage, per-category counts, and the link to run foundry.assert_policy: