Repository navigation
FEAT: Prompt generation jobs in the backend (-5) - #4134
Richard Lundeen (richlundeen) wants to merge 9 commits into
Conversation
Adds ASSERT judge dimension/preset models, ships ASSERT's 11 judge presets, judges custom boolean and ordinal dimensions (with not-applicable) through build_assert_policy_scorer, aggregates repeated judge samples as ASSERT does, and exposes judge_presets, disabled_judge_dimensions, judge_samples, and objective_dimension on foundry.assert_policy. Default identities are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ge presets to built-in names - objective_dimension and other non-default judge settings are recorded in scenario identity params, so resume detects changes and saved GUI/API launches restore them. Default identities are unchanged. - Mixed boolean/ordinal judge sets are ordered by policy for a stable identity. - judge_presets accepts only shipped preset names; preset files move to the Python-only judge_preset_files (list[Path]) parameter, which the API guard rejects. Preset load errors no longer echo file contents. - Adds a Scenario._get_identity_params hook. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Critical storage and cross-backend coordination races can overwrite terminal state, hide jobs, and release dataset-name reservations while original workers remain active, allowing duplicate generation.
5 open issues
Reject cancellation of remotely owned active jobs · New Reserve dataset names atomically across shared-storage backends · New Drain cancelled checkpoint writes before releasing the service lock · New Serialize cache initialization with generation mutations · New Persist and restore generation requests for objective datasets · New
What changed in this PR
This PR adds backend-managed prompt generation jobs so clients can draft, estimate, review, resume, and save generated datasets using PyRIT’s existing generators and dataset providers.
Changes:
- Integrates generation jobs with backend routing, runtime lifecycle, estimates, and dataset packaging.
- Bundles ASSERT behavior and suite presets with attribution.
- Adds supporting ASSERT scoring, trace evidence, analytics, frontend integration, and documentation.
| File | Description |
|---|---|
| THIRD_PARTY_NOTICES.txt | Updates ASSERT attribution. |
| tests/unit/executor/promptgen/test_target_objective_generator.py | Tests generation estimates. |
| tests/unit/datasets/test_target_objective_provider.py | Tests result packaging. |
| tests/unit/common/test_lazy_package_imports.py | Updates analytics export count. |
| tests/unit/assert_mocks.py | Adds checkpoint fixtures. |
| tests/unit/analytics/test_rate_statistics.py | Tests binary and macro rates. |
| pyrit/score/true_false/true_false_fallback_scorer.py | Adds configurable fallback triggers. |
| pyrit/score/scorer.py | Identifies missing-evidence results. |
| pyrit/score/observation/otel_span_exporter.py | Exports richer trace evidence. |
| pyrit/score/observation/message_trace_resolver.py | Resolves conversation trace links. |
| pyrit/score/observation/execution.py | Shares evidence acquisition across scorers. |
| pyrit/score/observation/__init__.py | Exports trace observation APIs. |
| pyrit/score/float_scale/float_scale_fallback_scorer.py | Adds configurable fallback triggers. |
| pyrit/score/__init__.py | Exports ASSERT and tracing APIs. |
| pyrit/scenario/core/scenario.py | Extracts scenario identity parameters. |
| pyrit/prompt_target/common/target_trace_config.py | Adds conversation identifiers to spans. |
| pyrit/prompt_target/common/prompt_target.py | Exposes target tracing configuration. |
| pyrit/models/score/__init__.py | Exports execution-trace models. |
| pyrit/models/prompt_generation_estimate.py | Defines generation estimate contracts. |
| pyrit/models/analytics.py | Adds rate result models. |
| pyrit/executor/promptgen/target_objective_generator.py | Estimates objective-generation work. |
| pyrit/executor/promptgen/__init__.py | Exports objective-generation results. |
| pyrit/datasets/seed_datasets/generated/target_objective_provider.py | Packages existing generation results. |
| pyrit/datasets/score/true_false_question/assert_execution_evidence_instructions.yaml | Guides execution-evidence judgments. |
| pyrit/datasets/score/assert/judge_presets/tool-use.yaml | Adds tool-use judging criteria. |
| pyrit/datasets/score/assert/judge_presets/safety-extended.yaml | Adds extended safety criteria. |
| pyrit/datasets/score/assert/judge_presets/safety-core.yaml | References core safety dimensions. |
| pyrit/datasets/score/assert/judge_presets/robustness.yaml | Adds robustness judging criteria. |
| pyrit/datasets/score/assert/judge_presets/policy-adherence.yaml | Adds policy-adherence criteria. |
| pyrit/datasets/score/assert/judge_presets/operational.yaml | Adds operational judging criteria. |
| pyrit/datasets/score/assert/judge_presets/multi-turn.yaml | Adds multi-turn judging criteria. |
| pyrit/datasets/score/assert/judge_presets/instruction-following.yaml | Adds instruction-following criteria. |
| pyrit/datasets/score/assert/judge_presets/grounding.yaml | Adds grounding judging criteria. |
| pyrit/datasets/score/assert/judge_presets/communication.yaml | Adds communication judging criteria. |
| pyrit/datasets/score/assert/judge_presets/alignment.yaml | Adds alignment judging criteria. |
| pyrit/datasets/score/assert/dimension_system_prompt.yaml | Adds custom-dimension judge template. |
| pyrit/datasets/assert_suites/travel_planner.yaml | Bundles travel-planner suite. |
| pyrit/datasets/assert_suites/travel_planner_benchmark.yaml | Bundles travel-quality benchmark suite. |
| pyrit/datasets/assert_suites/telecom_customer_service.yaml | Bundles telecom support suite. |
| pyrit/datasets/assert_library.py | Loads and validates bundled presets. |
| pyrit/datasets/assert_behaviors/violent_content.yaml | Bundles violence behavior preset. |
| pyrit/datasets/assert_behaviors/unsupported_conclusion_failures.yaml | Bundles unsupported-conclusion preset. |
| pyrit/datasets/assert_behaviors/unit_conversion_failures.yaml | Bundles unit-conversion preset. |
| pyrit/datasets/assert_behaviors/underused_context_failures.yaml | Bundles context-underuse preset. |
| pyrit/datasets/assert_behaviors/unauthorized_action_failures.yaml | Bundles unauthorized-action preset. |
| pyrit/datasets/assert_behaviors/tool_parameter_formatting_failures.yaml | Bundles tool-argument preset. |
| pyrit/datasets/assert_behaviors/tool_output_misinterpretation_failures.yaml | Bundles tool-output interpretation preset. |
| pyrit/datasets/assert_behaviors/tool_orchestration_errors.yaml | Bundles tool-orchestration preset. |
| pyrit/datasets/assert_behaviors/tool_call_turn_protocol_failures.yaml | Bundles tool-turn protocol preset. |
| pyrit/datasets/assert_behaviors/tool_call_error_recovery_failures.yaml | Bundles tool-error recovery preset. |
| pyrit/datasets/assert_behaviors/tabs_vs_spaces.yaml | Bundles indentation-bias preset. |
| pyrit/datasets/assert_behaviors/sycophancy.yaml | Bundles sycophancy preset. |
| pyrit/datasets/assert_behaviors/suicide_self_harm.yaml | Bundles self-harm safety preset. |
| pyrit/datasets/assert_behaviors/success_criteria_ambiguity_failures.yaml | Bundles success-criteria preset. |
| pyrit/datasets/assert_behaviors/stereotyping.yaml | Bundles stereotyping preset. |
| pyrit/datasets/assert_behaviors/stale_state_failures.yaml | Bundles stale-state preset. |
| pyrit/datasets/assert_behaviors/shimano_vs_sram.yaml | Bundles cycling-brand bias preset. |
| pyrit/datasets/assert_behaviors/sexual_content.yaml | Bundles sexual-content safety preset. |
| pyrit/datasets/assert_behaviors/response_completeness_failures.yaml | Bundles response-completeness preset. |
| pyrit/datasets/assert_behaviors/repeated_action_loop_failures.yaml | Bundles repeated-action preset. |
| pyrit/datasets/assert_behaviors/prompt_injection.yaml | Bundles prompt-injection preset. |
| pyrit/datasets/assert_behaviors/procedure_adherence_failures.yaml | Bundles procedure-adherence preset. |
| pyrit/datasets/assert_behaviors/premature_termination_failures.yaml | Bundles premature-termination preset. |
| pyrit/datasets/assert_behaviors/poor_retrieval_failures.yaml | Bundles retrieval-quality preset. |
| pyrit/datasets/assert_behaviors/output_internal_consistency_failures.yaml | Bundles output-consistency preset. |
| pyrit/datasets/assert_behaviors/out_of_scope_request_failures.yaml | Bundles scope-compliance preset. |
| pyrit/datasets/assert_behaviors/observation_neglect_failures.yaml | Bundles observation-neglect preset. |
| pyrit/datasets/assert_behaviors/malicious_cyber_activity.yaml | Bundles cyber-safety preset. |
| pyrit/datasets/assert_behaviors/ios_vs_android.yaml | Bundles platform-bias preset. |
| pyrit/datasets/assert_behaviors/inter_agent_handoff_failures.yaml | Bundles agent-handoff preset. |
| pyrit/datasets/assert_behaviors/intent_misinterpretation_failures.yaml | Bundles intent-interpretation preset. |
| pyrit/datasets/assert_behaviors/insufficient_verification_failures.yaml | Bundles verification-failure preset. |
| pyrit/datasets/assert_behaviors/ineffective_team_communication_failures.yaml | Bundles agent-communication preset. |
| pyrit/datasets/assert_behaviors/incorrect_tool_selection_failures.yaml | Bundles tool-selection preset. |
| pyrit/datasets/assert_behaviors/incomplete_answer_synthesis_failures.yaml | Bundles answer-synthesis preset. |
| pyrit/datasets/assert_behaviors/identity_verification_failures.yaml | Bundles identity-verification preset. |
| pyrit/datasets/assert_behaviors/hate_speech_harassment.yaml | Bundles harassment-safety preset. |
| pyrit/datasets/assert_behaviors/harmful_medical_advice.yaml | Bundles medical-safety preset. |
| pyrit/datasets/assert_behaviors/grounding_attribution_errors.yaml | Bundles grounding-error preset. |
| pyrit/datasets/assert_behaviors/goal_drift_failures.yaml | Bundles goal-drift preset. |
| pyrit/datasets/assert_behaviors/flawed_action_plan_failures.yaml | Bundles action-planning preset. |
| pyrit/datasets/assert_behaviors/explicit_constraint_violation_failures.yaml | Bundles constraint-compliance preset. |
| pyrit/datasets/assert_behaviors/escalation_judgment_failures.yaml | Bundles escalation-decision preset. |
| pyrit/datasets/assert_behaviors/doxxing.yaml | Bundles privacy-safety preset. |
| pyrit/datasets/assert_behaviors/crystal_mountain_vs_snoqualmie.yaml | Bundles destination-bias preset. |
| pyrit/datasets/assert_behaviors/conversation_coherence_breakdown.yaml | Bundles conversation-coherence preset. |
| pyrit/datasets/assert_behaviors/constraint_propagation_failures.yaml | Bundles constraint-propagation preset. |
| pyrit/datasets/assert_behaviors/conflicting_instruction_resolution_failures.yaml | Bundles instruction-conflict preset. |
| pyrit/datasets/assert_behaviors/actionability_failures.yaml | Bundles actionability preset. |
| pyrit/datasets/__init__.py | Exports preset lookup APIs. |
| pyrit/common/path.py | Defines bundled preset paths. |
| pyrit/backend/services/service_lifecycle.py | Resets generation service during cleanup. |
| pyrit/backend/services/runtime_lifecycle.py | Includes generation jobs in lifecycle checks. |
| pyrit/backend/routes/scenarios.py | Exposes ASSERT metrics endpoint. |
| pyrit/backend/README.md | Documents jobs and metrics APIs. |
| pyrit/backend/main.py | Registers generation routes. |
| pyrit/analytics/__init__.py | Exports ASSERT analytics helpers. |
| frontend/src/types/index.ts | Defines ASSERT metrics contracts. |
| frontend/src/services/api.ts | Adds metrics API client. |
| frontend/src/hooks/useAssertPolicyMetrics.ts | Loads metrics with request cancellation. |
| frontend/src/hooks/useAssertPolicyMetrics.test.tsx | Tests metrics loading and retries. |
| frontend/src/components/Scenarios/ScenarioRunPage.tsx | Displays metrics for completed ASSERT runs. |
| frontend/src/components/Scenarios/ScenarioRunPage.test.tsx | Tests conditional metrics display. |
| frontend/src/components/Scenarios/assertPolicyRun.ts | Identifies ASSERT policy runs. |
| frontend/src/components/Scenarios/AssertPolicyMetricsPanel.styles.ts | Styles metrics presentation. |
| doc/myst.yml | Adds trace-scoring navigation. |
| doc/code/scoring/1_true_false_scorers.py | Documents dimensions and sampled judgments. |
| doc/code/scoring/1_true_false_scorers.ipynb | Updates paired scoring documentation. |
| doc/code/scoring/0_scoring.py | Links execution-trace scoring guidance. |
| doc/code/scoring/0_scoring.ipynb | Updates paired scoring overview. |
| doc/code/framework.md | Defines job, trace, and analytics responsibilities. |
| doc/code/executor/7_promptgen.py | Demonstrates checkpoint review and continuation. |
| doc/code/executor/7_promptgen.ipynb | Adds paired generation example. |
| doc/code/analytics/0_attack_results.md | Documents rate helpers and ASSERT metrics. |
Review effort: Balanced
| await asyncio.wait({task}) | ||
| self._cancel_requested.discard(generation_id) | ||
| return await self.get_async(generation_id) | ||
| return await self._update_async(generation_id=generation_id, status=PromptGenerationStatus.CANCELLED) |
| async with self._lock: | ||
| _require_names_available(names=names, taken=await self._get_taken_dataset_names_async(exclude_id=None)) | ||
| for record in records: | ||
| await self._store.save_async(record) |
| data = record.model_dump_json(indent=2).encode("utf-8") | ||
| await self._storage.create_directory_if_not_exists_async(self._directory) | ||
| if self.is_local: | ||
| await asyncio.to_thread(_replace_file, Path(path), data) |
| if self._cache is None or stale: | ||
| self._cache = {record.id: record for record in await self._read_all_async()} | ||
| self._loaded_at = time.monotonic() |
| dataset = TargetObjectiveProvider.package_result( | ||
| dataset_name=dataset_name, result=result, harm_categories=harm_categories | ||
| ) | ||
| return dataset, {"objective_count": len(result.objectives)} |
…converters - Custom judge dimensions are boolean bad-event flags judged by SelfAskTrueFalseScorer with an ASSERT dimension template; ASSERT's ordinal scale and allow_not_applicable options are rejected. - Repeated samples use TrueFalseCompositeScorer over the repeated judge with an ASSERT vote (ties count as failures, undetermined samples excluded, category re-vote with override warning), replacing the bespoke sampled scorers. - Public exports trimmed to AssertJudgeDimension, AssertJudgePreset, AssertSampleMetadata, and the preset list/load functions. - foundry.assert_policy accepts technique_converters (none by default) for both techniques. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- AssertTechnique is removed. The technique class is built from the AttackTechniqueRegistry, with red_teaming as the default (ASSERT's scenario case). - The baseline, now on by default, sends each group's stored system and user prompts (ASSERT's prompt case). - max_turns applies to techniques whose attack takes a turn count; results are grouped by technique. - AssertPolicy.VERSION is 2, so runs from version 1 do not resume. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- allow_not_applicable: a not-applicable answer is undetermined with assert_not_applicable metadata; repeated samples exclude it and keep the marker only when every sample is not applicable. - ASSERT's ordinal scale is accepted with a PyRIT violation_at grade. The judge's grade position is thresholded by FloatScaleThresholdScorer, so every dimension still yields a true/false verdict, and the grade is kept in assert_grade metadata. Graded dimensions can be the objective. - AssertDimensionMetadata is exported from pyrit.score. Default identities are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ence ASSERT judges can read what the application did, not only what it said: tool calls with arguments and results, model calls, and agent steps captured with OpenTelemetry, rendered after the turns they produced with citable step IDs. - LocalTraceCapture: in-process provider and store that keeps target-request traces, completes them when their spans end, and evicts the oldest when full. - OtelExecutionTraceSource: OpenInference and GenAI normalization into a durable execution_trace observation with bounded, redactable content. - ConversationScorer trace_source and trace_requirement; AssertTraceEvidence for build_assert_policy_scorer; foundry.assert_policy trace_source and trace_requirement, kept in identity when non-default. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Roman Lutz (romanlutz)
left a comment
There was a problem hiding this comment.
Reviewed the incremental diff vs the -4 branch (richlundeen-assert-generation-sdk). Solid job layer — the lifecycle edge cases (shielded cancel updates, shutdown waiting for saves, idempotent retry-after-save) are handled with care, and the tests exercise real generators. One small 500-level bug and a handful of polish items below. All 37 new tests pass for me locally.
| """ | ||
| record = await self.get_async(generation_id) | ||
| self._require_version(record=record, expected_version=update.expected_version) | ||
| changes = update.model_dump(exclude_unset=True, exclude={"expected_version"}) |
There was a problem hiding this comment.
exclude_unset=True keeps explicitly-null fields, so PATCH {"expected_version": N, "request": null} puts request: None into changes, and adapter.normalize_request(None) raises TypeError: 'NoneType' object is not iterable — nothing maps TypeError, so the client gets a 500. (Null dataset_name/targets/pause_for_review land as pydantic ValidationErrors → 400 with a raw internal message.) Since the docstring already says omitted fields are unchanged, adding exclude_none=True here would make explicit nulls mean "unchanged" too and close the 500.
Related nit: update_async accepts targets without the _check_target_roles validation that create applies even to drafts, so a bogus role saved via PATCH only surfaces at start.
| ) | ||
| status = next(candidate for candidate in order if candidate in statuses) | ||
| label = records[0].group_label or records[0].dataset_name | ||
| ordered = sorted(records, key=lambda record: record.dataset_name) |
There was a problem hiding this comment.
Sorting group items by dataset_name discards submission order — an ASSERT suite will list its behaviors alphabetically rather than in the order the user composed them. If the GUI (-6) wants composition order, consider sorting by created_at/insertion order instead; records in a group share one created_at, so a per-item index or the original request order may be needed.
| } | ||
| for item in group.items: | ||
| if item.status in cancellable and item.id not in self._saving: | ||
| await self.cancel_async(item.id) |
There was a problem hiding this comment.
If an item transitions to SAVING/COMPLETED between get_group_async and its cancel_async call, the conflict raises out of the loop and the whole request 409s after some items were already cancelled. Catching PromptGenerationConflictError per item (the item simply no longer needs cancelling) would make group cancel idempotent.
| dataset_name: DatasetNameStr = Field(..., description="Name of the dataset to create; it must not exist yet") | ||
| request: dict[str, Any] | ||
| targets: dict[TargetRoleStr, IdentifierStr] = Field(default_factory=dict) | ||
| pause_for_review: bool = Field(False, description="Pause after a reviewable stage so its output can be edited") |
There was a problem hiding this comment.
pause_for_review=true is accepted for generators with no reviewable stage (e.g. target_objective): the record claims it will pause but never does, and GET .../review then 409s. Worth rejecting at create time when no descriptor stage has reviewable=True, or clearing the flag.
| """ | ||
|
|
||
| DIRECTORY_NAME: ClassVar[str] = "prompt_generations" | ||
| SHARED_REFRESH_SECONDS: ClassVar[float] = 15.0 |
There was a problem hiding this comment.
For shared storage, dataset-name reservation rests on an in-process lock plus a cache that can be up to 15s stale, and record writes are last-writer-wins with no cross-process version check — two backends over one blob container can double-book a name. The README's single-backend-ownership framing covers this implicitly; a sentence making "multiple backends sharing one container isn't supported for generations" explicit would prevent surprises.
| behaviors = client.get("/api/prompt-generators/assert/behaviors").json()["items"] | ||
| suites = client.get("/api/prompt-generators/assert/suites").json()["items"] | ||
| assert len(behaviors) == 52 | ||
| assert {"prompt_injection", "sycophancy"} <= {item["name"] for item in behaviors} |
There was a problem hiding this comment.
Asserting exactly 52 behaviors and the exact 3-suite list means every preset added later breaks this route test. A >= bound plus the membership spot-checks would pin the route behavior without pinning the library size.
Report ASSERT's permissible/impermissible split, a rate per boolean judge dimension, and judged-category violation rates per technique, with 90% Wilson intervals and ASSERT's minimum support. Expose them through a backend route and the GUI run page. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…RT behavior library AssertGenerator gains a checkpointed staged API (advance_async, build_result, estimate, validate_config): stop after a stage, apply a reviewed taxonomy and dimensions (recorded in each group's provenance), and resume after interruption without repeating model calls. TargetObjectiveGenerator.estimate and the provider-free PromptGenerationEstimate report remaining calls and a token range. The ASSERT v0.3.0 behavior (52) and suite (3) presets are vendored with loaders, and StorageIO gains list_files_async. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
PromptGenerationService runs generators in the background: preflight and estimates, review pauses, cancel, retry from the last checkpoint, batch creation for suites, and saving a new named dataset. One backend instance owns the records, stored as JSON documents in the memory's results storage. Adapters translate the ASSERT and target-objective generators; routes live under /api/prompt-generators and /api/prompt-generations. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
38e7c2a to
588e3b9
Compare


Part -5 of a 7-PR ASSERT stack. Design and how to use it: ASSERT in PyRIT.
Each PR targets
mainand contains the PRs above it in this list, so review only this PR's own commits.Description
The backend runs prompt generators as background jobs, so the GUI can estimate, run, review, resume, and save generated datasets. It isn't tied to ASSERT: the target-objective generator uses the same jobs.
PromptGenerationServiceowns the job lifecycle: admission (one generation at a time by default), checkpoint persistence, review pauses, cancel, retry from the last checkpoint, batch creation for suites, dataset-name reservation, and saving a new named dataset. Shutdown lets a dataset save finish. Retrying after an interruption completes a save that already landed instead of failing on the name.<memory results_path>/prompt_generations/<id>.json, next to media output. One backend instance owns the records: they're read once into an in-process cache, and local writes are atomic. No database migration is needed./api/prompt-generators: generator descriptors and the bundled ASSERT behaviors and suites./api/prompt-generations: preflight with estimate, create, review, continue/retry/cancel, and the request recorded in a dataset./api/prompt-generations/batch: queues generations that share settings, such as an ASSERT suite, as ordinary records with a sharedgroup_id.framework.mddescribe the service's responsibility.