Skip to content

Model review for PRs to .github: self-review rule R-self and scripted setup - #37

Merged
irvingpop merged 2 commits into
mainfrom
irvingpop/github-self-review
Nov 14, 2026
Merged

irvingpop merged 2 commits into
mainfrom
irvingpop/github-self-review

Conversation

@irvingpop

Copy link
Copy Markdown
Member

Closes #36.

PRs to .github get no model review today: the file runs there as a normal pull_request_target workflow from main, and rule R1 accepts only the release SHA. This adds rule R-self for that case only.

What changed

  • analyze.py: picks R-self only when the runner's GITHUB_REPOSITORY_ID is .github and GITHUB_EVENT_NAME is pull_request_target. Everything else keeps R1. R-self ships as an all-zero placeholder, so .github runs still fail closed until the rule exists.
  • release.py: guard so a release never moves R-self.
  • make self-review-plan / make self-review-setup: ephemeral ant login, cleared on every exit (shared with the release tool). Reads and checks everything before any write, refuses unknown response shapes and wrong member roles, creates the workspace, adds the service account as workspace_developer, creates the rule, reads it back, then writes the rule ID into analyze.py atomically (refuses if the file changed during the run).
  • Spec: R-self claims, owner steps, direct-run claim evidence, residuals, decision log.

Owner steps after merge

  1. make self-review-plan (read-only).
  2. make self-review-setup.
  3. Console web UI: set a low monthly spend limit on workspace claude-review-self-review (ant has no command for it).
  4. Commit the one-line analyze.py change in a new PR.

Testing

  • cd review && uv run pytest tests/unit: 3409 passed, 2 deselected.
  • Codex review found 4 Important issues (direct-run claim contract, fail-open list parsing, unchecked member role, stale file write). All fixed and re-checked.

Not checked

  • Live ant response shapes (the tool refuses unknown shapes).
  • The job_workflow_ref value of a direct .github pull_request_target run. Evidence shows the claim exists; the first R-self run shows its value.

🤖 Generated with Claude Code

irvingpop and others added 2 commits November 14, 2026 11:49
PRs to .github run claude-review.yml from main, so R1 (bound to the
release SHA) refuses their token. Analyze now picks a second rule,
R-self, only when the runner's GITHUB_REPOSITORY_ID is .github and
GITHUB_EVENT_NAME is pull_request_target. Every other run keeps R1.
The R-self ID is all zeros until the owner creates the rule, so those
runs still fail with "federation not configured", never a pass.

make self-review-plan / self-review-setup create the workspace, the
service account membership and the rule (find first, create only what
is missing, never edit), read them back, and write the rule ID into
analyze.py. They reuse release.py's ephemeral ant login and exit rules
(run_guarded, Console.call, check_ant per resource). The workspace
spend limit stays manual: ant has no command for it.

The release tool still touches only R1; tests prove it, and its rule
test refuses a rule bound to a branch ref such as R-self.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…le and file checks

- Direct pull_request_target runs carry job_workflow_ref/job_workflow_sha:
  recorded V3 direct-run claims (run 40259047420) as a test fixture, and
  run 43631842435 passed the local job_workflow_sha check on .github. Spec
  records the evidence; tests use the recorded claim set.
- Setup reads and checks everything before any write. List output must be
  one item per JSONL line or a complete data page, and each item needs its
  fields, else it refuses before a create or add.
- The service account must have role workspace_developer, for an existing
  member and after an add.
- analyze.py is written atomically, only when it did not change during
  the run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@irvingpop
irvingpop requested a review from a team as a code owner November 14, 2026 20:02
@github-actions

Copy link
Copy Markdown

🔴 Comms down: the Claude of Duty review did not complete

Mission: Review the code. Protect the branch. Deploy the future.

Caution

The Review, Security and Supply chain reviews did not run, so this PR could not be judged. No waiver can clear this.
Next step: a maintainer re-runs the newest run of the required workflow once the cause is fixed. The author does not need to change anything.

Reviewed from the diff only (no full file): 5 file(s): docs/claude-review-design.md, review/tests/unit/fixtures/oidc/v3-direct-dispatch-claims.json, review/tests/unit/test_analyze.py, review/tests/unit/test_self_review_setup.py, review/tools/self_review_setup.py.

Severity Effect Where Finding Found by
⚠️ Not run Blocks, no waiver Review, Security and Supply chain passes authentication failed. The pass returned no result. Workflow (code, no model)
File checks Review Security Supply chain Dependency review Coverage
✅ ok, 2 signals ❌ failed ❌ failed ❌ failed ✅ ok ✅ complete

Run 43635344322, attempt 1 for head 01777b5 into main. The required workflow in the merge box is the result that counts.

Claude of Duty never approves a PR. A human reviewer gives the final approval.

📋 Full review record: reasons, checks, signals, finding details, model summaries, token usage

Reviewed snapshot

Head SHA Merge base SHA Base branch Run
01777b52256a85bcf91afaafd43e66c26ef6eac9 c129aa953f1ebb0844af0dc2b3b6b5ad09342049 main 43635344322, attempt 1

Reasons the gate failed

Reason Waivable? Detail
A review pass failed 🚫 no pass review failed: authentication failed, pass security failed: authentication failed, pass supply-chain failed: authentication failed

No waiver line is shown. At least one reason cannot be waived, so a waiver would not change the result.

Checks (code, no model)

Check Result Detail
Hard blocks ✅ none
Coverage ✅ complete files in comparison 9, files collected 9, trees complete
Model view diff-only estimate 171,935 tokens, limit 200,000 tokens, multiplier 1x
Diff only ℹ️ patch only docs/claude-review-design.md
Diff only ℹ️ patch only review/tests/unit/fixtures/oidc/v3-direct-dispatch-claims.json
Diff only ℹ️ patch only review/tests/unit/test_analyze.py
Diff only ℹ️ patch only review/tests/unit/test_self_review_setup.py
Diff only ℹ️ patch only review/tools/self_review_setup.py
Dependency changes ✅ none
Dependency review job ✅ success

Signals (context only, never block)

  • ⚠️ Sensitive path changed: change to Makefile, in Makefile
  • ℹ️ Very long line: lines longer than 500 characters, in docs/claude-review-design.md, lines 89 884 885 887 963 1296

Dependency changes

None.

Finding details

Model output from untrusted input. Check it before you act on it. Verified means only that the quoted text is at that place. It does not mean the claim is true.

Blocking is decided by code from category and severity, never by the model. A finding in a blocking category blocks only at high or critical severity.

No findings from the passes that ran. The Review, Security, Supply chain pass did not run, so this is not a clean result.

Model summaries

Model output from untrusted input. Check it before you act on it. Verified means only that the quoted text is at that place. It does not mean the claim is true.

Pass Status Model Summary
Review ❌ failed No summary. Reason: authentication failed
Security ❌ failed No summary. Reason: authentication failed
Supply chain ❌ failed No summary. Reason: authentication failed

Token usage

No token usage was recorded for any pass.

@irvingpop
irvingpop merged commit fdd905c into main Nov 14, 2026
4 of 5 checks passed
@irvingpop
irvingpop deleted the irvingpop/github-self-review branch November 14, 2026 20:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Model review for PRs to .github itself (second federation rule)

1 participant