fix(reviewer): stop bot from flagging future-to-it version numbers #63

Merged

rcsheets merged 1 commit from fix/version-claims-prompt into main

2026-05-05 22:09:05 +00:00

rcsheets commented

2026-05-05 22:07:06 +00:00

Owner

Summary

Strengthen the version-number guard in QuickScanPrompt and FullReviewPrompt: future-to-the-model versions are real releases that postdate training and warrant no commentary at all.
Drop the prior Go 2.16 vs 1.26 example (which the model was reading either way) in favor of a clearer framing.
Name the common pinning files (go.mod, package.json, pyproject.toml, Dockerfiles, lockfiles) and explicitly forbid the three failure modes seen on #17: claiming a version "doesn't exist yet," asserting the "current latest," and suggesting a downgrade.

Closes #17.

Test plan

gofmt -l ., go vet ./..., go build ./...
go test -race ./internal/reviewer/...
Observe a subsequent review on a PR that pins a Go/Python/Node version newer than the model's training data and confirm no false version claims appear.

## Summary - Strengthen the version-number guard in `QuickScanPrompt` and `FullReviewPrompt`: future-to-the-model versions are real releases that postdate training and warrant **no commentary at all**. - Drop the prior `Go 2.16 vs 1.26` example (which the model was reading either way) in favor of a clearer framing. - Name the common pinning files (`go.mod`, `package.json`, `pyproject.toml`, Dockerfiles, lockfiles) and explicitly forbid the three failure modes seen on #17: claiming a version "doesn't exist yet," asserting the "current latest," and suggesting a downgrade. Closes #17. ## Test plan - [x] `gofmt -l .`, `go vet ./...`, `go build ./...` - [x] `go test -race ./internal/reviewer/...` - [ ] Observe a subsequent review on a PR that pins a Go/Python/Node version newer than the model's training data and confirm no false version claims appear.

rcsheets added 1 commit

2026-05-05 22:07:06 +00:00

fix(reviewer): stop bot from flagging future-to-it version numbers

ci / check (pull_request) Successful in 44s

Details

6f6906b2e8

The previous guard told the model "don't flag versions you don't recognize,"
but kept an example pair (Go 2.16 vs 1.26) that the model could read either
way. In practice it still asserted things like "Go 1.26 doesn't exist yet
(current latest is 1.23)." Reframe the rule: future-to-you versions are
real releases that postdate training and warrant no commentary at all.
Name the common pinning files (go.mod, package.json, pyproject.toml,
Dockerfiles, lockfiles) and explicitly forbid the three observed failure
modes — "doesn't exist yet," asserting the "current latest," and
suggesting a downgrade.

Closes #17

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

pr-reviewer-bot commented

2026-05-05 22:07:08 +00:00

Collaborator

Automated review by pr-reviewer v0.28.2 | Safety Check | Claude Sonnet 4 | tracking id r-fa6a0c-cf0ee7
This is an AI-generated review and may contain mistakes.

Status: ✅ Completed

✅ Verdict: LGTM — This is a safe prompt text modification that improves AI reviewer behavior without introducing any security, breaking, or correctness issues.
Complexity: trivial

Safety Check Review

This PR modifies the prompts used for automated code reviews to strengthen guidance against flagging newer software versions.

Changes reviewed:

File: internal/reviewer/prompts.go
Change type: Prompt text modification (2 lines replaced)
Scope: Updates both QuickScanPrompt and FullReviewPrompt constants

Analysis:

The changes replace the existing version number guidance with more explicit and comprehensive instructions:

Before: Basic warning not to flag unrecognized versions with a simple example
After: Detailed guidance that:

Explicitly states newer versions are likely real releases post-training
Lists specific file types where versions appear (go.mod, package.json, etc.)
Prohibits three specific failure modes: claiming non-existence, asserting "current latest," suggesting downgrades

Safety considerations:

✅ No security issues: This is a prompt text change with no executable code
✅ No breaking changes: Constants are being modified but the interface remains the same
✅ No credentials or secrets: Only contains instructional text
✅ No obvious bugs: The text replacement appears correct and maintains proper string formatting

The change is purely instructional text meant to improve the behavior of an AI reviewer system. The modifications are reasonable and address a specific issue (#17) where the bot was incorrectly flagging legitimate version numbers.

*Automated review by [pr-reviewer](https://git.brooktrails.org/brooktrails/pr-reviewer) v0.28.2 | Safety Check | Claude Sonnet 4 | tracking id `r-fa6a0c-cf0ee7`* *This is an AI-generated review and may contain mistakes.* **Status:** ✅ Completed --- **✅ Verdict: LGTM** — This is a safe prompt text modification that improves AI reviewer behavior without introducing any security, breaking, or correctness issues. **Complexity:** trivial # Safety Check Review This PR modifies the prompts used for automated code reviews to strengthen guidance against flagging newer software versions. ## Changes reviewed: - **File**: `internal/reviewer/prompts.go` - **Change type**: Prompt text modification (2 lines replaced) - **Scope**: Updates both `QuickScanPrompt` and `FullReviewPrompt` constants ## Analysis: The changes replace the existing version number guidance with more explicit and comprehensive instructions: **Before**: Basic warning not to flag unrecognized versions with a simple example **After**: Detailed guidance that: - Explicitly states newer versions are likely real releases post-training - Lists specific file types where versions appear (go.mod, package.json, etc.) - Prohibits three specific failure modes: claiming non-existence, asserting "current latest," suggesting downgrades ## Safety considerations: ✅ **No security issues**: This is a prompt text change with no executable code ✅ **No breaking changes**: Constants are being modified but the interface remains the same ✅ **No credentials or secrets**: Only contains instructional text ✅ **No obvious bugs**: The text replacement appears correct and maintains proper string formatting The change is purely instructional text meant to improve the behavior of an AI reviewer system. The modifications are reasonable and address a specific issue (#17) where the bot was incorrectly flagging legitimate version numbers.

rcsheets force-pushed fix/version-claims-prompt from 6f6906b2e8

ci / check (pull_request) Successful in 44s

Details

to 89946421dd

ci / check (pull_request) Successful in 44s

Details

2026-05-05 22:08:05 +00:00

Compare

rcsheets merged commit ae3500d6d9 into main

2026-05-05 22:09:05 +00:00

rcsheets deleted branch fix/version-claims-prompt

2026-05-05 22:09:06 +00:00