Skip to content

Refresh coderabbit-review skill: fix timeout/retry guidance, match current CLI contract#6

Draft
nehal-a2z wants to merge 1 commit into
mainfrom
nehal/refresh-coderabbit-review-skill
Draft

Refresh coderabbit-review skill: fix timeout/retry guidance, match current CLI contract#6
nehal-a2z wants to merge 1 commit into
mainfrom
nehal/refresh-coderabbit-review-skill

Conversation

@nehal-a2z

Copy link
Copy Markdown

Refreshes plugins/coderabbit/skills/coderabbit-review/SKILL.md to stop the skill from causing review failures, and re-audits every factual claim in it against the current CLI source (apps/cli in the mono repo, as of 2026-07-15).

Why (incident data, Jul 15 2026)

  • code_review_fail: extension_emitter_missing accelerated 1.2k → 4.1k/day. Fail times spike sharply at exactly 600s — this skill's own "10-minute window" is killing coderabbit review mid-run. The server review is fire-and-forget over one WebSocket and cannot survive any disconnect.
  • Killed runs get blind-retried: 83% of the fail increase came from clients failing ≥5x in 2 days, and each retry starts a new full server review (burning rate limit). Codex-agent CLI reviews fail at 20.6% vs 8.7% for claude-agent.
  • Completed CLI reviews run p50 ~4.5 min / p90 ~12 min, server ceiling ~30 min — so a 10-minute window kills roughly the slowest 15–25% of reviews. Additionally, clientId is machine-scoped, so concurrent reviews from parallel sub-agents on one machine share delivery state and can corrupt each other.

Behavior changes

  1. Timeout: replaced the 10-minute wait window with "reviews can take up to 30 minutes; set the exec/tool timeout to ≥1800s or run as a background task". Liveness is now judged by the CLI's own heartbeat NDJSON events (every 45s) instead of a wall-clock silence budget.
  2. New guardrail — no blind retries: on timeout/connection drop, do not immediately re-run coderabbit review; retry at most once per session; report persistent failures to the user.
  3. New guardrail — no concurrency: at most one coderabbit review per machine; never launch reviews from parallel sub-agents.
  4. Scoping strengthened from optional to recommended: prefer -t uncommitted / --base <branch> / --dir <path> over full reviews; full review only when the user asks for everything.

Also bumps plugin.json 1.1.1 → 1.2.0 (repo convention: substantive skill changes bump the version, cf. the 1.1.1 bump commit).

Stale claims corrected (audited against apps/cli)

Skill claimed Code actually says Evidence (mono repo)
Treat a running review as healthy for up to 10 minutes; report timeout after 10 minutes The CLI's own client-side review watchdog is 60 minutes (const timeoutMs = 60 * 60 * 1000); completed reviews run p50 ~4.5 min / p90 ~12 min with a ~30 min server ceiling. 10 minutes was never a real limit — it was killing healthy reviews apps/cli/src/services/review/review.ts:374
NDJSON events are finding / status / error The agent-mode event union has six types: review_context, status, heartbeat, finding, complete, error (discriminator field type) apps/cli/src/commands/review/run/execute.ts:65-92, apps/cli/src/commands/review/run/agent-review-context-event.ts:1-8
(silence during review implied no output at all) Agent mode emits a heartbeat event every 45s (AGENT_HEARTBEAT_INTERVAL_MS = 45_000) — usable as a liveness signal apps/cli/src/commands/review/run/execute.ts:119-120
(no completion event documented) Stream ends with a complete event carrying status: "review_completed"|"review_skipped" and a findings count apps/cli/src/commands/review/run/execute.ts:77-83
(finding shape unspecified; "include the file path…" implied lines available) finding = {type, severity, fileName, codegenInstructions, suggestions[, comment]}comment only when codegenInstructions is empty; no line numbers on the wire apps/cli/src/commands/review/run/format.ts:70-83
Severities grouped as critical / major / minor Display order is critical, major, minor, trivial, info (severity union also includes none, excluded from counts) apps/cli/src/commands/review/run/execute.ts:110-116, pr-reviewer-saas/src/review/parse-review.ts:31-37
-t committed / -t uncommitted shown; default unstated -t, --type <type> accepts exactly all (default), committed, uncommitted apps/cli/src/program.ts:121-125, apps/cli/src/types/review-state.ts:41
(no directory scoping mentioned) --dir <path> reviews only git changes inside a directory — added, since scoped reviews are now the recommendation apps/cli/src/program.ts:135
-c shown with single-file phrasing -c, --config <files...> is variadic (multiple files in one flag) apps/cli/src/program.ts:126-129

Verified still correct (no change)

  • Install one-liner curl -fsSL https://cli.coderabbit.ai/install.sh | shapps/cli/install.sh:10
  • coderabbit --versionapps/cli/src/program.ts:87
  • coderabbit auth status --agent / coderabbit auth login --agentapps/cli/src/commands/auth.ts:29-135
  • --base <branch> / --base-commit <commit>apps/cli/src/program.ts:130-134

Deliberately not added

  • --light: the flag exists (apps/cli/src/program.ts:115) but its server-side effect is gated per-user behind the cli_review_policy_flag PostHog flag (pr-reviewer-saas/src/event-client/cli-review-policy.ts:161-174 — flag off silently falls back to the default policy), so it is not recommendable as generally available.
  • No references to --plain / --fast / --interactive / --cwd / --prompt-only: all removed from the CLI (mono #21906, 2026-07-13; the dead "plain" mode concept cleaned up in #21249). The skill did not mention them before and still does not.

🤖 Generated with Claude Code

…rrent CLI contract

Incident-driven fixes:
- Replace the 10-minute wait window (which was killing the slowest
  15-25% of reviews at exactly 600s) with a >=1800s exec/tool timeout
  or background execution; reviews can take up to 30 minutes.
- Guardrail: never blind-rerun after a timeout/disconnect (the server
  review keeps running and each rerun burns rate limit); retry at most
  once per session.
- Guardrail: at most one review per machine; no parallel sub-agent
  reviews (clientId is machine-scoped, concurrent runs corrupt each
  other's delivery state).
- Recommend scoped reviews (-t uncommitted / --base / --dir) over full
  reviews by default.

Staleness refresh against apps/cli (mono):
- Document the real agent NDJSON contract: review_context / status /
  heartbeat (45s) / finding / complete / error; finding carries
  fileName but no line numbers; completion is a complete event with a
  findings count.
- Severity order extended to critical, major, minor, trivial, info.
- Note -t accepts all (default) besides committed/uncommitted; add
  --dir scoping; -c is variadic.

Bump plugin version 1.1.1 -> 1.2.0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: c9a5c615-e2b3-4d9b-a8d8-2062675d2e85

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch nehal/refresh-coderabbit-review-skill
✨ Simplify code
  • Create PR with simplified code
  • Commit simplified code in branch nehal/refresh-coderabbit-review-skill

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant