Skip to content

Fix silent failure on large PRs (406 diff too large)#82

Open
MaxwellCalkin wants to merge 1 commit intoanthropics:mainfrom
MaxwellCalkin:fix/silent-failure-large-prs
Open

Fix silent failure on large PRs (406 diff too large)#82
MaxwellCalkin wants to merge 1 commit intoanthropics:mainfrom
MaxwellCalkin:fix/silent-failure-large-prs

Conversation

@MaxwellCalkin
Copy link

Summary

Fixes #80 — When a PR's unified diff exceeds GitHub's 20,000-line API limit, the action silently reports 0 findings instead of surfacing the error. This gives a false sense of security since the PR appears clean when no analysis actually occurred.

Changes

  • Detect HTTP 406 in get_pr_diff() — Instead of letting raise_for_status() throw a generic error that gets swallowed, we now explicitly check for status 406 and fall back gracefully
  • Fall back to local git diff — When the API returns 406, we run git diff origin/<base_ref>...HEAD locally (the action already checks out code), falling back to git diff HEAD~1 if needed
  • New DiffTooLargeError exception — Raised only when both the API and all local diff fallbacks fail
  • Structured "incomplete scan" output — When the diff cannot be obtained at all, the Python script outputs JSON with scan_status: "incomplete" and skip_reason: "diff_too_large" instead of an empty findings list
  • Surface errors in action.yml — The shell wrapper now detects scan_status == "incomplete" and emits ::error:: annotations so the failure is visible in the GitHub Actions UI (instead of silently passing with 0 findings)

How it works

  1. get_pr_diff() requests the diff from GitHub API
  2. If HTTP 406 → calls _get_local_diff(base_ref) which tries:
    • git diff origin/main...HEAD (using PR's actual base branch)
    • git diff HEAD~1 (fallback)
  3. If local diff succeeds → scan proceeds normally with locally-obtained diff
  4. If local diff also fails → raises DiffTooLargeError, main() outputs structured error JSON, action.yml surfaces it as an error annotation

Tests

Added 9 new tests (176 total pass):

  • 406 triggers local git diff fallback
  • Tries origin/base...HEAD before HEAD~1
  • Falls back to HEAD~1 when base ref fails
  • Works without base_ref (uses only HEAD~1)
  • Raises DiffTooLargeError when all fallbacks fail
  • Handles git diff timeout gracefully
  • Non-406 errors still raise normally
  • Local diff output filters generated files
  • main() produces structured incomplete scan output for DiffTooLargeError

Test plan

  • All 176 existing tests pass (no regressions)
  • 9 new unit tests covering all fallback paths
  • Manual test with a PR exceeding 20K lines (requires large PR)

When the GitHub API returns HTTP 406 for diffs exceeding the 20,000-line
limit, the action previously swallowed the error and reported 0 findings,
giving a false sense of security.

This fix:
- Detects the 406 response in get_pr_diff() and falls back to local
  git diff (origin/base...HEAD, then HEAD~1)
- If local diff also fails, raises DiffTooLargeError with a structured
  JSON output including scan_status: "incomplete" and skip_reason
- Updates action.yml to detect incomplete scans and surface them as
  ::error:: annotations instead of silently passing

Fixes anthropics#80

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Silent failure on large PRs: 406 diff too large swallowed as 0 findings

1 participant