fix(agent): stop unprocessable images from re-triggering errors every turn#3410
Closed
posthog[bot] wants to merge 1 commit into
Closed
fix(agent): stop unprocessable images from re-triggering errors every turn#3410posthog[bot] wants to merge 1 commit into
posthog[bot] wants to merge 1 commit into
Conversation
… turn An image the API can't process (unsupported type, or over the 5 MB per-image limit) gets baked into the resumable session transcript — as a direct attachment or as a Read tool result on an image file. On every subsequent turn the whole history is replayed, so that one bad image keeps 400ing while the agent otherwise makes progress, quietly burning tokens and credits. The transcript sanitizer already heals empty/malformed blocks on resume; extend it to neutralize unprocessable image blocks too (walking tool_result content, the most common poison path), and guard oversized inline images at prompt-build time so they never enter the transcript. Reuse a single shared 5 MB constant + base64 size estimator. Generated-By: PostHog Code Task-Id: f53b98d2-720a-4fcb-96a1-225c7ba55906
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
|
React Doctor found no issues in the changed files. 🎉 Reviewed by React Doctor for commit |
Contributor
|
superseded by #3424 due to us wanting to ship also circuit breaker on top |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
For anyone running longer agent tasks (reported for Fable in cloud sessions): once an image the API can't process enters the conversation, the same image-processing error re-triggers after nearly every subsequent search or file read. The agent otherwise keeps making progress, so the repeated failure is easy to miss while it quietly accumulates token and credit usage.
Root cause: an unprocessable image — either a direct attachment or the result of a
Readon an image file — gets baked into the resumable Claude session transcript. That transcript is replayed to the provider on every following turn, so one bad image keeps getting rejected on each turn. The repo already heals empty/malformed blocks in a poisoned transcript on resume (sanitizeSessionJsonl), but had no equivalent for image blocks, and did not size-guard images before embedding them.Changes
sanitizeSessionJsonlnow neutralizes unprocessable image blocks — unsupported media type, or decoded size over the 5 MB per-image limit — replacing them with a short text note. It walkstool_resultcontent arrays too, since aReadon a large image lands its bytes there (the most common poison path). This clears the offending image from the agent context so it stops re-triggering the error.acp-to-sdk) instead of being embedded, mirroring the existing unsupported-MIME-type handling.MAX_CLAUDE_IMAGE_BYTES) and base64 size estimator (estimateBase64Bytes), deduping the copy that lived incloud-prompt.ts.Scope note: the heal takes effect at session hydration/resume (the established mechanism), so it clears the bad image on the next resume rather than mid-turn. It also does not catch images that are under 5 MB but exceed the provider's pixel limits, which would need image decoding — a possible follow-up.
How did you test this?
sanitizeSessionJsonl(oversized image nested in atool_result, unsupported top-level MIME type, and that small/supported and URL-sourced images are left untouched) and for theacp-to-sdkoversized-image guard.jsonl-hydrationandacp-to-sdksuites (all new and existing tests pass; one unrelated pre-existing test fails only because the sandbox runs as root, wherechmodcan't make a dir read-only).cloud-prompttests,typecheckfor shared/core/agent, and Biome lint (including zeronoRestrictedImportsin core).Automatic notifications
Created with PostHog Code from an inbox report.