Skip to content

fix(agent): stop unprocessable images from re-triggering errors every turn#3410

Closed
posthog[bot] wants to merge 1 commit into
mainfrom
posthog-code/heal-unprocessable-image-in-session
Closed

fix(agent): stop unprocessable images from re-triggering errors every turn#3410
posthog[bot] wants to merge 1 commit into
mainfrom
posthog-code/heal-unprocessable-image-in-session

Conversation

@posthog

@posthog posthog Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Problem

For anyone running longer agent tasks (reported for Fable in cloud sessions): once an image the API can't process enters the conversation, the same image-processing error re-triggers after nearly every subsequent search or file read. The agent otherwise keeps making progress, so the repeated failure is easy to miss while it quietly accumulates token and credit usage.

Root cause: an unprocessable image — either a direct attachment or the result of a Read on an image file — gets baked into the resumable Claude session transcript. That transcript is replayed to the provider on every following turn, so one bad image keeps getting rejected on each turn. The repo already heals empty/malformed blocks in a poisoned transcript on resume (sanitizeSessionJsonl), but had no equivalent for image blocks, and did not size-guard images before embedding them.

Changes

  • Heal poisoned transcripts on resume. sanitizeSessionJsonl now neutralizes unprocessable image blocks — unsupported media type, or decoded size over the 5 MB per-image limit — replacing them with a short text note. It walks tool_result content arrays too, since a Read on a large image lands its bytes there (the most common poison path). This clears the offending image from the agent context so it stops re-triggering the error.
  • Prevent poisoning at the source. Oversized inline base64 images are dropped to a text note at prompt-build time (acp-to-sdk) instead of being embedded, mirroring the existing unsupported-MIME-type handling.
  • Reuse a single shared 5 MB constant (MAX_CLAUDE_IMAGE_BYTES) and base64 size estimator (estimateBase64Bytes), deduping the copy that lived in cloud-prompt.ts.

Scope note: the heal takes effect at session hydration/resume (the established mechanism), so it clears the bad image on the next resume rather than mid-turn. It also does not catch images that are under 5 MB but exceed the provider's pixel limits, which would need image decoding — a possible follow-up.

How did you test this?

  • Added unit tests for sanitizeSessionJsonl (oversized image nested in a tool_result, unsupported top-level MIME type, and that small/supported and URL-sourced images are left untouched) and for the acp-to-sdk oversized-image guard.
  • Ran the jsonl-hydration and acp-to-sdk suites (all new and existing tests pass; one unrelated pre-existing test fails only because the sandbox runs as root, where chmod can't make a dir read-only).
  • Ran cloud-prompt tests, typecheck for shared/core/agent, and Biome lint (including zero noRestrictedImports in core).

Automatic notifications

  • Publish to changelog?
  • Alert Sales and Marketing teams?

Created with PostHog Code from an inbox report.

… turn

An image the API can't process (unsupported type, or over the 5 MB
per-image limit) gets baked into the resumable session transcript — as a
direct attachment or as a Read tool result on an image file. On every
subsequent turn the whole history is replayed, so that one bad image
keeps 400ing while the agent otherwise makes progress, quietly burning
tokens and credits.

The transcript sanitizer already heals empty/malformed blocks on resume;
extend it to neutralize unprocessable image blocks too (walking
tool_result content, the most common poison path), and guard oversized
inline images at prompt-build time so they never enter the transcript.

Reuse a single shared 5 MB constant + base64 size estimator.

Generated-By: PostHog Code
Task-Id: f53b98d2-720a-4fcb-96a1-225c7ba55906
@trunk-io

trunk-io Bot commented Jul 14, 2026

Copy link
Copy Markdown

Merging to main in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

@github-actions

Copy link
Copy Markdown

React Doctor found no issues in the changed files. 🎉

Reviewed by React Doctor for commit 7f891c5.

@tatoalo

tatoalo commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

superseded by #3424 due to us wanting to ship also circuit breaker on top

@tatoalo tatoalo closed this Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant