feat(storybook): add visual regression testing for stories by pauldambra · Pull Request #3392 · PostHog/code

pauldambra · 2026-07-13T12:51:07Z

What

Ports posthog/posthog's Storybook visual regression approach: every story is screenshot in Chromium in both dark and light themes using @storybook/test-runner, and CI submits the captures to the PostHog Visual Review product for diffing, PR gating, and review — exactly like posthog/posthog. No PNGs are committed to the repo.

How it works

Harness (apps/code/.storybook/test-runner.ts): waits for fonts, quill spinners/skeletons, image loads, and DOM stability; disables animations; flips the theme global per snapshot so one render pass produces <story-id>--dark.png and --light.png. Component snapshots clip to the union of #storybook-root and any Radix-portaled dialogs/popovers. Per-story knobs via parameters.testOptions (viewport, waitForSelector, themes, snapshotTargetSelector) and tags: ["test-skip"].
Determinism: under the test runner the preview freezes the clock with mockdate and re-seeds Math.random before every story render (needed because GeneratingIndicator renders a random verb and a live timer). Two consecutive full verify passes ran clean locally (358/358), and a CI verify run passed against a CI capture.
CI (.github/workflows/code-storybook.yml): builds workspace packages + static Storybook, captures all stories, then vr submits the PNGs to Visual Review (CLI built from posthog/posthog master, same as posthog's own CI). VR diffs against the signed manifest at apps/code/snapshots.yml, posts the verdict on the PR, and commits the approved manifest back to the branch on approval. Main pushes submit with --purpose observe (tracking, non-gating). PNGs and diff/failure output are gitignored; failure screenshots upload as artifacts; junit goes to Trunk.

One-time setup required before VR gates anything

The vr step silently no-ops until this is done (the capture still runs and catches broken stories):

Register PostHog/code in Visual Review settings (project 2) — mints the repo UUID.
Create a personal API key with the visual_review scope; add it as the VR_API_TOKEN Actions secret here.
Commit the seeded apps/code/snapshots.yml (template in docs/testing.md).

Bugs fixed along the way (both pre-existing)

8 story suites couldn't load in Storybook at all: the @posthog/agent dist bundles are Node-targeted (tsup createRequire banner, fs/path imports) and crashed at module load in the browser. The Storybook Vite config now shims node:module/node:fs and resolves node:path to pathe (Storybook-only).
Every ConversationView story containing a user message crashed into the error boundary: the Storybook DI stub handed out a fresh truthy Proxy per service resolution, sending useFeatureFlag into an infinite setState loop (React fix: hide electron process from tray #185). Stubs are now cached per service id and FEATURE_FLAGS is bound to a fake returning real booleans.

Local usage

pnpm --filter code build-storybook
cd apps/code && pnpm exec http-server storybook-static --port 6006 --silent &
pnpm --filter code test:visual:update   # capture locally (PNGs are gitignored)

Docs in docs/testing.md.

Note for history: earlier iterations of this branch auto-committed baseline PNGs; those commits remain in the branch history and will fold away if the PR is squash-merged.

Created with PostHog Code

Port posthog/posthog's Storybook test-runner approach: every story is snapshot-tested in Chromium in dark and light themes via @storybook/test-runner + jest-image-snapshot (ssim, 1% threshold). Baselines are Linux-rendered, live in apps/code/.storybook/__snapshots__, and are auto-committed to the PR branch by CI through the GitHub API so they satisfy the signed-commit ruleset and show as reviewable PNG diffs. The preview freezes time (mockdate) and seeds Math.random per story render under the test runner so elapsed timers and randomized status verbs render identically on every pass. Also fixes 8 story suites that could not load in Storybook at all: the @posthog/agent dist bundles are Node-targeted (createRequire banner, fs/path imports), so the browser build now shims node:module and node:fs and resolves node:path to pathe. Generated-By: PostHog Code Task-Id: c8fa7392-2afe-4919-b8be-e6e3b56e1419

trunk-io · 2026-07-13T12:51:11Z

Merging to main in this repository is managed by Trunk.

To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3cf7b509cf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

pauldambra

Note

🤖 Automated comment by QA Swarm — not written by a human

QA Swarm review complete. See inline comments.

pauldambra · 2026-07-13T13:05:40Z

Note

🤖 Automated comment by QA Swarm — not written by a human

Multi-perspective review: qa-team (specialists + generalists), paul-reviewer, xp-reviewer, security-audit

Verdict: ⚠️ REQUEST CHANGES _{(round 1 @ 3cf7b50)}

The harness and determinism work are solid, but the workflow ships with a verified fail-open bug: the paths-filter never matches, so the visual-regression job silently skips on every PR — the feature is inert as merged.

Key findings

🟠 HIGH — predicate-quantifier: every + four disjoint positive globs means the change filter can never match; visual-regression always skips (green-but-skipped required check). All four reviewers flagged this independently; verified against dorny/paths-filter source.
🟡 MEDIUM — bare wait after backgrounded package builds swallows build failures.
🟡 MEDIUM — auto-commit loop has no circuit breaker; the in-file comment's "snapshots-only diff skips the run" rationale is wrong (paths-filter diffs the whole PR, not the push). Real guard is render determinism only.
🟡 MEDIUM — baselines are append-only: orphaned PNGs from renamed/removed stories are never pruned; the commit script's deletions path is dead code.
🟢 LOW — app token minted unscoped and handed to PR-controlled code (prefer actions/create-github-app-token with repo/permission narrowing); porcelain parsing fragile for exotic paths; stale-head race dumps raw GraphQL error; no timeout-minutes; docs' cleanup command misses untracked PNGs; 500ms magic sleep per theme.

Convergence

paths-filter every bug: all four reviewers (highest confidence).
porcelain/rename parsing fragility: qa-team + paul + xp.
auto-commit loop risk: qa-team + security-audit.
token scoping: qa-team + security-audit.
500ms sleep and navigator?. nit: paul + xp.

Reviewer summaries

Reviewer	Assessment
🔍 qa-team	Careful port, but a verified showstopper in change detection plus three mediums (wait-swallow, loop guard, no pruning).
👤 paul	Really nice work with why-comments throughout; wants the filter question nailed down before merge — a check that goes green by skipping is the worst kind.
📐 xp	Clean infra, comments earn their place; the `every` filter makes the guard protect nothing while showing green (Rule 1: does it actually work?).
🛡 security-audit	Fundamentals sound (fork gating, no injection, pathspec-constrained writes, SHA pins); fail-open filter is the main defect, plus token least-privilege hardening.

Automated by QA Swarm — not a human review

Fixes the paths-filter predicate-quantifier bug that made the workflow never trigger, makes background build failures fail loudly instead of being swallowed by a bare `wait`, adds a real commit-loop circuit breaker, swaps to actions/create-github-app-token with narrowed permissions, prunes obsolete snapshot baselines via jest-image-snapshot's outdated-snapshot reporter, hardens the snapshot-commit script's git status parsing and stale-branch handling, and tightens several small issues in the test runner and preview decorator. Generated-By: PostHog Code Task-Id: c8fa7392-2afe-4919-b8be-e6e3b56e1419

Extract wait_all helper in the workflow build step, compute the auto-commit/fork conditions once, and tie the post-theme-flip wait to an actual paint (plus a short settle) instead of a flat sleep alone. Generated-By: PostHog Code Task-Id: c8fa7392-2afe-4919-b8be-e6e3b56e1419

…k-visual-regression # Conflicts: # pnpm-lock.yaml

github-actions · 2026-07-13T16:36:58Z

React Doctor found no issues in the changed files. 🎉

_{Reviewed by React Doctor for commit 0dd04ec.}

…ping Skipping the retriggered run left the PR head with a skipped check that never proved the committed baselines pass. Verify mode still breaks the auto-commit loop (it never commits) but actually executes the suite. Generated-By: PostHog Code Task-Id: c8fa7392-2afe-4919-b8be-e6e3b56e1419

The story container handed out a fresh Proxy on every unbound service resolution, so hooks using the service as an effect dependency (useFeatureFlag) re-ran every render, and isEnabled returning a truthy Proxy meant setState never settled - an infinite update loop that crashed every ConversationView story with a user message into the error boundary (their committed baselines were error screens). Cache stubs per service id and bind a real FeatureFlags fake that returns booleans. Generated-By: PostHog Code Task-Id: c8fa7392-2afe-4919-b8be-e6e3b56e1419

…mitting PNGs Match posthog/posthog's actual model: PNGs are gitignored and never committed; CI captures every story and submits the images to the PostHog Visual Review product via the vr CLI (built from posthog/posthog master). VR diffs against the signed manifest at apps/code/snapshots.yml, gates the PR, and commits the approved manifest back to the branch. Removes the auto-commit script, the verify-mode circuit breaker (no commit loop exists anymore), and the 358 committed baseline PNGs. The vr step no-ops until one-time setup lands: VR repo registration for PostHog/code, a VR_API_TOKEN secret, and the seeded snapshots.yml (see docs/testing.md). Generated-By: PostHog Code Task-Id: c8fa7392-2afe-4919-b8be-e6e3b56e1419

Repo registered in Visual Review (project 2); this is the empty signed manifest the vr CLI reads its config from. VR populates it via approval commits once VR_API_TOKEN is set. Generated-By: PostHog Code Task-Id: c8fa7392-2afe-4919-b8be-e6e3b56e1419

With the baseline now committed, an empty token would fail the submit with an auth error instead of degrading gracefully. Generated-By: PostHog Code Task-Id: c8fa7392-2afe-4919-b8be-e6e3b56e1419

358 updated Run: bb85fa31-86d5-4e80-b96e-aaa8f237c1a5 Co-authored-by: pauldambra <984817+pauldambra@users.noreply.github.com>

The VR repo registration had no baseline_file_paths configured, so the approval commit landed at the backend default (root .snapshots.yml) while CI reads apps/code/snapshots.yml. Set baseline_file_paths to storybook: apps/code/snapshots.yml in Visual Review settings so future approval commits land on the path CI uses. Generated-By: PostHog Code Task-Id: c8fa7392-2afe-4919-b8be-e6e3b56e1419

…shes MockDate.set was gated on inStorybookTestRunner() at preview module scope, but the test runner appends its UA marker via an injected script after the page loads, so the check read false and the clock was never frozen - rendered wall-clock times drifted between runs (caught by Visual Review on the autoresearch dashboard stories; the drift sat just under jest-image-snapshot's 1% threshold, which is why local verify passes missed it). Freeze the clock in the outermost decorator, where the marker is visible, and pin the autoresearch fixture epoch since module-scope fixtures evaluate before any decorator runs. Also skip the visual-regression job when a push is the VR bot committing an approved baseline: paths-filter diffs the whole PR, so the existing filter exclusion can never catch it; compare the push delta instead, as posthog/posthog does. Generated-By: PostHog Code Task-Id: c8fa7392-2afe-4919-b8be-e6e3b56e1419

posthog · 2026-07-13T20:55:24Z

✅ Visual changes approved by @pauldambra — baseline updated in 0dd04ec.

View this run in PostHog

11 changed.

Changed

Snapshot	Before	After
`archive-archivedtasksview--branch-not-found-dialog--light`
`archive-archivedtasksview--default--dark`
`archive-archivedtasksview--default--light`
`archive-archivedtasksview--many-tasks--dark`
`archive-archivedtasksview--many-tasks--light`
`archive-archivedtasksview--with-missing-task--dark`
`archive-archivedtasksview--with-missing-task--light`
`autoresearch-configurable-full-dashboard--completed-run--dark`

…and 3 more — view all in PostHog.

11 updated Run: b446dbac-f55e-4036-9076-46c5a1535517 Co-authored-by: pauldambra <984817+pauldambra@users.noreply.github.com>

chatgpt-codex-connector Bot reviewed Jul 13, 2026

View reviewed changes

Comment thread .github/workflows/code-storybook.yml

pauldambra commented Jul 13, 2026

View reviewed changes

pauldambra added 3 commits July 13, 2026 14:16

Merge remote-tracking branch 'origin/main' into posthog-code/storyboo…

547cc1b

…k-visual-regression # Conflicts: # pnpm-lock.yaml

releaser-array Bot and others added 13 commits July 13, 2026 16:42

chore(storybook): update visual regression snapshots (1/4)

1720dd0

chore(storybook): update visual regression snapshots (2/4)

ae44495

chore(storybook): update visual regression snapshots (3/4)

a5a7c6f

chore(storybook): update visual regression snapshots (4/4)

001fc69

chore(storybook): update visual regression snapshots

5e41b12

fix(storybook): skip Visual Review submit while VR_API_TOKEN is unset

4b2d44d

With the baseline now committed, an empty token would fail the submit with an auth error instead of degrading gracefully. Generated-By: PostHog Code Task-Id: c8fa7392-2afe-4919-b8be-e6e3b56e1419

chore(visual): update storybook baselines

9526fdd

358 updated Run: bb85fa31-86d5-4e80-b96e-aaa8f237c1a5 Co-authored-by: pauldambra <984817+pauldambra@users.noreply.github.com>

chore(visual): update storybook baselines

0dd04ec

11 updated Run: b446dbac-f55e-4036-9076-46c5a1535517 Co-authored-by: pauldambra <984817+pauldambra@users.noreply.github.com>

pauldambra requested a review from a team July 13, 2026 21:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(storybook): add visual regression testing for stories#3392

feat(storybook): add visual regression testing for stories#3392
pauldambra wants to merge 18 commits into
mainfrom
posthog-code/storybook-visual-regression

pauldambra commented Jul 13, 2026 •

edited

Loading

Uh oh!

trunk-io Bot commented Jul 13, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

pauldambra left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pauldambra commented Jul 13, 2026

Uh oh!

github-actions Bot commented Jul 13, 2026 •

edited

Loading

Uh oh!

posthog Bot commented Jul 13, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

pauldambra commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How it works

One-time setup required before VR gates anything

Bugs fixed along the way (both pre-existing)

Local usage

Uh oh!

trunk-io Bot commented Jul 13, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

pauldambra left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pauldambra commented Jul 13, 2026

Verdict: ⚠️ REQUEST CHANGES (round 1 @ 3cf7b50)

Key findings

Convergence

Reviewer summaries

Uh oh!

github-actions Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

posthog Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

pauldambra commented Jul 13, 2026 •

edited

Loading

Verdict: ⚠️ REQUEST CHANGES _{(round 1 @ 3cf7b50)}

github-actions Bot commented Jul 13, 2026 •

edited

Loading

posthog Bot commented Jul 13, 2026 •

edited

Loading