Visual review vs visual regression testing: when to use each

Visual regression testing compares new screenshots of your interface with approved baseline images and fails when they differ beyond a threshold: it answers “did something change that we did not mean to change?”. Visual review is a person looking at the screens that were meant to change and deciding whether they are right. The two answer different questions, so a team that ships interface changes often needs both, in a set order.

Two practices, two questions

	Visual review	Visual regression testing
Question	Is this change right?	Did anything else change?
Who decides	A person: designer, product owner, developer	An image comparison, then a person when it fails
When	Before code, on a prototype, and before merge	On every run of the test suite, usually in CI
Input	The screens that were meant to change	Every screen that has a test
Catches	Wrong layout, wrong copy, missing states, weak design	A shifted margin, a lost style, a broken font
Misses	Screens nobody thought to look at	Screens without a test, and whether a change is good

Review is about intent. Regression testing is about everything else staying where it was.

How visual regression testing works

A test opens a page, takes a screenshot and compares it with a stored baseline. With Playwright Test, one assertion does it:

import { test, expect } from "@playwright/test";

test("invoices list", async ({ page }) => {
  await page.goto("/invoices"); // baseURL is set in playwright.config
  await expect(page.getByRole("row").nth(1)).toBeVisible();
  await expect(page).toHaveScreenshot("invoices-list.png", {
    maxDiffPixels: 100,
    mask: [page.getByTestId("last-sync")],
  });
});

As Playwright’s visual comparisons guide describes, the first run finds no baseline, says so and writes the current screenshot as the expected image, which you commit. Later runs compare against it. Before comparing, toHaveScreenshot() waits until two consecutive screenshots are identical, and it disables animations by default.

Two settings decide how strict the comparison is. threshold is the accepted color difference for one pixel, from 0 (strict) to 1 (lax), 0.2 by default. maxDiffPixels and maxDiffPixelRatio set how many pixels may differ in total. When the page changes on purpose, npx playwright test --update-snapshots rewrites the baselines.

What makes regression tests flaky, and the fixes

A visual test that fails for no reason gets ignored, then deleted. The usual causes:

• The machine. Playwright’s docs warn that rendering varies with the host OS, browser version, settings, hardware, power source and headless mode. Each baseline is named after the browser and platform, for example invoices-list-chromium-linux.png. Generate and compare baselines in one environment, typically the CI container.
• Time and data. Dates, relative times and records that change between runs. Freeze the clock and seed fixed demo data.
• Dynamic regions. Ads, avatars, live counters. Cover them with mask, or hide them with a stylesheet passed as stylePath.
• Hover. A pointer left on a button keeps its hover style. The docs suggest moving the mouse away before the assertion.
• Too many full pages. A full-page baseline fails on any change anywhere on the page. Screenshots of the viewport, or of one component with expect(locator).toHaveScreenshot(), fail closer to the cause.

If you already capture your screens with a script, the same stability rules apply: screenshot every page with Playwright goes through them.

What visual regression testing cannot tell you

A passing visual test means the pixels match the baseline. It does not mean the baseline was right. Three gaps matter:

1. Intended changes look like failures. Every redesign turns tests red until someone updates the baselines, and --update-snapshots accepts whatever the page shows, bugs included. Whoever runs it is making a design decision.
2. Untested states are unguarded. The error state, the empty list and the admin view are protected only if someone wrote a test for each.
3. Quality is out of scope. A clumsy layout that never changes passes forever.

The first gap matters more when an AI coding agent writes the interface. An agent asked to make the tests pass can update the baselines itself, and the suite turns green on a broken screen. Make baseline updates a separate commit, reviewed by a person with the before and after images side by side.

How visual review works

Visual review is a person comparing what they asked for with what they got, one screen state at a time. It works best with:

• The states side by side, not one screenshot of the happy path: empty, loading, error, full, each role.
• Before and after next to each other, at the same size.
• A short checklist per state: the copy is final, the layout holds with long text, error messages say what to do, nothing moved that should not have.
• Review before code. Judging a prototype is cheaper than judging a merged feature: the change is still a picture, and saying no costs nothing.

The guide on how to review UI changes made by an AI agent has a fuller checklist, and comparing design variants before coding covers the step where you choose between directions.

One loop for both

The two practices fit together in a fixed order:

1. Review the intended change on a prototype or on screenshots, state by state, and choose a version.
2. Implement the chosen version.
3. Run the visual tests. Failures on the screens you meant to change are expected; failures anywhere else are regressions to fix.
4. Update the baselines of the intended screens only, in their own commit, checked against the version chosen in step 1.
5. Add tests for any new state, so the next change is guarded too.

Review decides what should change. Regression testing makes sure that only that changed.

Where agentappbuilder fits

agentappbuilder is a visual review tool, not a test runner. With your agreement, your coding agent captures each state of your running app with a Playwright script, or it draws each state as HTML, and each state becomes a frame with its own URL. When you ask for a change, the agent adds a new version, and Compare shows two versions of the same state side by side, including the capture of the app today next to the HTML version the agent proposes. You choose the version, then ask your agent to implement it.

It does not compare pixels or run your tests today. Acceptance tests generated from a version’s contract are planned, and a quality harness (specs, tests, CI) that your agent installs in your repository is coming next: neither is available today. The architecture page lists what is available and what is planned, and the canvas documentation describes Compare.

Use review to decide, and regression tests to hold the line. Neither replaces the other, and the handoff between them, the baseline update, deserves the same care as the change itself.