Adopt Playwright for End-to-End Testing

Status: accepted
Deciders: Platform Team
Date: 2026-06-25 (documented 2026-06-29)
Parent: 0003 — Self-service module automation

Technical Story: E2E runner selection for a modularly federated retail host

Context and Problem Statement

WhiskerPlatform composes many independently deployed modules (federation remotes and iframe consumers) into one in-store product. End-to-end tests must:

Drive a running whisker-host — login, navigation, module load, iframe boundaries
Be owned by module teams but aggregated by the platform (nightly merge reports)
Work in CI against deployed environments, not only localhost
Stay maintainable across squads without per-team licence costs

We evaluated mainstream E2E runners and AI-assisted tooling before committing.

E2E runner landscape

Core runners

Runner	License / cost	MFE / iframe fit	Notes
Playwright	Free (MIT, Microsoft)	Excellent — `frameLocator`, multi-origin, trace viewer, `merge-reports`	Chosen. Native report merging, strong CI story, codegen built in.
Cypress	Free OSS + paid Cypress Cloud	Adequate — iframe support exists but cross-origin is awkward; no first-class federation story	Popular DX; weaker for multi-app MFE than Playwright. Cloud features are paid.
Selenium / WebDriver	Free (Apache 2.0)	Works everywhere	Industry default for a decade; verbose APIs, flaky-by-default reputation, heavy ops. Playwright largely supersedes it for greenfield JS.
WebdriverIO	Free (MIT)	Good	Selenium-compatible layer with nicer DX; still WebDriver-centric.
TestCafe	Free (MIT)	Adequate	Used in some legacy Nike apps (e.g. `rcf-cash-frontend`); smaller ecosystem than Playwright/Cypress.
Puppeteer	Free (Apache 2.0)	Library, not a test runner	Low-level browser control. Use inside Playwright or for scripts — not a full E2E framework on its own.

AI-assisted / agentic tooling

Tool	Cost	What it does	Verdict for Whisker
`@playwright/mcp`	Free (OSS). LLM cost = existing Cursor subscription.	Official Playwright MCP server. Agent drives the browser, observes DOM, writes Playwright test code.	Best free agentic option. One-time `~/.cursor/mcp.json` setup per developer; `whisker doctor` can warn if missing.
Playwright Codegen	Free (built in)	`playwright codegen` records clicks → test code.	Use for bootstrapping specs; layer fixtures (`authedPage`) on top.
Stagehand (Browserbase)	OSS library; LLM API calls cost money per action at runtime	Natural-language steps in test files (`page.act('click sign in')`).	Interesting later; slower and adds API cost at scale.
Healenium	OSS free (Selenium only); Playwright requires Pro (paid)	Reactive self-healing — retries failed locators using a stored baseline. See below.	Strong enhancement candidate once suites exist; Playwright path needs Pro budget.
AgentQL	Free trial + $0 Starter tier; paid per API call above limits	Proactive semantic locators — AI resolves elements at query time. See below.	Useful for brittle MFE selectors; watch API call volume in CI.
Applitools Eyes	Enterprise pricing (~$1K+/yr)	AI visual regression.	Defer until visual surface area justifies it.
Percy (BrowserStack)	Free tier limited; paid from ~$599/mo	Screenshot diffing.	Defer.
Mabl / Testim / Momentic	SaaS, $500–2K+/mo typical	Full no-code / low-code platforms.	Not worth it when Playwright + MCP is already free.

Healenium — self-healing locators

Healenium addresses the highest ongoing E2E cost in an MFE platform: tests that pass functionally but fail because a selector moved — common when a module team restyles a federated remote or iframe consumer without the host team knowing.

How it works

Healenium is reactive, not generative. It does not write tests; it rescues failing ones at runtime.

Baseline pass — On successful runs, every locator that resolved is stored in a Healenium backend (Postgres + report service), keyed to the element’s DOM context.
UI change — A later run uses the same locator (e.g. #sku-cell). The element is gone or moved; Playwright/Selenium would normally fail here.
Healing — Healenium catches the failure, runs the LSC algorithm (Locator Storage & Comparison): compares the current page state to the stored baseline, scores candidate replacement locators, and retries the command with the best match.
Report — Each heal is logged with before/after locator, screenshot, and success feedback — useful for deciding whether to update the test source permanently.

  Test client ──► Healenium Proxy ──► Browser (Selenium or Playwright server)
                       │
                       ▼
                 Healenium backend
                 (locator DB + reports)

Combining with Playwright

Edition	Playwright?	Cost	Integration
Healenium OSS	No — Selenium & Appium only	Free, self-hosted (Docker)	Drop-in WebDriver proxy; tests use healed `WebElement` wrappers
Healenium Pro	Yes — via Playwright Proxy	Paid (30-day trial; then ~$1.14/hr on AWS Marketplace per healenium.io; enterprise support tiers from ~$10K/yr)	Point `playwright.config.ts` at the proxy WebSocket instead of a local browser

Pro Playwright integration is a transparent proxy — tests stay @playwright/test; only the connection changes:

// playwright.config.ts (Healenium Pro)
use: {
  connectOptions: {
    wsEndpoint: process.env.PLAYWRIGHT_SERVER_URL
      ?? 'ws://localhost:8080/playwright-proxy',
  },
},

The proxy forwards commands to a real Playwright server. When a locator fails, the same LSC healing engine used for Selenium kicks in, retries, and records the heal. Pro also adds an AI reasoning engine that can propose durable locator fixes and open GitHub PRs from heal reports — closing the loop from “healed at runtime” to “fixed in source.”

Why this is a strong enhancement for Whisker: Our nightly runner clones dozens of module repos against a live host. A CSS rename in one module should not red the entire platform sweep. Healenium keeps suites green while teams get visibility into what drifted. The trade-off is ops + licence cost for the Playwright path — OSS Healenium does not help our Playwright-native suites without either adopting Selenium (non-starter) or budgeting for Pro.

Current stance: Defer until we have enough nightly coverage that locator churn is a measured problem, then pilot Healenium Pro + Playwright Proxy on the host suite before rolling to module repos.

AgentQL — semantic / AI locators

AgentQL addresses the same underlying problem from the opposite direction: instead of healing a broken CSS selector after failure, tests use natural-language or schema queries that resolve at runtime via an AI model.

What it does

AgentQL wraps a Playwright Page (agentql.wrap(page)) and adds query methods on top of the normal Playwright API:

API	Use case
`page.getByPrompt('first SKU in the inventory table')`	Single element by plain English
`page.getByAi('Search button that starts the search')`	Drop-in replacement for `getByRole` / `getByTestId`
`page.queryElements('{ search_box, search_btn }')`	Multiple named elements via AgentQL query syntax
`page.queryData(...)`	Extract text/numbers without clicking

Returned elements are standard Playwright Locators — clicks, fills, and assertions work as usual. Each query is an API call to AgentQL’s service (or self-hosted deployment at Enterprise tier).

Example (replaces brittle CSS):

import { wrap } from 'agentql';
import { test } from '@playwright/test';

test('inventory row', async ({ page }) => {
  const aq = await wrap(page);
  await aq.goto('/inventory');
  const sku = await aq.getByPrompt('first SKU in the inventory table');
  await sku.click();
});

Pricing tiers (as of 2026; verify at agentql.com/pricing)

Tier	Monthly cost	Included API calls	After limit	Rate limit
Free trial (one-time)	$0, no card	300 calls	—	10/min
Starter	$0	50 calls/month	$0.02/call	10/min
Professional	$99	10,000 calls/month	$0.015/call	50/min
Enterprise	Custom	Negotiated	—	Dedicated env, on-prem option

Remote browser time is billed separately on paid tiers (Starter includes 10 hrs/mo; Professional includes 500 hrs/mo). Early-adopter accounts may retain more generous legacy free allowances.

CI cost implication: A 50-step test suite using getByAi on every interaction burns 50 API calls per run. At Starter’s 50 free calls/month, one nightly run exhausts the tier. Professional’s 10K calls supports ~200 full suite runs/month before overage — workable for a platform nightly + a handful of modules, but not free at scale.

Healenium vs AgentQL

	Healenium	AgentQL
When it acts	After locator failure	Before/during locator resolution
Test code	Keep existing selectors	Write semantic queries instead
Playwright (free)	OSS: no; Pro: yes (paid)	Yes — SDK wraps Playwright
Best for	Protecting existing suites from drift	New suites in volatile UIs

They are complementary, not either/or: AgentQL for authoring resilient locators in new module tests; Healenium Pro as a safety net under nightly runs so legacy CSS-based specs survive module-side refactors.

Current stance: AgentQL is worth local experimentation on Starter / trial tiers while authoring tests with @playwright/mcp. Do not wire into nightly CI until call volume is modelled. Revisit when selector breakage shows up in heal reports or nightly flakes.

Best free agentic stack (our recommendation)

@playwright/test — runner, assertions, traces, HTML/JSON reports
@playwright/mcp — AI-assisted authoring in Cursor (describe flow → agent writes spec)
Playwright Codegen — optional fast path for login flows (--save-storage for auth state)

Paid resilience layer (evaluate after coverage exists):

Healenium Pro + Playwright Proxy — strongest fit for platform-wide nightly stability; keeps existing specs alive when modules change selectors
AgentQL — optional semantic locator layer for new specs in high-churn module UIs; budget API calls before CI adoption

Visual regression (Applitools, Percy) remains explicitly later.

Decision Drivers

Zero licence cost for platform-wide mandate across many squads
MFE + iframe — cross-origin, frameLocator('iframe[title="…"]'), nightly clone-and-run per module repo
Report aggregation — playwright merge-reports across host + modules
AI-first authoring — module teams already use Cursor; MCP is zero marginal cost
Credential pattern — Cerberus in CI (same path as sim-web-ui-automation-tests); .env.local locally

Decision

Adopt Playwright (@playwright/test) as the sole E2E runner for WhiskerPlatform. Pair it with @playwright/mcp for agentic test authoring. Do not standardise on Cypress, Selenium, or paid SaaS runners.

Platform shape

Layer	Owner	Tooling
Host E2E	Platform team	`whisker-host/tests/e2e/` + `Jenkinsfile.nightly`
Module E2E	Module team	`tests/e2e/` + plain `@playwright/test` specs today; `whiskerTest` fixture from `@nike/whisker-module-kit/e2e` is planned, not shipped
Nightly aggregate	Platform	Clone each manifest `repository`, run `pnpm test:e2e`, merge reports → S3
Compliance (non-blocking)	Kit / devkit	Warn if `tests/e2e/*.spec.ts` or `playwright.config.ts` missing

See Automation, devkit & E2E for mechanism and pipeline phases.

Consequences

Positive

One E2E dialect across host and all modules
Free tooling end-to-end; no vendor lock-in
MCP + Codegen lower the cost of first test suite for legacy iframe apps
merge-reports gives a single nightly HTML view without custom infra

Negative / accepted

Developers must run npx playwright install chromium once per machine
@playwright/mcp is manual Cursor config — cannot be repo-automated
Locator maintenance in MFE remains hard; Healenium Pro is the leading candidate to address it once nightly suites are large enough to justify proxy infra and licence cost
Module teams on legacy TestCafe/Cypress must migrate when joining the platform

Manual setup (not automatable)

Tracked in devkit checklist / docs/manual-setup-required.md:

Install Playwright browsers
Add @playwright/mcp to ~/.cursor/mcp.json
Local .env.local for WHISKER_TEST_* (Cerberus is CI-only)