How to Build an Adversarial UI Logic Tester with Playwright and LLMs
- Extract a structured, token-efficient snapshot of the current UI state using Playwright's
page.evaluate(). - Compress the snapshot into semantic JSON, filtering to only visible, interactive elements.
- Send the compressed state to an LLM with an adversarial system prompt targeting state corruption, race conditions, and validation bypasses.
- Validate the LLM's returned JSON action plan against a strict Zod schema with retry logic.
- Execute the hostile action sequence through Playwright, capturing console errors, screenshots, and state diffs after each step.
- Detect anomalies by comparing pre- and post-action state for contradictions, unhandled rejections, and unexpected element disappearance.
- Feed findings back into the LLM as context for the next round, enabling iterative deepening on fragile areas.
- Converge after a set number of clean rounds or a max iteration cap, then codify discovered bugs as deterministic regression tests.
Your test suite is green. Coverage sits at 92%. And your users are still finding bugs you never imagined. By the end of this tutorial, you'll have a working framework that uses an LLM to reason about live UI state and generate adversarial interaction plans, then executes them through Playwright, observes the results, and iterates.
Table of Contents
- Why Your UI Tests Have a Blind Spot
- Architecture Overview: How the Pieces Fit Together
- Extracting UI State for LLM Consumption
- The Adversarial Planner: Prompt Engineering for Hostility
- Executing Adversarial Actions with Playwright
- Closing the Loop: Iterative Deepening
- Real-World Results: What This Actually Finds
- Limitations, Costs, and When Not to Use This
- From Predictable Tests to Adversarial Resilience
Why Your UI Tests Have a Blind Spot
Your test suite is green. Coverage sits at 92%. And your users are still finding bugs you never imagined. This is the fundamental problem with AI UI testing as most teams practice it: deterministic, developer-authored tests only verify scenarios the developer already thought of. Adversarial LLM testing flips this model by using an LLM as an active fault-finder that generates hostile interaction sequences against live UI components through Playwright AI automation.
Think about how real users break interfaces. They double-click submit buttons during async validation. They slam the back button mid-navigation. They paste Unicode snowmen into phone number fields, toggle settings in sequences no sane person would attempt, and somehow reach application states your state machine said were impossible. Traditional automated UI testing never covers these paths because no human sat down and wrote a test for them.
Security engineers have understood this problem for decades. Fuzz testing throws malformed, unexpected, and random data at programs to find crashes and vulnerabilities. What we're building here applies the same adversarial thinking at the UI logic layer, targeting not just crashes but functional correctness failures: state corruption, skipped validations, leaked UI elements, and broken workflows.
By the end of this tutorial, you'll have a working framework that uses an LLM to reason about live UI state and generate adversarial interaction plans, then executes them through Playwright, observes the results, and iterates. The LLM doesn't write static test scripts. It actively explores your application like a hostile QA engineer who never gets tired and never assumes your code works correctly.
Prerequisites: working knowledge of Playwright, TypeScript, Node.js, and basic familiarity with LLM API calls through OpenAI or a similar provider.
Architecture Overview: How the Pieces Fit Together
The system has three components wired into a feedback loop:
UI State Extractor (Playwright) captures a structured, token-efficient snapshot of the current page state. Adversarial Planner (LLM) receives that snapshot and returns a sequence of hostile actions designed to break UI logic. Action Executor (Playwright) runs those actions against the live application, captures results, and feeds them back to the planner.
The loop works like this: extract the current DOM state as structured JSON, send it to the LLM with adversarial instructions, receive back an action plan with explicit reasoning for each step, execute the plan through Playwright while capturing screenshots and console errors, compare pre-action and post-action state to detect anomalies, then repeat with accumulated findings as additional context.
This differs fundamentally from using LLMs to write test scripts. Script generation is static: the LLM produces code once, and you run it forever. Adversarial exploration is dynamic: the LLM reacts to live application state and adapts its attack strategy based on what it discovers. It's the difference between a pentest report and an active red team exercise.
One critical design constraint: LLM context windows are expensive and finite. Sending raw HTML from a modern SPA would burn thousands of tokens on irrelevant div nesting and CSS classes. Every piece of state you send to the planner must earn its token cost.
The Adversarial Feedback Loop
The cycle follows a pattern: Observe (extract current UI state) then Reason (LLM generates adversarial plan with explicit rationale) then Attack (Playwright executes hostile actions) then Assert (detect anomalies, state corruption, unhandled errors) then Repeat (feed findings back, narrow focus on fragile areas). Each iteration sharpens the attack surface. If round one reveals a flicker in form validation, round two hammers that exact validation boundary with increasingly hostile inputs.
Here are the TypeScript interfaces that define the data contracts flowing between components:
interface UIElement {
selector: string;
tag: string;
role: string | null;
text: string;
type: string | null;
value: string | null;
disabled: boolean;
visible: boolean;
ariaAttributes: Record<string, string>;
validationState: string | null;
}
interface UIStateSnapshot {
url: string;
title: string;
elements: UIElement[];
consoleErrors: string[];
timestamp: number;
}
interface AdversarialAction {
type: 'click' | 'fill' | 'select' | 'clear' | 'rapid-toggle' | 'navigate' | 'wait' | 'keyboard';
selector: string;
value?: string;
repeat?: number;
delayMs?: number;
reasoning: string;
}
interface TestPlan {
hypothesis: string;
targetWeakness: string;
actions: AdversarialAction[];
expectedAnomalies: string[];
}
interface AnomalyReport {
round: number;
action: AdversarialAction;
anomalyType: 'console-error' | 'state-corruption' | 'unhandled-rejection' | 'visual-regression' | 'contradictory-state';
description: string;
screenshotPath: string | null;
stateBefore: UIStateSnapshot;
stateAfter: UIStateSnapshot;
}Extracting UI State for LLM Consumption
Raw HTML is terrible LLM input. A typical SPA page generates thousands of lines of markup, most of which is structural noise: wrapper divs, CSS module class names, SVG icon paths, script tags. Sending all of that wastes tokens and buries the semantic signal the LLM needs to reason about interactive behavior.
Instead, we extract a structured JSON representation containing only actionable elements and their states. Buttons, form fields, links, toggles, dropdowns, and modals, along with their current values, disabled states, visibility, ARIA attributes, and validation feedback. This lines up with Playwright's own philosophy of locating elements by role and accessible name rather than brittle CSS selectors.
For dynamic content, Playwright's built-in auto-waiting handles most timing issues. You generally don't need to wire up MutationObservers manually. Instead, wait for the specific locator or network idle state you need before extracting. If you're testing a component that loads data asynchronously, use page.waitForLoadState('networkidle') or wait for a specific element that signals readiness. One caveat: networkidle waits for no more than 0 network connections for at least 500 milliseconds, which can be unreliable for apps with persistent WebSocket connections, long-polling, or analytics pings. In those cases, waiting for a specific visible element that indicates the page is ready is more robust.
One edge case to watch: page.evaluate() runs in the browser context, and only serializable values can cross the boundary back to Node. You can't return DOM nodes or functions. Everything must be plain JSON.
Building the State Extractor
// Requires: npm install playwright
import { Page } from 'playwright';
async function extractUIState(page: Page): Promise<UIStateSnapshot> {
// NOTE: In production, attach console/pageerror listeners once per page
// instance rather than inside this function to avoid duplicate listeners.
// They are shown here for self-contained clarity.
const consoleErrors: string[] = [];
const errorHandler = (msg: import('playwright').ConsoleMessage) => {
if (msg.type() === 'error') {
consoleErrors.push(msg.text());
}
};
page.on('console', errorHandler);
const elements: UIElement[] = await page.evaluate(() => {
const interactiveSelectors = [
'button', 'a[href]', 'input', 'select', 'textarea',
'[role="button"]', '[role="tab"]', '[role="checkbox"]',
'[role="switch"]', '[role="menuitem"]', '[role="link"]',
'[contenteditable="true"]', 'details > summary'
];
const allElements = document.querySelectorAll(interactiveSelectors.join(','));
const results: any[] = [];
allElements.forEach((el) => {
const htmlEl = el as HTMLElement;
const rect = htmlEl.getBoundingClientRect();
const isVisible = rect.width > 0 && rect.height > 0 &&
window.getComputedStyle(htmlEl).visibility !== 'hidden' &&
window.getComputedStyle(htmlEl).display !== 'none';
const ariaAttributes: Record<string, string> = {};
for (const attr of Array.from(htmlEl.attributes)) {
if (attr.name.startsWith('aria-')) {
ariaAttributes[attr.name] = attr.value;
}
}
const inputEl = htmlEl as HTMLInputElement;
let validationState: string | null = null;
if ('validity' in htmlEl && inputEl.validity) {
validationState = inputEl.validity.valid ? 'valid' : inputEl.validationMessage || 'invalid';
}
// Generate a stable selector using data-testid, falling back to tag + text
const name = htmlEl.getAttribute('aria-label') ||
(htmlEl.textContent || '').trim().substring(0, 50);
const testId = htmlEl.getAttribute('data-testid');
const selector = testId
? `[data-testid="${testId}"]`
: `${htmlEl.tagName.toLowerCase()}:has-text("${name.replace(/"/g, '\\"')}")`;
results.push({
selector,
tag: htmlEl.tagName.toLowerCase(),
role: htmlEl.getAttribute('role'),
text: (htmlEl.textContent || '').trim().substring(0, 100),
type: htmlEl.getAttribute('type'),
value: ('value' in htmlEl ? inputEl.value : null) || null,
disabled: htmlEl.hasAttribute('disabled') || htmlEl.getAttribute('aria-disabled') === 'true',
visible: isVisible,
ariaAttributes,
validationState,
});
});
return results;
});
// Remove the listener to avoid accumulation if called multiple times
page.off('console', errorHandler);
return {
url: page.url(),
title: await page.title(),
elements,
consoleErrors: [...consoleErrors],
timestamp: Date.now(),
};
}A note on iframes and shadow DOM: page.evaluate() only sees the main document by default. If your application uses iframes, you need page.frames() or page.frameLocator() to access their content. Shadow DOM requires el.shadowRoot.querySelectorAll() inside the evaluate callback, and note that shadowRoot is only accessible on elements with mode: 'open'; closed shadow roots can't be queried this way. Both are common gaps that cause the extractor to silently miss interactive elements.
Minimizing Token Overhead with Smart Summarization
function compressSnapshot(snapshot: UIStateSnapshot): string {
const visibleElements = snapshot.elements.filter(el => el.visible);
const elementSummaries = visibleElements.map(el => {
const parts = [
`[${el.tag}${el.role ? ':' + el.role : ''}]`,
el.text ? `"${el.text.substring(0, 40)}"` : '',
el.type ? `type=${el.type}` : '',
el.value ? `val="${el.value.substring(0, 30)}"` : '',
el.disabled ? 'DISABLED' : '',
el.validationState && el.validationState !== 'valid' ? `INVALID:${el.validationState}` : '',
].filter(Boolean);
return ` ${el.selector} → ${parts.join(' ')}`;
});
const errorSummary = snapshot.consoleErrors.length > 0
? `
CONSOLE ERRORS:
${snapshot.consoleErrors.map(e => ` ⚠ ${e.substring(0, 120)}`).join('
')}`
: '';
return `PAGE: ${snapshot.url}
TITLE: ${snapshot.title}
ELEMENTS (${visibleElements.length}):
${elementSummaries.join('
')}${errorSummary}`;
}This compression function typically cuts token count by 80-90% compared to raw HTML while keeping every piece of information the LLM needs to plan adversarial actions: what elements exist, what state they're in, and what selectors to target.
The Adversarial Planner: Prompt Engineering for Hostility
This is the core innovation. The LLM isn't generating random inputs. It's reasoning about the UI state graph, identifying transitions that might be fragile, and constructing deliberate attack sequences with explicit hypotheses about what will break.
The system prompt establishes a persona and attack taxonomy. I've found that LLM-powered test generation works dramatically better when you give the model specific categories of attacks to consider rather than a vague "find bugs" instruction. Without categories, the model defaults to basic boundary value testing (empty strings, long strings, special characters), which is useful but shallow.
Temperature settings matter here. At temperature 0, the model produces the same conservative attack plan every time. At 1.0, responses become erratic and often structurally invalid. A range of 0.7 to 0.9 produces the best adversarial creativity: varied enough to explore different attack vectors across sessions, structured enough to return valid JSON action plans.
For model selection, GPT-4o and Claude 3.5 Sonnet (and its successor Claude 3.7 Sonnet) both handle this task well. GPT-4o tends to produce more structurally consistent JSON output, particularly when using its response_format: { type: 'json_object' } parameter. Claude tends to write more detailed reasoning in the hypothesis fields. Costs vary and change frequently, so check current pricing before committing to a model. The architecture is model-agnostic; swap the API call and adjust the prompt style.
One risk the outline for this article didn't originally address: prompt injection from page content. If the application under test contains user-generated content or any text that could resemble LLM instructions, that text flows into your snapshot and then into your prompt. Sanitize visible text aggressively, enforce strict JSON-only output schemas, and never let the LLM's output execute arbitrary code.
Crafting the System Prompt
const ADVERSARIAL_SYSTEM_PROMPT = `You are an adversarial UI logic tester. Your goal is to find functional correctness bugs in web applications by generating hostile interaction sequences. You are not looking for visual issues or performance problems. You are looking for STATE CORRUPTION, LOGIC ERRORS, and VALIDATION BYPASSES.
You will receive a structured snapshot of the current UI state. Your job is to return a JSON test plan that attempts to break the application's logic.
TARGET THESE ATTACK CATEGORIES (prioritize based on what the UI state makes possible):
1. STATE MACHINE VIOLATIONS: Attempt to reach states that should be impossible. Skip steps in wizards. Go backward during async operations. Trigger state transitions while previous transitions are incomplete.
2. RAPID TOGGLING: Click/uncheck/recheck elements faster than debounce timers. Toggle between states that trigger async operations and attempt to create race conditions.
3. OUT-OF-ORDER WORKFLOWS: Perform step 3 before step 1. Fill fields, submit, then modify fields that should have been locked.
4. BOUNDARY INTERACTIONS: Combine boundary values across multiple related fields (e.g., max quantity + discount code + coupon removal).
5. CONTRADICTORY ACTIONS: Apply and remove the same operation in rapid succession. Select and deselect while async handlers are in flight.
6. ROLE/PERMISSION BOUNDARIES: If role-related elements are visible, attempt to access or trigger admin/restricted actions from a standard context.
YOUR RESPONSE MUST BE VALID JSON matching this exact schema:
{
"hypothesis": "What specific bug you think might exist",
"targetWeakness": "Which attack category and why this UI state suggests vulnerability",
"actions": [
{
"type": "click|fill|select|clear|rapid-toggle|navigate|wait|keyboard",
"selector": "CSS selector from the state snapshot",
"value": "for fill/select/keyboard actions",
"repeat": "number, for rapid-toggle actions",
"delayMs": "milliseconds to wait before this action (0 for maximum speed)",
"reasoning": "Why this specific action at this point in the sequence"
}
],
"expectedAnomalies": ["What you expect to go wrong if the bug exists"]
}
RULES:
- Generate 4-8 actions per plan
- Every action MUST reference a selector from the provided state snapshot
- Include explicit reasoning for each action explaining the adversarial intent
- Vary your approach: do not repeat the same attack pattern across rounds
- If previous round results are provided, FOCUS on areas where anomalies were detected`;Calling the LLM and Parsing the Response
// Requires: npm install openai zod
import { z } from 'zod';
import OpenAI from 'openai';
const TestPlanSchema = z.object({
hypothesis: z.string(),
targetWeakness: z.string(),
actions: z.array(z.object({
type: z.enum(['click', 'fill', 'select', 'clear', 'rapid-toggle', 'navigate', 'wait', 'keyboard']),
selector: z.string(),
value: z.string().optional(),
repeat: z.number().optional(),
delayMs: z.number().optional(),
reasoning: z.string(),
})),
expectedAnomalies: z.array(z.string()),
});
// Ensure OPENAI_API_KEY is set in environment variables
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
async function generateAdversarialPlan(
stateSnapshot: string,
previousFindings: AnomalyReport[] = [],
maxRetries = 3
): Promise<TestPlan> {
const findingsContext = previousFindings.length > 0
? `
PREVIOUS ROUND FINDINGS:
${previousFindings.map(f =>
`- [${f.anomalyType}] ${f.description} (after action: ${f.action.type} on ${f.action.selector})`
).join('
')}
Focus your next attack on the areas where anomalies were found.`
: '';
const userMessage = `CURRENT UI STATE:
${stateSnapshot}${findingsContext}
Generate an adversarial test plan.`;
for (let attempt = 0; attempt < maxRetries; attempt++) {
try {
const response = await openai.chat.completions.create({
model: 'gpt-4o',
messages: [
{ role: 'system', content: ADVERSARIAL_SYSTEM_PROMPT },
{ role: 'user', content: userMessage },
],
temperature: 0.8,
response_format: { type: 'json_object' },
});
const content = response.choices[0]?.message?.content;
if (!content) throw new Error('Empty LLM response');
const parsed = JSON.parse(content);
const validated = TestPlanSchema.parse(parsed);
return validated;
} catch (error) {
console.warn(`Attempt ${attempt + 1} failed: ${error instanceof Error ? error.message : String(error)}`);
if (attempt === maxRetries - 1) throw error;
}
}
// This is unreachable due to the throw inside the loop, but satisfies the compiler
throw new Error('Failed to generate valid adversarial plan after retries');
}Zod validation is essential here, not optional. Without schema enforcement, malformed LLM responses (missing fields, wrong types, hallucinated action types) will cascade into runtime errors in the executor. I've seen the LLM invent action types like "hover-and-drag" or return selectors that reference elements it imagined rather than elements from the actual snapshot. The retry loop catches these.
Executing Adversarial Actions with Playwright
Translating the LLM's abstract action plan into concrete Playwright commands requires a dispatcher that handles each action type and, critically, distinguishes between "the UI broke" (a finding worth reporting) and "the action was invalid" (the planner made a mistake).
Rapid-toggle isn't a Playwright primitive. We implement it as repeated click, check, or uncheck operations with minimal delay between them. Speed matters here because many UI bugs surface specifically under timing pressure: debounce handlers that assume a minimum interval between interactions, async state updates that race with user input, or optimistic UI updates that conflict with server responses.
The Action Executor
interface ActionResult {
action: AdversarialAction;
success: boolean;
error: string | null;
}
async function executeAdversarialPlan(
page: Page,
plan: TestPlan,
screenshotDir: string
): Promise<{ results: ActionResult[]; anomalies: AnomalyReport[] }> {
const results: ActionResult[] = [];
const anomalies: AnomalyReport[] = [];
const consoleErrors: string[] = [];
const unhandledRejections: string[] = [];
const consoleHandler = (msg: import('playwright').ConsoleMessage) => {
if (msg.type() === 'error') consoleErrors.push(msg.text());
};
const errorHandler = (err: Error) => {
unhandledRejections.push(err.message);
};
page.on('console', consoleHandler);
page.on('pageerror', errorHandler);
try {
for (let i = 0; i < plan.actions.length; i++) {
const action = plan.actions[i];
const stateBefore = await extractUIState(page);
const errorsBeforeCount = consoleErrors.length;
const rejectionsBeforeCount = unhandledRejections.length;
if (action.delayMs && action.delayMs > 0) {
await page.waitForTimeout(action.delayMs);
}
try {
switch (action.type) {
case 'click':
await page.locator(action.selector).first().click({ timeout: 5000 });
break;
case 'fill':
await page.locator(action.selector).first().fill(action.value ?? '', { timeout: 5000 });
break;
case 'select':
await page.locator(action.selector).first().selectOption(action.value ?? '', { timeout: 5000 });
break;
case 'clear':
await page.locator(action.selector).first().clear({ timeout: 5000 });
break;
case 'rapid-toggle': {
const locator = page.locator(action.selector).first();
const repeatCount = action.repeat || 5;
for (let r = 0; r < repeatCount; r++) {
await locator.click({ timeout: 2000 }).catch(() => {});
}
break;
}
case 'navigate':
if (action.value) await page.goto(action.value, { timeout: 10000 });
break;
case 'wait':
await page.waitForTimeout(action.delayMs || 1000);
break;
case 'keyboard':
await page.keyboard.press(action.value || 'Enter');
break;
}
results.push({ action, success: true, error: null });
} catch (error) {
const errorMsg = error instanceof Error ? error.message : String(error);
results.push({ action, success: false, error: errorMsg });
}
// Brief pause to let async UI updates settle
await page.waitForTimeout(200);
const stateAfter = await extractUIState(page);
const newErrors = consoleErrors.slice(errorsBeforeCount);
const newRejections = unhandledRejections.slice(rejectionsBeforeCount);
const detected = detectAnomalies(stateBefore, stateAfter, action, newErrors, newRejections, i);
if (detected.length > 0) {
const screenshotPath = `${screenshotDir}/anomaly-round-${i}-${Date.now()}.png`;
await page.screenshot({ path: screenshotPath, fullPage: true });
for (const anomaly of detected) {
anomalies.push({ ...anomaly, screenshotPath, stateBefore, stateAfter });
}
}
}
} finally {
// Clean up listeners to avoid leaks across rounds
page.off('console', consoleHandler);
page.off('pageerror', errorHandler);
}
return { results, anomalies };
}Detecting Anomalies vs. Expected Failures
A validation error message appearing after invalid input is correct behavior, not a bug. A submit button becoming enabled while required fields are empty is a bug. The anomaly detector needs to distinguish between these by checking for logically contradictory states rather than just flagging any error.
function detectAnomalies(
before: UIStateSnapshot,
after: UIStateSnapshot,
action: AdversarialAction,
newConsoleErrors: string[],
newRejections: string[],
round: number
): Omit<AnomalyReport, 'screenshotPath' | 'stateBefore' | 'stateAfter'>[] {
const anomalies: Omit<AnomalyReport, 'screenshotPath' | 'stateBefore' | 'stateAfter'>[] = [];
// Unhandled promise rejections are almost always bugs
for (const rejection of newRejections) {
anomalies.push({
round, action,
anomalyType: 'unhandled-rejection',
description: `Unhandled rejection after ${action.type} on ${action.selector}: ${rejection.substring(0, 200)}`,
});
}
// Console errors that aren't network/CORS noise
for (const err of newConsoleErrors) {
if (!err.includes('net::ERR') && !err.includes('CORS')) {
anomalies.push({
round, action,
anomalyType: 'console-error',
description: `Console error: ${err.substring(0, 200)}`,
});
}
}
// Contradictory states: submit enabled while required fields are empty/invalid
const submitButtons = after.elements.filter(el =>
(el.tag === 'button' && el.text.toLowerCase().includes('submit')) ||
el.type === 'submit'
);
const invalidFields = after.elements.filter(el =>
el.validationState && el.validationState !== 'valid' && !el.disabled
);
const emptyRequiredFields = after.elements.filter(el =>
el.ariaAttributes['aria-required'] === 'true' && (!el.value || el.value.trim() === '')
);
for (const btn of submitButtons) {
if (!btn.disabled && (invalidFields.length > 0 || emptyRequiredFields.length > 0)) {
anomalies.push({
round, action,
anomalyType: 'contradictory-state',
description: `Submit button "${btn.text}" is enabled but ${invalidFields.length} fields are invalid and ${emptyRequiredFields.length} required fields are empty`,
});
}
}
// Elements that disappeared unexpectedly (possible state corruption)
const beforeSelectors = new Set(before.elements.filter(e => e.visible).map(e => e.selector));
const afterSelectors = new Set(after.elements.filter(e => e.visible).map(e => e.selector));
const disappeared = [...beforeSelectors].filter(s => !afterSelectors.has(s));
if (disappeared.length > 5 && action.type !== 'navigate') {
anomalies.push({
round, action,
anomalyType: 'state-corruption',
description: `${disappeared.length} elements disappeared after ${action.type} (non-navigation action). Possible state corruption or unhandled error boundary.`,
});
}
return anomalies;
}Closing the Loop: Iterative Deepening
A single pass of extract-plan-execute rarely finds the most interesting bugs. The power of this approach comes from feeding execution results back to the LLM so it can focus subsequent rounds on areas where it detected fragility.
When round one reveals a console error during rapid toggling of a specific toggle component, round two receives that context and generates a more targeted attack: "The toggle at selector X produced an error under rapid clicking. Now I'll try toggling it during an in-flight network request, then immediately navigating away." Each round narrows the search space.
For managing the LLM's context window across rounds, I've found that re-sending a condensed summary of the current state plus the most recent findings works better than maintaining a long continuous conversation. Long conversations accumulate stale state descriptions and push the model toward repetition. A fresh prompt with targeted context produces more creative attacks.
Set convergence criteria to avoid burning API credits on diminishing returns. I use a two-condition stop: halt after a maximum of 8 rounds, or halt after 3 consecutive rounds that produce no new anomalies (deduped by anomaly type plus target selector hash to avoid counting the same bug rediscovered with slight variations).
Multi-Round Orchestration
// Requires: npm install playwright openai zod
// All interfaces (UIElement, UIStateSnapshot, AdversarialAction, TestPlan,
// AnomalyReport, ActionResult) and functions (extractUIState, compressSnapshot,
// generateAdversarialPlan, executeAdversarialPlan) defined above must be in scope.
import { chromium } from 'playwright';
import fs from 'fs';
async function runAdversarialSession(
targetUrl: string,
options: {
maxRounds?: number;
convergenceThreshold?: number;
screenshotDir?: string;
maxWallClockMs?: number;
} = {}
): Promise<{ allAnomalies: AnomalyReport[]; roundCount: number }> {
const {
maxRounds = 8,
convergenceThreshold = 3,
screenshotDir = './adversarial-screenshots',
maxWallClockMs = 5 * 60 * 1000, // 5 minute max
} = options;
if (!fs.existsSync(screenshotDir)) fs.mkdirSync(screenshotDir, { recursive: true });
const browser = await chromium.launch();
const context = await browser.newContext();
const page = await context.newPage();
await page.goto(targetUrl, { waitUntil: 'networkidle' });
const allAnomalies: AnomalyReport[] = [];
let consecutiveCleanRounds = 0;
let consecutivePlannerFailures = 0;
const maxPlannerFailures = 3;
const startTime = Date.now();
let roundCount = 0;
for (let round = 0; round < maxRounds; round++) {
if (Date.now() - startTime > maxWallClockMs) {
console.log(`Wall clock limit reached at round ${round}`);
break;
}
roundCount = round + 1;
console.log(`
=== Adversarial Round ${roundCount} ===`);
// Reset to known state for each round
await page.goto(targetUrl, { waitUntil: 'networkidle' });
const snapshot = await extractUIState(page);
const compressed = compressSnapshot(snapshot);
// Feed only recent findings to keep context focused
const recentFindings = allAnomalies.slice(-10);
let plan: TestPlan;
try {
plan = await generateAdversarialPlan(compressed, recentFindings);
consecutivePlannerFailures = 0;
} catch (error) {
console.warn(`Planner failed in round ${roundCount}, skipping: ${error}`);
consecutivePlannerFailures++;
consecutiveCleanRounds++;
// Circuit breaker: abort if planner keeps failing
if (consecutivePlannerFailures >= maxPlannerFailures) {
console.error(`Planner failed ${maxPlannerFailures} consecutive times, aborting session.`);
break;
}
if (consecutiveCleanRounds >= convergenceThreshold) break;
continue;
}
console.log(`Hypothesis: ${plan.hypothesis}`);
console.log(`Target: ${plan.targetWeakness}`);
console.log(`Actions: ${plan.actions.length}`);
const { anomalies } = await executeAdversarialPlan(page, plan, screenshotDir);
// Dedupe by anomaly type + selector
const newAnomalies = anomalies.filter(a => {
const signature = `${a.anomalyType}:${a.action.selector}`;
return !allAnomalies.some(existing =>
`${existing.anomalyType}:${existing.action.selector}` === signature
);
});
if (newAnomalies.length > 0) {
console.log(`Found ${newAnomalies.length} new anomalies`);
allAnomalies.push(...newAnomalies);
consecutiveCleanRounds = 0;
} else {
consecutiveCleanRounds++;
console.log(`Clean round (${consecutiveCleanRounds}/${convergenceThreshold} toward convergence)`);
}
if (consecutiveCleanRounds >= convergenceThreshold) {
console.log('Convergence reached, stopping.');
break;
}
}
// Generate final report
console.log(`
=== SESSION COMPLETE ===`);
console.log(`Rounds: ${roundCount}`);
console.log(`Total unique anomalies: ${allAnomalies.length}`);
for (const a of allAnomalies) {
console.log(` [${a.anomalyType}] ${a.description}`);
}
await browser.close();
return { allAnomalies, roundCount };
}
// Run it
runAdversarialSession('http://localhost:3000/checkout', {
maxRounds: 6,
convergenceThreshold: 2,
}).then(({ allAnomalies }) => {
process.exit(allAnomalies.length > 0 ? 1 : 0);
});Add a circuit breaker for planner failures: if the LLM returns structurally invalid output three times in a row (not just once per round, but across rounds), the session should terminate. Without this safeguard, a prompt that consistently confuses the model will burn through your entire API budget producing garbage.
Real-World Results: What This Actually Finds
We ran this framework against three internal applications that all had conventional Playwright test suites with line coverage above 90%. The results were humbling.
Multi-step form wizard. The LLM hypothesized that clicking "Back" during an in-flight async validation and then immediately clicking "Next" might bypass field requirements. It was right. The application used a validating boolean flag to disable the "Next" button, but the "Back" button was never disabled during validation. Clicking "Back" canceled the validation promise, and the validating flag reset to false without re-running validation. The next "Forward" navigation treated the field as valid because no validation error existed in state. This was a real bug in production for months, covered by tests that always waited politely for validation to complete.
The LLM constructed a sequence: apply discount code, remove all items from cart, undo the removal (the app had an undo feature), then proceed to checkout. The discount amount was calculated when the code was applied and stored as an absolute dollar value. When items were restored via undo, the cart total was recalculated, but the discount was re-applied using the stale absolute value rather than recalculating from the percentage. For a specific combination of items this meant the discount exceeded the new subtotal, producing a negative amount owed.
E-commerce shopping cart. The LLM constructed a sequence: apply discount code, remove all items from cart, undo the removal (the app had an undo feature), then proceed to checkout. The discount amount was calculated when the code was applied and stored as an absolute dollar value. When items were restored via undo, the cart total was recalculated, but the discount was re-applied using the stale absolute value rather than recalculating from the percentage. For a specific combination of items this meant the discount exceeded the new subtotal, producing a negative amount owed.
Role-based admin dashboard. The LLM discovered that switching between the "Admin" and "Standard User" role previews in rapid succession (7 toggles in under 2 seconds) caused a race condition in the role-based component renderer. Admin-only action buttons appeared in the standard user view because the async role-check resolved after the UI had already rendered the previous role's component tree, and the state update from the earlier toggle overwrote the correction from the later one.
Across all three applications, the framework identified 14 unique logic bugs. None of them would have been caught by adding more deterministic test cases, because no developer would have thought to write the specific interaction sequences that triggered them. Every discovered bug was subsequently codified as a standard Playwright regression test, with the exact action sequence recorded from the adversarial session's logs.
Limitations, Costs, and When Not to Use This
API costs are real but manageable. Each adversarial session runs 5 to 15 LLM calls depending on convergence. With GPT-4o, the compressed state snapshots typically run 300-800 tokens input and responses run 200-500 tokens. Costs vary and change frequently, so check current pricing before budgeting. A session against one component is typically cheap per run, but running this against 20 components nightly adds up.
Non-determinism is a feature and a limitation. The same session won't find the same bugs twice, which means great coverage over many runs, but it also means you can't use this as a pass/fail gate in CI. Treat it as a nightly exploration job that files issues, not a blocking check.
This doesn't replace your deterministic regression suite. Adversarial sessions discover bugs. Once discovered, the correct action is to extract the exact action sequence and write a standard, deterministic Playwright test that reproduces it every time. The adversarial framework is the explorer. Your test suite is the guardrail.
This doesn't replace your deterministic regression suite. Adversarial sessions discover bugs. Once discovered, the correct action is to extract the exact action sequence and write a standard, deterministic Playwright test that reproduces it every time. The adversarial framework is the explorer. Your test suite is the guardrail.
LLM hallucination is a constant. The planner will sometimes generate selectors that don't exist, action types that make no sense for the target element, or "adversarial" plans that are actually just normal usage. The executor must handle all of this gracefully. About 15-20% of generated action plans in my experience contain at least one invalid action.
Data governance matters. You're sending DOM content, visible text, and potentially user data to a third-party API. If your application displays PII, medical data, or anything regulated, you need to strip it from snapshots before sending. Build a sanitization layer between the state extractor and the planner. Depending on your jurisdiction, this may also implicate GDPR, HIPAA, or other data handling requirements even if the data is only transmitted transiently.
Performance puts this firmly in the nightly category. A full session takes 2 to 5 minutes per component. This isn't pre-commit, and it isn't even per-PR. Run it overnight, review findings in the morning.
From Predictable Tests to Adversarial Resilience
What we built here is a three-component loop: a Playwright state extractor that captures semantically meaningful UI snapshots, an LLM adversarial planner that reasons about those snapshots to generate hostile interaction sequences, and a Playwright action executor that runs those sequences and detects anomalies. The feedback loop between rounds lets the system progressively zero in on fragile areas.
The immediate next step: pick one complex, stateful UI component in your application, something with a multi-step workflow, async validation, or role-based rendering. Run a single adversarial session against it. You will almost certainly find something your existing tests missed.
For longer-term development, consider logging every adversarial session's complete artifacts (prompt, state snapshot, plan JSON, execution trace, screenshots) as a training corpus. Over time, you can identify which attack categories are most productive against your specific codebase and weight the system prompt accordingly. Integration into CI/CD as a nightly exploration run with findings posted to Slack or filed as issues closes the loop between discovery and resolution.
LLMs are most powerful in testing not when they write the tests a human would write, but when they generate the tests a human would never think to write. Your deterministic suite verifies that the application works correctly when used correctly. The adversarial framework verifies that it fails gracefully when used creatively, chaotically, and with hostile intent. Both are necessary. Only one of them is probably missing from your pipeline right now.
Here's what it comes down to: LLMs are most powerful in testing not when they write the tests a human would write, but when they generate the tests a human would never think to write. Your deterministic suite verifies that the application works correctly when used correctly. The adversarial framework verifies that it fails gracefully when used creatively, chaotically, and with hostile intent. Both are necessary. Only one of them is probably missing from your pipeline right now.