Artificial intelligence for automatic testing

A first draft of a test can now often come from a plain description of what should happen — "go to the search page, search for X, check the results mention X" — rather than starting from a blank file. Several tools do this today, including Playwright's own codegen, which records real clicks into a script, combined with an AI layer that lets you describe intent in natural language and get a starting point to edit.

Some tools also apply AI to locators — the selectors a test uses to find an element on the page. A "self-healing" locator uses AI to re-locate an element when its selector changes, for example when a developer renames an id, instead of the test simply breaking.

Once a test suite is running in CI/CD on every commit, new problems show up that only exist at scale, and AI-based analysis has started to help with a few of them:

• Flaky-test detection. Distinguishing "this test fails sometimes because of a real intermittent bug" from "this test is just badly written" by analyzing historical pass/fail patterns, so a team doesn't waste hours chasing a ghost that isn't actually there.
• Intelligent test selection. In a large test suite, running only the subset of tests actually likely to be affected by a given code change, to keep CI/CD fast instead of re-running everything on every commit.
• Log and anomaly analysis after a deploy. Flagging unusual error-rate or latency patterns automatically, rather than relying on a human watching a dashboard and noticing something looks off.

See Continuous integration for automatic testing for what CI/CD is in the first place, if that term is new to you.

Before a test can be written, someone has to decide what's worth testing. AI can help with the step before that: synthesizing a pile of user interviews, support tickets, or feedback into concrete, testable acceptance criteria. "As a patient, I want to search NHS Wales for a service" is a vague goal; "search for a service and confirm the results page shows it" is something a test can actually check. AI can help draft that translation.

This site's own sibling repos are a real worked example of the destination, even though they were written by hand rather than by an AI-discovery step: someone identified real user journeys on a real public site — find the home page, find "About Us", search for help — and turned them into concrete, asserted test scenarios. See About for the full list, or demo-playwright-javascript-for-nhs-wales directly.

AI can help with the "turn a vague idea into a concrete scenario" step, but a human still has to confirm the AI understood the actual user need correctly. That step doesn't disappear — it moves, from writing the acceptance criteria by hand to reviewing the ones a tool drafted.

This site itself is a concrete example of that discipline being applied, not just described. This site was built with AI assistance (Claude), under a human maintainer's direction, and AI_STATEMENT.md in the repository is the honest account of how — what was AI-generated, what the human decided, and what was actually verified rather than assumed.

Back to Learn How to start learning →