Locator-based vs vision-based mobile app testing
When a test needs to tap a button, it has to find the button first. There are two ways to do that. It can ask the app for its element tree and look the button up there. Or it can take a screenshot and look at it.
Until a few years ago, only the first one worked.
Why locator-based test automation won
Selenium came out in 2004. It was built for the web, where the DOM is the page: a button is an element, it has an id, you look it up and click it.
Mobile app testing copied this model. Appium runs on the WebDriver protocol, the same one Selenium uses, so mobile tests were written the same way as web tests. Espresso, XCUITest and Detox all do the same thing underneath: find the element in the tree, then act on it.
There was a good reason to build it this way. In 2004 a machine could not look at a screen. Computer vision could match a template or read text with OCR. It could not tell you that the rounded rectangle with “Login” written in it is a button you can tap. The element tree was the only description of a UI that software could read.
And it works well. A tree lookup takes under a millisecond and costs nothing. It returns the same element every time, or it fails. There is no guessing. Millions of these tests run in CI right now and catch real bugs. If your tree is stable, this is the right tool and there is no reason to change.
Why locators break
Locators are flaky
A locator identifies an element by its place in the app’s structure: a resource id on Android, an accessibility identifier on iOS. The test looks the element up by that identifier and acts on it. These identifiers live in the app’s code, so when the code changes (an id renamed in a refactor, a view moved) the locator stops matching and the test fails even though the app works. iOS and Android expose different structures, so the same flow needs a separate set of locators on each.
Modals change the structure at runtime too. A permission dialog, an ATT prompt, or a rate-us popup appears over the current screen, and the element a test is looking for isn’t there on that run. Because they appear conditionally, the same test can pass one run and fail the next. Coding agents can now rewrite a failed locator automatically, which makes the fix faster, but it doesn’t change how often the failures happen.
Some screens have no tree
On mobile, the tree was never the UI itself. It’s a description of the screen, generated alongside it for accessibility tools. Usually it’s there. Sometimes it isn’t.
A Unity or Unreal game draws a texture that looks like a button, and there is no element behind it to look up. Canvas UIs are the same. And anything outside your app was never in your tree to begin with: a payment sheet, an OAuth page, the system keyboard. Login and checkout flows run straight through those. There’s nothing to query, so a locator-based tool can’t run on these screens at all.
How vision-based mobile testing works
LLMs changed what a machine can read. An LLM can now look at a screenshot and tell you: this is a login form, that’s the primary button, that’s the OS asking for camera permission. Not perfectly, but well enough to act on.
So a vision-based test runs a loop: take a screenshot, decide the next action, tap or type, screenshot again to check what happened.
This removes several of the problems above at once:
- No ids to add. Any build is testable as-is; no developer has to make a screen testable first.
- One test for both platforms. The iOS and Android screenshots both show a login button.
- No popups to handle. A dialog on top of the screen is just what the next screenshot shows.
- Test cases in plain language. Sign in with the saved account and open the Orders tab. No code, so a manual QA can write them without a programmer.
Manual vs locator-based vs vision-based
Mobile app testing has three options now, not two.
| Manual QA | Locator-based | Vision-based | |
|---|---|---|---|
| Who writes the test | Anyone who knows the flow | An automation engineer | Anyone who knows the flow |
| Cost to write and maintain | Low | Low per fix, but never stops | Low |
| Cost per run | A person’s hour, every time | Near zero | Near zero (some LLM calls) |
| Speed per action | Seconds | Milliseconds | Seconds |
| Runs in parallel | Hire more people | Yes | Yes |
| Games, OS dialogs, canvas | Yes | No | Yes |
| Repeatable | Varies | Exactly | Varies |
Manual: cheap to write, expensive to run. A checklist takes a minute to write, then costs a person’s hour on every run. It doesn’t scale: twenty flows cost twenty times the hours, so teams test the few flows they worry about most instead of all of them.
Locator-based: cheap now on both. Maintenance used to be the tax, and agents that rewrite locators have cut it. But it only works where there’s a stable element tree. No tree, no test.
Vision-based: cheap on both too, and it needs no tree, so it reaches what locators can’t: games, canvas, OS dialogs, one flow across both platforms. Less repeatable than locators, and it replaces the manual pass, not the locator suite.
Choosing a mobile testing approach
The two automated approaches are complementary. Cost no longer separates them, coverage does. Use locators where the tree is stable; use vision where it isn’t, or where the alternative is testing by hand.
Once a test can read the screen, things you could only check by hand become automatable: playing through a level, editing a photo, following a pin on a map, or running a sign-up end to end through the email code, the paywall, and the permission popup. Each of these used to need a person before every release; now each is a sentence that runs unattended.
The open question is the last row of the table: reproducibility. A locator lookup returns the same element or fails. An LLM makes judgement calls, and it can reach the same goal by a different route on the next run. How much that matters, and how to constrain it without losing the flexibility that makes vision useful, is a topic on its own. That’s a separate post.
Want to see vision-based testing on your own app? Start free: describe a flow in one sentence and watch the agent run it on a real device, including on games and screens with no element tree. New to the idea? Read what agentic testing actually means.