01 / AI EVALUATION · IN PROGRESS
Synthetic UAT
Can AI agents acting as synthetic users find the same usability friction as real people?
THE BRIEF / 60-SECOND OVERVIEW
Same task. Two interfaces. Real people and synthetic ones.
The question
Testing with real users and experts is slow and expensive. If AI agents can surface the same friction, teams get a faster first read. So where do agents and people hit the same friction when configuring a cloud budget, and where do they not?
My part
As team lead and UX researcher and designer, I scoped the study, designed the test platform and proof of concept, built the survey that fed the synthetic personas, ran the human sessions, and interviewed UX experts. The study is still running, so comparative findings aren’t available yet.
CONTEXT
My role
UX Researcher & Designer · Team lead · Contract project at Google
- Team · 7 people: 5 in UX, 2 in data science
- Timeline · August – December 2026 (in progress)
- Scope of my work · Research design, platform and prototype design, survey, human sessions, expert interviews, and leading the team
01 / WHY SYNTHETIC USERS
Real users are the best signal, and the slowest.
Recruiting people and experts to test a workflow takes a long time and costs a lot. If an AI agent playing a realistic persona could point to the same trouble spots, a team could check a design in hours instead of weeks.
But a faster read only helps if we know when to trust it. The question I care about isn’t whether an agent can click through a form. It’s whether the friction it reports matches the friction a person actually feels.
02 / RESEARCH QUESTION
A task can look finished and still be wrong.
A saved cloud budget can monitor the wrong costs, notify the wrong people or trigger on forecast spend when the user intended actual spend. A generic completion event misses those differences.
The study asks whether humans and AI agents run into similar friction, and whether a guided workflow changes where it occurs. The shared task gives both groups a concrete goal without prescribing their clicks.
03 / WHAT I DID
From survey to synthetic persona to side-by-side comparison.
- Surveyed the people we were designing for. I wrote the survey for our target persona and collected data from human participants, which became the basis for the synthetic personas.
- Designed the test platform. I designed the UAT platform for billing configuration, used with both human participants and synthetic personas, and the proof-of-concept (POC) platform.
- Ran human UAT sessions. I conducted and observed the sessions, then compared what I saw with the reports from the synthetic personas.
- Asked experts to challenge it. I interviewed UX experts about the POC prototype we were building and collected their suggestions.
04 / SCOPING DECISION
One persona, one workflow.
I scoped the proof of concept to a single persona and a single workflow, cloud billing configuration. The aim was a study that fit the project’s scope while still exercising most of our testing criteria.
Why it matters. Narrowing the study means a difference between people and agents can be traced to something specific, instead of disappearing into many personas and tasks.
05 / DESIGN TRADE-OFF
One task, two ways to configure it.
The advanced form keeps related settings visible together. The guided flow asks one question per screen and adds a review step. Task wording, fixture data, surrounding screens and saved configuration stay consistent.
SAME TASK · FICTIONAL BILLING ACCOUNT
Set a $1,000 monthly budget for Atlas. Alert only its project owner when actual spend exceeds 80%.
A / DENSE CONFIGURATION
See the settings together.
A compact form supports fast scanning for experienced users. Users must notice how scope affects recipient eligibility and interpret role-based options.
B / GUIDED QUESTIONS
Decide one step at a time.
A sequence of questions reduces what’s visible at once. It introduces extra navigation and makes the review step important for catching dependencies.
Illustrative walkthrough on a fictional account. Both prototypes save the same budget configuration.
What the comparison can isolate. The two flows change several things together, including navigation, hierarchy and wording. A difference would justify investigating the guided experience as a whole. A follow-up experiment could isolate the effect of a single change.
06 / EVALUATION
Score the outcome, then inspect the path.
The same evaluator checks the saved configuration for both interfaces. Recipient accuracy comes from the set of people who would actually get an alert, since different controls can lead to the same recipients.
Screen-by-screen comparisons would favor whichever design splits the task into more pages. Comparing friction by setting makes it possible to ask where people and agents struggle with scope, alert rules or recipients across both versions.
Agents interact through screenshots, mouse and keyboard. The platform logs actions and distinguishes a bad configuration from interruptions such as a timeout or loop.
07 / PLATFORM
Keep the experiment interpretable.
The interface stores the assigned version on the session. Both variants use the same configuration shape and evaluator. Saved configurations are scored independently from whether the application accepted them, so a valid budget can still fail the assigned task.
The research console supports session creation, event inspection, re-scoring and export. Agent action records include failures such as timeouts or invalid actions. This helps separate an interaction failure from an incorrect budget decision.
08 / HOW I’LL READ THE RESULTS
What would change the design direction?
If both groups miss the same recipient dependency, that would motivate testing how scope and recipient consequences are explained. If only agents miss it, the next investigation should inspect perception and action traces before generalizing the result to people.
If the guided flow improves accuracy but takes longer, the product decision depends on the cost of a wrong configuration and the needs of frequent users. Completion time and final accuracy should be interpreted together.
Disagreement matters as much as agreement. A model may misread a visible control that a person understands, while a person may hesitate over the consequences of sending an alert. That distinction decides which findings need direct human follow-up.
STATUS
Still running, and under NDA.
The study is still in progress, so there are no comparative results to report. Because of the NDA, I can’t share participant numbers, findings or product screens, only the concepts and the work described above.
If you’d like to hear more about the approach, contact me at sc3625@cornell.edu.