The effect that users of an AI tool feel and the outcome that you measure move separately. Good preference and speed metrics do not guarantee good accuracy and conformance metrics, so design the two metric types separately before you decide on adoption.
After a team adopts an AI tool, the first evidence to arrive is usually user surveys and task times. Preference and confidence are effects that users feel directly. Speed is an effect that shows immediately after the task. Learning, error rate, and rule conformance need a separate judgment procedure before you can measure them.
It is easy to assume that good results on the first group of metrics mean good results on the second group. Two studies published in 2026 show, in different ways, that you cannot accept this assumption as given.
One study measured both metric types on the same participants and found that they diverged. The other study measured speed directly but did not directly measure the design consistency named in its title.
A table for measuring time, quality, rework, and cost per task is in What to Check When Using AI at Work. This post gives criteria to put each metric in that table on the correct axis. It also gives criteria to find what reported figures leave out.
An Experiment Where Preference and Learning Diverged
Cipriano et al. (arXiv:2608.25382, 2026-08, accepted at ASSETS 2026) compared two interfaces with 16 screen reader users. The participants were 13 totally blind and 3 low-vision users, and the study used a within-subjects design: each person used both interfaces. The order of interfaces and fictional worlds was counterbalanced, and each participant did two 90-minute sessions on different days.
The study compared a hyperlinked Document Interface (DI) with a Question-Answer Interface (QAI). There were two fictional worlds, and each world had 25 documents, including an index.
The QAI used Gemini 2.5 Pro, a large language model (LLM). All 25 documents were loaded into its context in advance, and a prompt limited its answers to content in the documents. The authors report that they observed no hallucinations or incorrect responses from the QAI.
After exploration, each participant described the world without access to the interface. A researcher used that description to build a concept map and count topics, connections, and errors. Then participants applied their knowledge to a policy dilemma task.
| Measured metric | DI | QAI | Reported test result |
|---|---|---|---|
| Unique documents visited | 11.44 | 9.31 | p=0.03, η²=0.27 |
| Total document visits | 17.31 | 17.81 | p=0.85 |
| Concept map topics | 9.44 | 5.63 | p=0.02, η²=0.78 |
| Concept map subtopics | 9.63 | 8.69 | p=0.55 |
| Concept map errors | 3.31 | 6.00 | p<0.01, η²=0.62 |
| Concept map error rate | 0.12 | 0.26 | p<0.01, η²=0.71 |
| Application task error rate | 0.03 | 0.29 | p=0.01 |
Values are means over the 16 participants, and η² is the effect size. For topics, the p value is 0.02 in the text and 0.00 in the figure, so this table uses the text value. The authors also report an error rate because the concept maps differed in size, but the text does not state its denominator.
The application task was difficult, and 6 participants gave no coherent answer in at least one session. Of the 5 who failed in only one session, 4 failed in the QAI session. The paper does not say how the metrics include these cases.
The QAI responses contained no observed incorrect answers. Thus the added errors probably came from the step where participants combined answers into their own understanding. Subtopic counts were similar and only topic counts fell, so participants probably formed fewer higher-level categories that group the facts. Both of these readings are judgments.
Perception Against Measurement
At the end of the second session, participants said which interface helped them explore more, learn more, and answer with more confidence. For the same items, the authors counted which interface gave each participant better measured results. Then they compared the two counts.
| Item | Perceived DI | Perceived QAI | Measured DI better | Measured QAI better | Measured equal | Perception matched measurement |
|---|---|---|---|---|---|---|
| Explored more | 10 | 6 | 11 | 4 | 1 | 8 (DI 6, QAI 2) |
| Learned more | 6 | 10 | 11 | 5 | 0 | 9 (DI 5, QAI 4) |
| Answer confidence | 7 | 9 | 12 | 1 | 3 | 5 (DI 5, QAI 0) |
For exploration, the majority pointed in the same direction (10 perceived DI, 11 measured better with DI), but only 8 of 16 participants matched individually. For learning, 10 participants said they learned more with the QAI, but only 5 had better measured results with it. For confidence, 9 participants chose the QAI, and none of them had better measured results with the QAI.
Overall preference was close to even: 8 to 8 in the text, and 7 for DI to 8 for QAI in the table. The authors report no clear mapping between perceived advantages and final preference. The QAI had lower mean measured outcomes, but it did not lose on preference.
The authors propose two hypotheses for the cause. First, the DI required more effort, which possibly caused deeper processing. Second, the QAI removes the structure that documents provide in advance. Users must then build that structure through their questions, and this possibly leaves fewer cognitive resources for learning.
In their conclusion, the authors write that conversational question answering can feel broader than it is. They also recommend that accessible technologies be evaluated on accurate understanding and exploration, not only on speed and convenience.
An Experiment That Measured Only Speed Directly
Silva et al. (arXiv:2607.13156, 2026-07, SBES Industry 2026) ran a controlled experiment at Zup Innovation, a Brazilian technology company. The authors put 49 front-end and mobile developers in three groups: 7 manual, 21 with only the Design System (DS), and 21 with AI assistance. The AI group used StackSpot AI, an internal tool contextualized with the company design system.
The task was to implement two screens from a high-fidelity mockup in each developer's stack (Angular, iOS, or Android). The experiment ran in two cycles, in June and October 2025, and the manual group took part only in the first cycle. The analysis combines both cycles.
Participants submitted each section, and specialists reviewed it immediately. A failed section went back to the participant while the timer continued. Thus the time-to-delivery is the time until the work passed review, and it includes rework and breaks.
| Stack | Manual mean (min) | DS mean (min) | AI mean (min) | AI vs. manual reduction | AI vs. DS reduction |
|---|---|---|---|---|---|
| Angular | 536.15 | 216.75 | 164.00 | 69.4% | 24.4% |
| iOS | 593.00 | 374.00 | 316.00 | 46.7% | 15.5% |
| Android | 387.75 | 213.25 | 163.25 | 57.9% | 23.4% |
Mean task completeness, as judged by specialists, was 68% for manual, 85% for DS, and 96% for AI. The authors report that the AI group was significantly faster than both the manual and DS groups (p<0.05). In this experiment, speed and completeness moved in the same direction.
What Was Reported and What Is Missing
The paper title says that it evaluates productivity and design consistency. But the only quantitative metrics in the text are time-to-delivery and specialist-judged completeness. The authors present completeness as evidence of consistency. But no metric directly measures design system conformance, such as token usage rate or component rule conformance rate.
| Claim in the paper | Evidence given | What is missing |
|---|---|---|
| 46.7 to 69.4% shorter delivery | Mean and standard deviation per stack, p<0.05 | Test name, p value per stack, effect size, participants per stack |
| Higher completeness | Specialist-judged means of 68%, 85%, 96% | Scale and criteria, automated or multi-reviewer judgment |
| Design consistency | Statements in the title and conclusion, completeness that includes visual fidelity (Table 2) | A direct metric such as a conformance rate |
| Avoided non-compliant code suggestions | A statement in the lessons learned | Counts per suggestion |
| Lower performance variability | Absolute standard deviation | Variation relative to the mean, a variance test |
| Less workflow friction | Supervisor observation of breaks | Break frequency figures, a workload scale |
| Effect of contextualization | Name of the internal tool | Context injection method, base model |
You can check the variability claim again with the coefficient of variation, which is the standard deviation divided by the mean. The values below come from the means and standard deviations in Table 1 of the paper.
Coefficient of variation (CV) = standard deviation / mean (calculated from Table 1 of Silva et al.)
Stack Manual DS AI
Angular 27.0% 39.2% 25.6%
iOS 27.0% 29.9% 16.1%
Android 28.4% 35.2% 21.4%For iOS and Android, the AI group was also the most stable relative to its mean. For Angular, the AI value of 25.6% is close to the manual value of 27.0%. Thus relative variability barely changed, and the smaller standard deviation mostly follows the shorter mean time. In all three stacks, the DS group had the highest coefficient of variation.
The sample composition is also a condition for reading the figures. The 7 manual developers are split across three stacks, so each stack has fewer than 3 on average. The paper does not report the count per stack. The mean of this small group is the denominator of the 46.7 to 69.4% reductions.
The paper gives an 8-hour workday as the time limit for the task. It does not explain why the manual means for Angular and iOS (536.15 and 593.00 minutes) are more than 480 minutes. This experiment measured the speed axis directly and filled the conformance axis with specialist judgment and statements.
Conditions Where the Claim Does Not Hold
The two studies had 16 and 49 participants, and each is one experiment by one research team or one company. This evidence is not sufficient to generalize that perception and outcomes diverge for all AI tools. Under the conditions below, the claim of this post becomes weaker or does not hold.
First, the two axes can move in the same direction. In Silva et al., shorter delivery and higher completeness occurred together. The claim of this post is that you cannot estimate one axis from the results of the other. It does not say that the two axes always move in opposite directions.
Second, the concept map outcomes in Cipriano et al. measure the understanding that remained when participants could not use the interface. For work where people find answers with the tool turned on, the results can be different, and this study did not measure that condition. The authors also told participants not to focus on recall during exploration, and they note that instructions that emphasize memorization can give different outcomes.
Third, the participants and the period were narrow. All participants were screen reader users, and the authors write that more research is necessary to find differences with sighted users. The sessions were two periods of 90 minutes, so usage patterns and outcomes after longer use are unknown.
Fourth, design choices are mixed into the results: only the DI had tactile input for spatial documents. The index document, which linked to all other documents, possibly affected how people navigated the DI. The authors write that they cannot separate the effects of these two choices. No hallucinations were observed in the QAI, so the gap can become larger or smaller in a real service with model errors. That last point is an estimate.
Fifth, the statistical reporting is incomplete. In Cipriano et al., some t test results use irregular notation. The authors also compared many metrics on 16 participants without a reported correction for multiple comparisons. Silva et al. do not name their test or report effect sizes. Reading the test results of both studies as exploratory comparisons is a judgment.
Before LLMs, research also found that preference and performance are not highly correlated. In a 2001 article, Nielsen analyzed 113 pairwise comparisons of user interfaces and found a correlation of 0.44 between measured performance and stated preference.
The square of 0.44 is about 0.19, so preference explains about 19% of the variance in performance. That source does not cover AI tools, so this post uses it only to support the direction of the claim.
The two studies leave some questions open. Cipriano et al. did not examine rule conformance, and Silva et al. did not report how participants perceived the tools. The relation between perception and conformance is still an open question. Whether hybrid interfaces or user training make the gap smaller is also open.
Criteria for Designing Separate Metrics
When you design an adoption evaluation, first mark the axis of each metric. Do not draw a conclusion about one axis from the results of another axis.
| Axis | Example metrics | Collection method | Check during design |
|---|---|---|---|
| Preference and confidence | Preferred tool, perceived learning, answer confidence | Surveys and interviews | Use as evidence of adoption intent, not as evidence of outcomes |
| Speed | Task time, delivery time | Execution records and timers | Define it as the time until the work passes review, so it includes rework |
| Accuracy | Error count and error rate, understanding | Independent judges; if the work requires understanding, a check task done without the tool | Fix the error definition and denominator before measurement |
| Conformance | Token and component rule conformance rate, check pass rate | Automated checks and a judgment rubric | Record automated judgment and human judgment separately |
If you collect both axes from the same participants, you can count matches between perception and measurement, as in the Cipriano et al. table. If the match rate is low, do not use survey results in outcome reports. A method to check the conformance of code output with automated checks and review is in Verifying AI-Generated Code.
When you read reported figures, use these questions to find what is missing.
- Does each claim in the title and abstract have a metric that measures it directly? If not, accept that claim only as a statement.
- Did the study compare perceived and measured metrics for each participant? If not, check whether it estimated outcomes from satisfaction.
- Are the test name, effect size, and count per group given? With only a p value, you cannot judge the size and stability of a difference.
- Does a variability claim depend only on absolute standard deviation? When the mean falls, the standard deviation can also fall, so check the coefficient of variation.
- Does the report describe the intervention? Without the context injection method and base model, there is no basis to transfer the effect to another tool.
- Did the study measure outcomes with the tool in use, or without it? The two conditions answer different questions.
How Companies Operate Coding Agent Harnesses describes how companies track verification pass, rework, and operational quality for coding agents across many teams.
References
- Q&A or Document-Based? The Effects of Interface Type on How Screen Reader Users Access Interconnected Documents, arXiv:2608.25382 (2026-08, accepted at ASSETS 2026)
- Design-System-Aware Development with AI: Evaluating Productivity and Design Consistency, arXiv:2607.13156 (2026-07, SBES Industry 2026)
- First Rule of Usability? Don't Listen to Users, Nielsen Norman Group (2001-08)
Every figure was checked against each source as of 2026-10. The coefficients of variation were calculated from the means and standard deviations in Table 1 of Silva et al.
Summary
With AI tools, preference, confidence, and speed can move separately from learning, error rate, and rule conformance, so you cannot estimate one from the other. In Cipriano et al., 10 participants said they learned more with the QAI, but QAI concept maps had fewer topics (5.63 vs. 9.44) and a higher error rate (0.26 vs. 0.12). Silva et al. measured 46.7 to 69.4% shorter delivery times, counted until the work passed review. They did not directly measure the design consistency named in the title.
The evidence comes from one research team's experiment with 16 participants and one company's experiment with 49 participants. In one of them, the two axes moved in the same direction. Thus the conclusion of this post goes only as far as this: design the two axes separately and report them separately. When you read a report, check that each claim has a direct metric and that the test, effect size, and intervention are described.