Every principal who has piloted an AI agent eventually gets asked the same question by a partner: is it worth what we're paying for it. Most practices answer with a guess dressed up as a number, usually some version of hours saved per week multiplied by a billing rate. That number is easy to produce and it rarely holds up to scrutiny, because it measures the wrong thing.
The Wrong Numbers to Start With
Vendor dashboards love to report volume: documents processed, RFIs logged, pages reviewed. None of that tells a practice whether the agent changed anything that shows up on a project's bottom line. A practice can process twice as many submittals through an agent and still lose the same amount of money to the same category of coordination error, because volume and value aren't the same measurement.
Time Saved Is the Easiest Number and the Least Useful One
The instinct is to time a task before and after, then multiply the difference by however many times a month the task happens. It feels rigorous. It usually isn't, because architects on salary don't convert saved hours into saved dollars automatically. The hour a project architect used to spend cross-checking a spec section doesn't disappear from the fee. It gets spent on something else, sometimes something billable, sometimes not, and the firm rarely tracks which.
This is why so many pilots report glowing time savings in month one and then struggle to point to anything different in the firm's financials by month six. The time was real. The value was never isolated from everything else happening on the project at the same time.
What to Measure Instead
The more honest metrics sit closer to risk and margin than to speed. Rework avoided is one: how many issues did the agent catch during document production that would otherwise have surfaced as a change order or a field conflict during construction. That number has a dollar figure attached to it already, because rework has a cost the firm already tracks for other reasons.
A second is claim exposure. A firm that can show an agent caught a code compliance gap before permit submission has a number that matters to a principal in a way that hours saved never will. A third, quieter metric is capacity: whether the practice took on a project it would have turned down before, because the agent freed up enough senior review time to make the schedule work.
The Denominator Problem
ROI calculations fail as often on the cost side as the benefit side. The subscription fee is the easy part to count. What gets left out is the oversight time: someone senior still has to review what the agent flags, correct what it misreads, and retrain it when a workflow changes. A firm that only counts the license fee against the benefit and ignores the review burden will always compute a rosier number than the one it's actually living with.
Include the setup and integration cost too, even after the pilot ends, since every new project type or consultant format tends to need a few hours of recalibration before the agent is reliable on it again.
Give It Longer Than One Project Type
A single project doesn't tell a firm much, because the agent's value shows up differently at different phases. During design development it might catch inconsistencies between drawings and specs. During construction documents it might catch code compliance gaps. Measuring only one phase and generalizing from it is how firms end up either overselling or dismissing a tool based on a partial picture.
When the Math Doesn't Work
There are practices where the honest answer is that the numbers don't support the cost, at least not yet. A two-person residential practice doing highly custom work on three projects a year doesn't generate enough repetitive volume to amortize the setup time, and the agent will look like a net cost no matter how it's measured. That isn't a failure of the tool. It's a mismatch between the workload and what pattern-matching software is good at.
The firms where the math works tend to run several concurrent projects with enough repeated document types, specs, submittals, drawing sets, that the agent sees the same category of problem often enough to get good at catching it. Below that volume, the setup and oversight cost outweighs whatever it prevents, and no amount of dashboard reporting changes that arithmetic.
The practices that get the clearest answer are the ones that stop asking whether the agent saved time and start asking whether it changed what the firm was able to take on, catch, or avoid. That's a harder number to produce in month one. It's also the only one that still means something in month twelve.