Measuring an AI agent on your website: metrics that matter

TL;DR

• Measure an AI agent the way you would measure a new hire: by what it gets done, not by how much it talks.
• Track four layers: usage, answer quality, outcomes such as resolved questions, leads and meetings, and business impact measured against a holdout group that never sees the agent.
• Read vendor definitions closely. Some count a visitor who leaves without a word as a "resolution."

Start with the job, not the dashboard

Many AI agent dashboards open with the number that is easiest to count: conversations. It is also the number least connected to whether the agent is worth paying for.

A better starting point is the HEART framework from Google researchers Kerry Rodden, Hilary Hutchinson and Xin Fu, presented at CHI 2010 (Google Research). HEART stands for Happiness, Engagement, Adoption, Retention and Task success, and it comes with a simple process: write down the goal, identify the signals that would show progress toward it, and only then choose metrics. Applied to a website agent:

Skipping this step is expensive. In June 2025, Gartner predicted that over 40% of agentic AI projects will be canceled by the end of 2027, citing rising costs, unclear business value and weak risk controls (Gartner). Business value you never defined is business value you cannot show.

The four layers of measurement

Cost runs across all four: cost per conversation means little until you can divide by outcomes.

Record a baseline before launch

Every metric above is easier to read with a "before" number next to it. In the four weeks before the agent goes live, write down:

• Support emails and tickets per week, and the top ten topics
• Demo requests, trial sign-ups or quote requests per week
• Your site-wide conversion rate for those actions
• The most common site search queries, and how many return nothing
• How long visitors wait for a first reply from your team

You will not get a perfect comparison, because seasons, campaigns and product changes all move these numbers. But a baseline turns "the agent is busy" into "support email on pricing questions fell from 40 a week to 25," which is a sentence a budget holder can act on.

Layer 1: usage

Engagement rate is the share of sessions where the agent was available and the visitor sent at least one message. It tells you whether visitors notice and trust the widget. Watch it, but don't target it. High usage can also mean your site is hard to navigate.

The most valuable usage data is qualitative: what visitors ask. Group conversations by topic every month. A rising topic is a signal about your product, your pricing page or your documentation, often before your sales or support team notices it.

Layer 2: answer quality

Supported answer rate

Sample conversations every week and grade each substantive answer: supported by the cited page, partly supported, unsupported, or correctly declined. The share of supported answers is the closest thing you have to an accuracy score, and unsupported answers, including any hallucination, are the ones to trace to a cause. Our guide to grounded answers explains the grading in detail.

Unanswered rate

How often the agent says it couldn't find an answer. A high rate is not a failure of the agent; it is a list of pages you haven't written yet. Track it by topic, and expect it to fall as you fill the gaps.

Handoff rate

How often conversations move to a person through human handoff. This is not a "lower is better" metric. A very low rate can mean the agent hangs on to conversations it should pass on; a very high one can mean it can't answer common questions. Record the reason for every handoff: visitor asked, agent couldn't answer, high-stakes topic, or buying intent.

Feedback buttons

Thumbs up and down on answers are useful pointers to what to review first. They are a poor score: few visitors click, and those who do are not typical.

Layer 3: outcomes

Resolution, defined strictly

"Resolution rate" is a favorite vendor metric, and its definition varies from vendor to vendor. Intercom, for example, documents that a resolution for its Fin agent is counted when, after Fin's last answer, the customer "either confirms the answer was satisfactory (confirmed resolution), or exits the conversation without requesting further assistance (assumed resolution)," and it bills $0.99 per outcome (Intercom). Intercom also deducts a resolution if the customer returns to the same conversation for more help.

That is a reasonable billing rule, and it is documented openly. But an assumed resolution counts silence as success, and a visitor who gave up looks the same as one who got the answer. For your own reporting, use a stricter definition, such as:

Resolved = the visitor confirmed the answer helped
           OR (the visitor left after an answer
               AND made no contact about the same topic
               through any channel within 7 days)

Report confirmed and assumed resolutions separately. If most of your resolutions are assumed, you know less than the headline number suggests.

Leads, qualification and meetings

For a sales-focused agent, track the funnel step by step: contacts captured, leads that pass lead qualification, meetings booked, meetings held, and pipeline created. Google Analytics 4 has recommended events for exactly these steps: generate_lead, qualify_lead, working_lead and close_convert_lead, and generate_lead accepts a lead_source parameter (GA4 events reference). Tag leads from the agent with their own source so you can follow them through to revenue.

// When the agent captures a contact
gtag("event", "generate_lead", {
  currency: "USD",
  value: 50, // your own estimate of what a lead is worth
  lead_source: "website_ai_agent",
});

// A custom event when a visitor asks for a person
gtag("event", "agent_handoff", { reason: "asked_for_human" });

Conversion rate, with a warning

It is tempting to compare the conversion rate of visitors who chatted with that of visitors who didn't. The chatters will usually convert better, and that proves little: people who start a conversation were more interested to begin with. That is why layer 4 exists.

Layer 4: impact, measured with a holdout

To know what the agent changed, you need visitors who were comparable in every way except that they never saw it. The standard method is a holdout: hide the agent from a random share of visitors, for example 10% to 20%, for a fixed period, then compare the two groups.

Here is an illustrative example with made-up numbers:

Group           Sessions   Demo requests   Rate
Agent shown       40,000             520   1.30%
Holdout           10,000             110   1.10%
Difference                                 +0.20 points (+18% relative)

An 18% lift looks impressive. But with these volumes, a standard two-proportion significance test gives a p-value of about 0.11, short of the usual 95% confidence threshold. The honest conclusion is "promising, keep the test running," not "the agent lifted demos by 18%." Small websites may need to run a holdout for several months, or accept that they can only measure the larger effects.

Use the same holdout to check support volume: did visitors who saw the agent send fewer emails and tickets than those who didn't?

Cost metrics that mean something

• Cost per conversation: total agent cost divided by conversations. Useful for budgeting, not for judging value.
• Cost per strict resolution: compare it with what a resolved ticket costs you when a person handles it.
• Cost per qualified lead: compare it with your cost per qualified lead from paid channels.

Metrics that mislead

• Total conversations. Grows with traffic and with confusing pages alike.
• Deflection without a repeat-contact check. A visitor who leaves the chat and emails support was not deflected.
• Average conversation length. Longer can mean engaged or stuck. On its own it says neither.
• Satisfaction from a tiny share of responders. Report the response rate next to the score.
• "Hours saved" estimates. Usually a conversation count multiplied by an assumed handling time. Treat as marketing, not measurement.

Extra metrics for other kinds of agents

The four layers apply to any website agent, but each type adds a few signals of its own:

• Site search: zero-result rate, how often visitors search again with different words, and how often they open a cited page.
• Quote calculators and conversational forms: start rate, completion rate, and the share of visitors who leave contact details after seeing an estimate.
• Voice: the share of visitors who grant microphone access, time from the end of the visitor's sentence to the first sound of the reply, and how often visitors switch from voice to text.
• Agents that take actions: actions completed, actions reversed or corrected by your team, and actions that needed a confirmation step. A reversed action is the action-taking version of an unsupported answer.

How to report it

When you present results to leadership, lead with outcomes and impact, not usage. Show the holdout comparison with its uncertainty stated plainly, put cost per outcome next to the alternative it replaces, and include two or three real conversations: one that went well, one that went wrong, and what you changed because of it. That is more convincing than any chart of conversation counts.

A one-page monthly scorecard

Set targets that are specific and measurable. Anthropic's guidance on success criteria makes the same point for AI systems in general: instead of a goal like "good performance," define something specific, and "use quantitative metrics or well-defined qualitative scales" (Claude docs).

Read conversations, not just dashboards

Numbers tell you where to look; transcripts tell you what happened. Set aside an hour a week to read a sample of conversations with whoever owns the website content. Note answers that were wrong, questions that had no page behind them, and moments when a person should have stepped in. Then fix the content, which is the subject of our guide to training an AI chatbot on your website, and see whether next month's scorecard moves.

If you have not launched yet, decide your metrics first and pick the system second. The choice between a chatbot and an agent that takes actions, covered in chatbot vs AI agent, depends on which outcomes you need to move.

ASDK's Chat agent, in early access, answers visitors from your own content, cites your pages and captures contacts, so the outcomes above are what it should be judged on.