Guardian Agents Who Know Your Goals: Defining AI Agent Success with Custom Metrics

How Pre-Defined Tags and Report Metrics in Wayfound let your guardian agent (and you) measure what actually matters to your business and in the plain language your team understands.

Every team measuring an AI agent runs into the same wall: a dashboard or monitoring report can show you counts and volumes (sessions, messages, response times) but not success the way your business sees it. Did the agent actually resolve your issue? Did it deflect the ticket that would have otherwise landed in a human's queue? Did it complete the outcome you're getting paid for?

And let's state something obvious: "success" isn't universal. A deflection for a support team, a qualified lead for a sales team, and a completed booking for a travel agent are wildly different things — and no pre-built metric can know which one matters to you.

So we stopped trying to guess and put a fix to this challenge into Wayfound. With Pre-Defined Tags and Report Metrics, you define an agent's success in your own words, and Wayfound's guardian agent applies and tracks it for you over time. As Wayfound supervises your agent or agents, it automatically determines which tag to apply to each agent session, based on the rules you define for each tag, and reports on that for you over time.

Here's how:

Step 1: Describe what you care about, in plain English

It starts by looking at the settings for each agent Wayfound is supervising, found under Pre-Defined Tags. A tag is just a name and a natural-language description of when it should apply. No rules engine, no regex, no code.

EXAMPLE = For a customer support agent, you might create:

• RESOLVED — "Apply this tag when the agent fully answers the user's question and the user did not need to be handed off to a human."
• ESCALATED — "Apply this tag when the conversation was handed off to a human agent or the user asked to speak to a person."

That's the entire configuration in Wayfound. Behind the scenes, every new session your agent handles is read by our LLM and scored against each tag's description. When a conversation clearly matches, the tag gets applied by Wayfound to that session. When it doesn't, it doesn't; the model is instructed to only tag conversations that clearly match, so you're not chasing false positives.

Because the definition is natural language, you can encode nuance that a keyword filter never could: "the user did not need to be handed off to a human" captures intent, not just words on the screen. And you can refine it anytime by tightening the wording, adding a tag, deactivating one and it applies those refinements to sessions going forward.

Step 2: Turn tags into a metric

Tags label individual conversations. Now, you want to turn them into a number you and the Wayfound guardian agent can watch and report on. You will add a Metric to a Report.

On a Report, click Add Metric and give it a name and a simple formula. The formula counts your tagged sessions against a built-in total_sessions value:

• Name: Resolution Rate
• Expression: RESOLVED / total_sessions * 100
• Format: Percent · Direction: Higher is better ▲

As you type the expression, Wayfound suggests your available tags and shows a live preview of how it's being read, so you don't need to memorize anything. The math itself is deterministic, and the LLM's job is the judgment call on each conversation; the metric is just arithmetic over those results. That split matters: your reported numbers are stable and auditable, even though the underlying classification is AI-powered.

The result shows up on your Report as a clean tile — say, 47.3% — with a period-over-period delta (▲ +3.2 pts vs prev) colored green or red based on whether the trend is going the direction you want.

The flagship example: Deflection rate

Deflection is the share of conversations your agent handled without needing a human and is the metric most support leaders are asked about. It's exactly what this feature in Wayfound was built for.

You define a few tags in natural language to describe the possible outcomes of a conversation:

• DEF-SUCCESS — the agent resolved it end to end.
• DEF-ESCALATION — it had to go to a human.
• DEF-NONE — no real support intent (spam, testing, a hello-and-goodbye) that shouldn't count either way.

Then your deflection metric excludes the noise and reports the true rate:

DEF-SUCCESS / (total_sessions - DEF-NONE) * 100

Now "deflection rate" isn't a number someone estimated in a spreadsheet once a quarter. It's a live, defensible metric that updates as your agent handles real conversations, with a definition your whole team agreed on and can see.

Step 3: Watch it move over time

A single number is a snapshot; the value is in the trend. Every Wayfound Report has a Trends tab that charts each metric over time, with a toggle between Per run and Monthly views.

Monthly is the one leaders tend to live in: it buckets your sessions by calendar month and recomputes the metric for each one (30 days, 90 days, a year, or all time). Because it's computed directly from the underlying conversations, a month shows up even if no report happened to run that week, and the current (in-progress) month is clearly marked "so far." You get a clean line: is resolution rate climbing as you improve the agent's knowledge base? Did deflection dip the month you shipped a change? The answer is right there.

A Guardian Agent that knows what matters: outcome-based value

Once you can define success in natural language and count it, a lot of business questions become answerable with the same two steps, including how much value your agent is actually delivering.

Example Use Case: If you're moving toward outcome-based pricing — i.e. charging per resolved issue, per completed booking, per qualified lead rather than per seat — you need a trustworthy count of the outcome you're billing on. The same mechanics apply: define a tag that describes the billable outcome in plain language (for example, a RESOLVED or BOOKING-COMPLETED tag), then build a Report metric that counts those sessions over the month.

This one you compose yourself rather than pull off a shelf, but that's the point. The outcome you charge for is specific to your business, so the definition should be yours. Wayfound gives you the primitives of natural-language judgment plus deterministic counting to measure it honestly and watch it every month.

Off-the-shelf analytics tell you how busy your agent is. Pre-Defined Tags and Report Metrics in Wayfound tell you whether any given agent is succeeding — by your definition of success, not ours. You now have a measure from your whole team can see and trust, and it's all automatically handled by Wayfound.

Ready to define your first metric? Open any agent's Pre-Defined Tags, then add a Metric to a Report.