How to Evaluate a Business Agent Before Production

Start with one workflow#

The fastest way to evaluate a business agent is not to ask whether it can automate an entire department. Start with one real workflow that already has an owner, a queue, a record, or a decision.

Good candidates include:

• triaging a waiting customer conversation;
• preparing an internal reply draft;
• answering a policy or product question from approved sources;
• routing a request to the right operator;
• proposing a reversible update while keeping the final commit behind an approval boundary.

The workflow should be narrow enough to inspect and important enough that someone cares about the outcome.

The five-question evaluation#

1. What does the agent actually receive?#

Write down the input, not the aspiration. Is it a conversation, a customer record, a document, a ticket, or a combination of sources?

The evaluation should make tenant, customer, workspace, and source scope visible. If the agent receives the wrong context, a fluent response can still be the wrong operational response.

2. What may it do?#

Separate capabilities into four groups:

• read-only;
• propose or draft;
• reversible action;
• customer-visible or irreversible action.

Then map each group to the tools and permissions available for the workflow. A production evaluation should never rely on a vague statement that the agent is “safe.” It should show which tools are available and which are unavailable.

3. Where does human authority apply?#

Some work can proceed automatically. Other work needs a person because it changes customer expectations, creates material risk, or depends on an ambiguous policy.

Define the exact pause point. “Human in the loop” is only useful when the operator knows what decision they are making, what evidence the agent used, and what will happen after approval.

4. What proves the outcome?#

Decide what receipt or durable state proves success. A chat message saying “done” is not enough.

Useful evidence can include:

• the scoped run or action identifier;
• the source evidence used;
• the approval decision;
• the backend record changed;
• the outbound effect count;
• the recovery or retry state.

The proof should also show what did not happen. For a draft-only workflow, a receipt that records outbound_send_count: 0 is meaningful evidence.

5. What happens when something fails?#

Test the boring cases before celebrating the happy path:

• the source is missing or conflicts with another source;
• the tool is not permitted;
• the request is retried with the same idempotency key;
• the operator rejects the proposal;
• the downstream system times out;
• the workflow must be replayed or recovered.

An agent is not production-ready because it succeeded once. It is closer to ready when failure states are explicit and recoverable.

What to record after the evaluation#

Create a short evaluation brief with:

This brief is more useful than a generic demo score because it tells the team what they actually evaluated.

Mirai’s developer documentation and safety and recovery guide use this staged, evidence-first posture for Atlas evaluation. The first-request guide is the next step when the workflow has a real owner and a clear boundary.

The production question#

The final question is not “Did the model sound intelligent?”

It is:

Can this workflow move forward inside explicit context, tool, authority, receipt, and recovery boundaries?

If the answer is not yet yes, the evaluation has still succeeded. It has shown exactly what must be fixed before production.