Vishal Shelar
AI systems & architecture

How Much Should You Trust an AI Agent?

·8 min read

A robot balances on a scale between "too much trust" (a shield, high risk, big consequences) and "too little trust" (a lock, low value, no autonomy).

As we build more AI agents, one question keeps coming up:

How much should we actually allow an AI agent to do?

The obvious answer is to add guardrails.

But I think that answer is incomplete.

Put too few controls around an agent and you create risk. Put too many controls around it and you remove the very thing that makes agents useful: autonomy.

So maybe the better question isn't:

"Should we trust AI agents?"

It's:

"How much should we trust an agent for this particular action?"

Not Every Agent Action Has the Same Risk

Consider a customer-facing AI agent.

A customer might ask:

"What are your store hours?"

There's very little consequence if the agent needs to retrieve that information.

Now consider:

"What's the status of my order?"

We need authentication and authorization because we're accessing customer information, but it's still primarily a read operation.

Then:

"Change my delivery address."

Now the agent is changing data.

And finally:

"Cancel my order and refund $5,000."

We're in a completely different risk category.

Yet we sometimes design agent architectures as though every tool call should be treated the same.

I don't think they should be.

The autonomy we give an agent should be proportional to the consequence of the agent being wrong.

Risk Isn't Just About the Tool

Initially, it seems easy to assign a risk level to every tool.

getStoreHours()       → Low Risk
getOrderStatus()      → Low Risk
updateAddress()       → Medium Risk
cancelOrder()         → High Risk
issueRefund()         → High Risk

But there's a problem.

Imagine updateAddress() is used to change one customer's delivery address.

Now imagine the same tool is somehow being used to change the addresses of 2,000 customers.

Same tool.

Very different risk.

Even reading information isn't automatically low-risk. getOrderStatus() might be harmless when a customer accesses their own order but a serious security issue when someone tries to access another customer's information.

So risk cannot come entirely from the tool.

It is a combination of things:

Action
   +
User Identity
   +
Data Sensitivity
   +
Transaction Value
   +
Scale
   +
Context
   ↓
Risk

And this is where AI itself can become useful.

Could an LLM Evaluate Risk?

Traditional policy engines are very good at deterministic questions:

  • Is this user authenticated?
  • Does this user have permission?
  • Is the refund greater than $500?
  • Is this data classified as sensitive?
  • Is this operation allowed?

But they aren't necessarily good at understanding context.

An LLM can look at the conversation and recognize that something unusual is happening.

For example:

"Update my email address."

and:

"Update the email addresses for these 2,000 accounts."

may ultimately involve similar capabilities, but the context and potential impact are dramatically different.

So an LLM could act as a contextual risk evaluator.

User Request
     ↓
AI Agent
     ↓
Proposed Action
     ↓
LLM Risk Evaluation
     ↓
LOW | MEDIUM | HIGH

But this creates another problem.

Do we really want an LLM to decide whether another LLM is allowed to perform a dangerous operation?

I don't think we should trust it completely.

Let AI Interpret Risk. Let Software Enforce the Boundaries.

LLMs are probabilistic.

They can misunderstand context. They can be manipulated. They can confidently produce an incorrect conclusion.

So while an LLM can help identify contextual risk, deterministic controls should still define the boundaries that cannot be crossed.

Imagine the organization has this policy:

Refund ≤ $500
Agent may process automatically

Refund > $500
Human approval required

The agent—or even an LLM risk evaluator—shouldn't be able to decide:

"This $2,000 refund looks legitimate, so I'll classify it as low risk."

The deterministic policy still wins.

A useful rule is:

AI can raise the risk level, but it shouldn't be able to lower a risk level enforced by deterministic policy.

For example:

Policy Engine → LOW
LLM Evaluator → HIGH
Final Risk → HIGH

Policy Engine → HIGH
LLM Evaluator → LOW
Final Risk → HIGH

When the two disagree, choose the safer boundary.

That disagreement should probably also become an observability signal. If it happens frequently, something in the model, policy, or risk classification may need attention.

Confirmation Is Not Authorization

Another tempting solution is simply asking the user to confirm important actions.

"You're about to issue a $5,000 refund. Do you want to continue?"

Confirmation is useful.

But confirmation answers only one question:

"Is this what the user intended?"

It doesn't answer:

"Is the user allowed to do this?"

And it certainly doesn't answer:

"Is this operation valid according to our business rules?"

Those are different responsibilities.

User Confirmation
"Is this what you intended?"

Authorization
"Are you allowed to do this?"

Risk Evaluation
"How dangerous is it if this goes wrong?"

Business Validation
"Is this operation valid?"

Execution
"Perform the operation."

Conflating these responsibilities is where agent architectures can become dangerous.

Don't Put Your Business Rules in the Agent

Consider our refund example again.

We could teach the agent:

Refunds under $500 are allowed automatically unless there has already been a previous refund, except under certain circumstances...

That logic will quickly become complicated.

Worse, we're asking a probabilistic system to enforce deterministic business rules.

Instead, give the agent a business capability:

issueRefund(orderId, amount)

The agent understands that the customer wants a refund.

The refund service determines whether the refund can actually happen.

It can check:

  • Original transaction amount
  • Previous refunds
  • Remaining refundable balance
  • Customer/account permissions
  • Duplicate requests
  • Approval thresholds
  • Other business rules

This leads to an architectural principle I think will become increasingly important with AI agents:

Don't expose primitive system operations when you can expose well-defined business capabilities.

The agent decides what needs to happen.

The business system determines how it is allowed to happen.

A Risk-Based Agent Architecture

Putting these ideas together gives us something like this:

Risk-based agent architecture: the user's request goes to the AI agent, which understands intent and produces a proposed action; risk evaluation combines LLM context with policy rules to classify the action as low, medium, or high risk; medium risk gets additional validation and high risk requires approval or escalation; the request then reaches the business API, which enforces business rules, authorization, and validation before reaching the system of record.

The important part isn't the exact boxes.

It's the separation of responsibility.

The agent understands intent.

AI helps interpret contextual risk.

Deterministic policies establish hard boundaries.

Business systems enforce business rules.

But There's Another Failure Mode

It's easy to make an agent safe by making everything high-risk.

Require approval for every update.

Confirm every action.

Block anything unusual.

Send everything questionable to a human.

Eventually you've built an AI agent that nobody wants to use.

A safe agent that blocks everything isn't a useful agent.

So the real engineering problem isn't maximizing restrictions.

It's calibrating autonomy.

Too Permissive
      ↓
Dangerous

Too Restrictive
      ↓
Useless

Well Calibrated
      ↓
Useful + Controlled

That calibration will probably require continuous work.

Better models.

Better prompts and context.

Better evaluations.

Better policies.

Real-world feedback.

And potentially fine-tuning where it makes sense.

As agents interact with more systems and perform more consequential actions, understanding where they make good and bad decisions will become just as important as measuring whether they completed the task.

The Goal Isn't Maximum Control

We're used to software where the developer defines the path.

IF X
THEN Y
ELSE Z

Agents change that relationship.

We're increasingly allowing software to determine the next action dynamically.

That doesn't mean abandoning deterministic systems.

It means being deliberate about where probabilistic reasoning ends and deterministic control begins.

For low-risk actions, give the agent room to operate.

For higher-risk actions, increase verification.

For critical actions, enforce deterministic boundaries and human approval where necessary.

The goal shouldn't be to remove autonomy from AI agents.

The goal should be to give them as much autonomy as the risk allows.

Because the real question isn't:

"Do we trust the AI agent?"

It's:

"What happens if the agent is wrong?"

The answer to that question should determine how much freedom we give it.