A practical architecture talk

Building agentic systems that can act without losing control

What agents are, how they evolved, and what production systems need

Capabilities Boundaries Evidence
Akhilesh Keshap Product Manager, Agentic Systems Pragya Keshap Architect, Agentic Systems

02

The anatomy of an agentic system

The model is one component inside a larger execution system.

GoalThe outcome to achieve
ModelInterprets and decides
ContextInstructions and evidence
ToolsData and actions
StateWork completed so far
LoopContinue, correct, or stop
ControlsRules and approvals
UnderstandPlanActObserveContinue or stop

A fixed workflow follows a path written in advance. An agentic workflow can choose the next step after observing the result.

05

An agentic system in action

A missing delivery case shows how the system moves from a request to a controlled action.

“My order was supposed to arrive six days ago. Can you check what happened and help me resolve it?”
  1. 1Understand the goalResolve the delivery problem
  2. 2Find the orderRetrieve order and shipment
  3. 3Inspect trackingNo movement for six days
  4. 4Check incidentsCarrier disruption found
  5. 5Apply policyReplacement allowed
  6. 6RecommendOffer a replacement
  7. 7Check authorityRequest approval if required
  8. 8Execute and explainComplete the action
“Your shipment has not moved for six days, and the carrier has reported a disruption. This order is eligible for a replacement. I can arrange that now, subject to the required approval.”

06

The delivery run uses every system component

The final response depends on the work performed across the complete system.

Goal

Resolve the delivery problem rather than answer a tracking question.

Model

Understands the intent and selects the delivery investigation capability.

Context

Provides the conversation, order history, and applicable delivery policy.

Tools

Access order, shipment, carrier, incident, and eligibility systems.

State

Records the order, six-day delay, incident check, and collected evidence.

Execution loop

Reviews each result and decides whether more evidence is needed.

Controls

Block unsupported refunds and require approval for consequential actions.

The valuable work happens before the final sentence: evidence retrieval, policy application, state management, and authority checks.

This is manageable with five distinct capabilities. What changes when the agent can see fifty overlapping tools?

03

Agentic systems exist on a spectrum

A system can investigate independently while keeping consequential actions under human control.

1

Assistant

Finds information and answers questions.

Human interprets the answer
2

Copilot

Investigates the case and recommends an action.

Human makes the decision
3

Approval agent

Prepares an action and supporting evidence.

Human approves or rejects
4

Bounded agent

Executes approved actions within policy limits.

Human handles exceptions

Autonomy has several controls: tool selection, sequencing, execution authority, stopping conditions, and approval requirements.

04

Five stages shaped modern agentic systems

Each stage increased what the system could access, decide, or execute.

  1. Generate and retrieve

    Models produced text and used retrieved documents for grounded answers.

  2. Reason and act

    ReAct showed how models could interleave reasoning with actions.

  3. Call functions

    Structured function calling connected models to application logic.

  4. Connect systems

    MCP standardized connections to tools and enterprise data.

  5. Govern execution

    Tracing, approvals, evaluation, and sandboxing became production concerns.

Sources: ReAct paper, OpenAI function calling announcement, Anthropic MCP announcement, and OpenAI agent documentation

05

Organizations increased autonomy with the risk of the action

Adoption usually began with information and advanced toward controlled execution.

Answer

Knowledge assistance

Search, summaries, policy questions, and drafted responses.

The human acts
Recommend

Decision support

Case findings, investigation summaries, and recommended actions.

The human decides
Prepare

Approval workflows

Completed forms, prepared refunds, and proposed account changes.

The human approves
Execute

Bounded automation

Routine and reversible actions performed within policy limits.

The human handles exceptions

Confidence grows through reliable evidence, explicit controls, and repeatable evaluation.

06

The problem: more connections create more decisions

Direct exposure makes the model choose among overlapping tools and rebuild repeatable workflows.

Early prototype

Three distinct tools

  • Search customer
  • Find order
  • Search policy
Production system

Overlapping tools and systems

  • Customer tools
  • Order tools
  • Payment tools
  • Shipping tools
  • Policy systems
  • Databases
  • MCP servers
  • Specialist agents
Which tool?Which order?Which evidence?When to stop?

The production challenge shifted from connecting capabilities to controlling decisions.

07

Direct tool exposure creates production costs

The impact extends beyond choosing the wrong tool.

01

Routing ambiguity

Similar tools make selection and argument choice less reliable.

02

Latency and failures

Sequential calls add waiting, retries, timeouts, and dependency failures.

03

Token and context cost

Tool schemas and raw results consume context that the model needs for evidence.

04

Inconsistent execution

The same request can follow a different sequence and produce a different outcome.

05

Evaluation and debugging

More possible paths make failures harder to reproduce, trace, and isolate.

06

Security and authority

The model may see sensitive data or action tools outside its responsibility.

Tool count alone does not create the problem. Risk grows when tools overlap, workflows repeat, and low-level implementation reaches the main model.

08

One delivery question creates nine tool decisions

Problem in practice: direct exposure turns a familiar customer request into a tool-planning task.

“Why has my order not arrived, and what can you do about it?”
get_customer()get_orders()get_shipment() get_carrier_events()get_support_history()get_active_incidents() search_policy()check_refund_eligibility()check_replacement_eligibility()
  1. 01Which tool applies?
  2. 02What arguments should it receive?
  3. 03What order should the calls follow?
  4. 04Which results matter?
  5. 05Does the model need another call?
  6. 06When should the investigation stop?

The tools are individually useful. The burden comes from rebuilding the same investigation through model choice on every case.

09

The model rebuilds the same workflow on every case

Production impact: reconstructing a valid investigation repeatedly introduces avoidable variation.

  1. 1Find the customerget_customer()
  2. 2Locate the orderget_orders()
  3. 3Find the shipmentget_shipment()
  4. 4Inspect movementget_carrier_events()
  5. 5Check disruptionget_active_incidents()
  6. 6Evaluate resolutioneligibility tools
  7. 7Retrieve policysearch_policy()
  8. 8Decide to stopenough evidence?
Similar tool selectedVerification skippedCalls repeatedResults conflictStopped too early

Every choice repeats: tool, arguments, sequence, relevant evidence, and stopping point.

10

The solution: domain tool abstraction

Keep the underlying connectivity. Reduce the decisions exposed to the main model.

Direct exposure

The model orchestrates every call

Main model
APIMCPDatabaseSearchPolicyWorkflow

Model owns routing, ordering, result selection, and stopping.

Domain abstraction

The model selects one business capability

Main model
investigate_delivery_issue()
Internal orchestrationAPIs, MCP tools, databases, rules, and bounded model tasks

Software owns the repeatable sequence and returns structured evidence.

Connectivity remainsAPIs, MCP tools, databases, and rules still perform the work.
Visibility narrowsThe model sees bounded capabilities aligned to business intent.
Authority stays controlledThe system continues to enforce permission and approval boundaries.

11

A delivery investigation starts with one bounded request

The contract defines what the capability may read, decide, and return.

“My order was supposed to arrive six days ago. What happened, and what can you do?”
Main model selects the capabilityinvestigate_delivery_issue(
  customer_id="C-1842",
  order_id="O-7831"
)
Purpose
Diagnose the delivery problem and identify allowed resolutions.
Inputs
Customer and order identifiers.
Output
Structured findings, eligibility, and evidence references.
Authority
Read-only investigation. It cannot issue a refund or replacement.
Failure behavior
Returns missing evidence and system errors explicitly.

The model identifies the intent and supplies the identifiers. The capability owns the investigation.

12

The capability runs the standard investigation

The internal sequence stays consistent while the evidence changes from case to case.

Internal stepSystemExample result
Retrieve orderOrder APIOrder found, shipment S-102
Retrieve shipmentShipment APIStill in transit
Inspect carrier eventsCarrier APINo movement for six days
Check incidentsIncident serviceDisruption I-74 affects the route
Evaluate resolutionsRules engineReplacement allowed, refund unavailable
Retrieve policyPolicy searchPolicy 5.2 supports the result
Assemble evidenceOrchestration layerOne normalized response

The capability handles retries, timeouts, and partial evidence. It performs no write action.

13

The model receives evidence and allowed options

The capability compresses the investigation into a result the model can explain.

{
  "issue": "delivery_delay",
  "delay_days": 6,
  "shipment_status": "stuck_in_transit",
  "active_incident": true,
  "refund_eligible": false,
  "replacement_eligible": true,
  "recommended_action": "offer_replacement",
  "evidence_ids": [
    "shipment-102", "incident-74", "policy-5.2"
  ]
}
Main model

Conversation and judgment

  • Explain what happened
  • Connect the evidence to the customer’s question
  • Present the supported option
  • Ask a clarifying question when needed
  • Request approval before a consequential action
“Your shipment has not moved for six days, and the carrier has reported a disruption affecting the route. This order is eligible for a replacement. I can arrange that now, subject to the required approval.”

Rules determine the supported options. The model explains them and manages the conversation.

14

Nine visible tools become one domain capability

The underlying systems remain. Their routine choreography moves behind a stable contract.

Direct tool exposureDomain capability
Nine low-level tools visibleOne delivery capability visible
Model selects and sequences callsSoftware owns the standard sequence
Multiple raw responses enter contextOne structured evidence object returns
Investigation path can vary by runStandard cases follow a consistent path
Policy and eligibility need separate routingRules and policy run inside the capability
Many paths need evaluationContract and internal steps can be tested separately
Broad backend accessBounded domain responsibility

The nine underlying capabilities still exist. The main model no longer orchestrates them individually.

15

Another example: a loan officer needs a governed decision

Illustrative example. The decision uses hypothetical product rules.

Problem statement

A loan officer needs to determine whether an applicant is eligible for a loan and explain the specific reason for the decision.

Approved factors

The capability checks three requirements

  1. Identity

    Verification is complete.

  2. Income

    Verified income meets the product requirement.

  3. Debt-to-income ratio

    The ratio is within the product limit.

Loan officer asks
“Is this applicant eligible for the loan? If not, what requirement failed?”
check_loan_eligibility(
  applicant_id,
  loan_product_id
)
DecisionNot eligible
ReasonDebt-to-income ratio exceeds the product limit

The loan officer receives a decision and its reason. The officer does not need to reconstruct the calculations.

Illustrative only. Real lending requires approved underwriting policies, fair lending controls, required disclosures, and governed exceptions.

16

Approach 1: the prompt owns the workflow

The model receives every low-level tool and the instructions for applying the decision logic.

You assist loan officers with product eligibility.

For the selected applicant and loan product:

1. Call get_identity_status(applicant_id)
2. Call get_verified_income(applicant_id)
3. Call get_debt_obligations(applicant_id)
4. Retrieve the current product thresholds
5. Calculate the debt-to-income ratio
6. Apply every eligibility requirement

If any required data is missing, do not decide.

Return:
- eligible: true or false
- failed requirement
- reason for the loan officer
The model must
  1. Select four tools
  2. Sequence the calls
  3. Interpret each response
  4. Perform the calculation
  5. Apply the product rules
  6. Handle missing data
  7. Explain the decision

One prompt now owns routing, sequencing, arithmetic, policy application, error handling, and explanation.

17

Approach 2: a domain capability owns the decision

The prompt becomes shorter because versioned software owns the repeatable logic.

Prompt shown to the model 
When a loan officer asks about eligibility,
call:

check_loan_eligibility(
  applicant_id,
  loan_product_id
)

Use the returned decision, reason code,
and evaluated factors exactly.

Do not recalculate or override the result.

If the result is unable_to_determine,
request the listed missing information.

Explain the result clearly.
Inside the capability
  1. Retrieve governed inputsIdentity, verified income, debt obligations
  2. Load versioned product rulesApproved thresholds and required factors
  3. Calculate the ratioDeterministic arithmetic in code
  4. Evaluate eligibilityRules engine returns the failed requirement
  5. Assemble the resultDecision, reason code, factors, missing data

The prompt selects the capability and explains the result. Versioned software owns the decision.

18

Implementation lessons from both cases

The delivery investigation and eligibility decision use the same design discipline.

  1. 01Start with a recurring business question

    Define the outcome the user needs and the stopping condition.

  2. 02Create one bounded domain capability

    Give it a clear purpose, inputs, output, authority, and failure behavior.

  3. 03Put predictable choreography in software

    Code should own standard retrieval, calculations, joins, and policy checks.

  4. 04Return structured evidence

    Include the result, reason code, evaluated factors, missing inputs, and evidence references.

  5. 05Separate investigation from action

    Reading evidence and recommending an option should not automatically authorize a consequential action.

  6. 06Evaluate each layer independently

    Test routing, retrieval, rules, evidence assembly, explanation, and authorization.

Give the model the smallest meaningful problem that still requires judgment.

19

Industry patterns for domain expertise and orchestration

Related patterns solve different problems. The names describe where expertise and control live.

Product architecture

Domain capability

Developers expose one business-level operation backed by APIs, rules, and workflows.

Best for

Stable, repeated, policy-sensitive work such as delivery investigations and loan eligibility.

OpenAI and Claude APIs

Programmatic Tool Calling

The model writes bounded code that coordinates several tools and reduces intermediate results.

Best for

Predictable multi-call processing with loops, conditions, filtering, or aggregation.

Claude and agent platforms

Skills and specialist agents

Skills package domain instructions, scripts, and resources. Specialist agents handle delegated work.

Best for

Reusable procedures or open-ended tasks that benefit from isolated expertise.

Terminology: Advanced tool use is a broad umbrella. These two cases use domain abstraction and bounded orchestration.

Sources: OpenAI Programmatic Tool Calling, Anthropic Advanced Tool Use, Anthropic Agent Skills

10

Design question 2

Where should reasoning happen?

Based on “Designing Reasoning Boundaries in Agentic Systems”

11

The three execution layers

Each layer owns a different kind of decision and exposes a clear contract to the next one.

Ordinary software

Compute and enforce

  • Retrieve records from known systems
  • Calculate exact values
  • Apply policy and permission rules
  • Validate schemas and allowed values
Bounded inner model

Interpret evidence

  • Classify unstructured notes
  • Extract findings from messy text
  • Return typed fields with evidence references
  • Stay inside a narrow semantic task
Main model

Understand and explain

  • Interpret the user’s goal
  • Ask for missing information
  • Synthesize the supported result
  • Explain the available options
Human approval boundaryConsequential actions cross an explicit authority check before execution.

Use software when the answer can be computed. Use a model when the input must be interpreted.

12

Two bad reasoning boundaries

Both failure modes assign work to a mechanism that cannot guarantee the right behavior.

Failure mode 1

The model becomes the calculator

“Read these timestamps, calculate the delay, apply the refund threshold, and verify the approval limit.”

  • Exact work depends on prompt interpretation
  • Small wording changes can alter the result
  • Policy enforcement becomes difficult to test
Failure mode 2

Code becomes the language interpreter

if "lost" or "stuck" or "not moving" ...

  • Rules multiply as language varies
  • Ambiguous evidence falls through brittle branches
  • Maintenance grows faster than coverage

Boundary rule: Put computable work in software. Use a bounded model call for semantic interpretation.

13

A practical reasoning-boundary test

Start with the nature of the work, then assign the narrowest mechanism that can perform it reliably.

Nature of the workMechanismExample
Retrieve a known recordAPI or databaseOrder and shipment lookup
Transform or calculate an exact resultCodeDays delayed
Apply a fixed policy conditionRulesReplacement eligibility
Interpret unstructured evidenceBounded LLMMeaning of a carrier note
Synthesize an open-ended responseMain LLMExplain supported options
Approve a high-impact exceptionHumanAction beyond agent authority

Treat every model output as untrusted input. Validate its schema, allowed values, evidence references, and required fields.

14

Design question 3

How do we know it worked?

Based on “How Do You Evaluate an Agent That Calls Tools That Call Other Tools?”

15

Evaluation measures how reliably the agent behaves

Agent quality includes the decisions and evidence behind the final answer.

Evaluation is a repeatable process for measuring whether an agent makes the right decisions, uses the correct evidence, follows policy, and completes the user’s task across representative cases.
Behavior varies

Similar requests can produce different tool choices and execution paths.

Failures stay hidden

A polished answer can conceal an incorrect record, rule result, or unsupported claim.

Changes create regressions

A model, prompt, tool, policy, or adapter change can shift behavior elsewhere.

Actions have consequences

Incorrect refunds, approvals, or account actions can create real operational harm.

Evaluation measures both the outcome and the path the system used to reach it.

16

Software testing and agent evaluation

Software tests verify deterministic components. Evaluation measures complete system behavior.

Software test

Does a component return the exact expected result?

last_movement = 6 daysactive_incident = true
Delay calculation
6 days
Refund eligible
false
Replacement eligible
true

One input, one exact output, pass or fail.

Agent evaluation

Does the complete agent behave correctly across representative cases?

“My package has not arrived.”

“Tracking has not changed for a week.”

“Can I get a refund for this delay?”

  • Select the delivery capability
  • Use the correct shipment and policy result
  • Avoid an unsupported refund
  • Explain the supported resolution

Many cases, layered checks, success rates, and failure categories.

Software testing remains part of evaluation. It covers the layers where the expected answer is exact.

17

A helpful answer can still be wrong

The final response may sound credible even when it contradicts the system’s evidence.

Customer request
“My order has not arrived. What can you do?”
Agent response
“Your shipment has been delayed by a carrier disruption. I have confirmed that you are eligible for a refund.”
Tool result 
{
  "delay_days": 6,
  "active_incident": true,
  "refund_eligible": false,
  "replacement_eligible": true,
  "recommended_action": "offer_replacement"
}

Conflict: The response promises a refund that the eligibility result did not allow.

A final-answer score may reward tone and helpfulness while missing the unsupported action.

18

One delivery run creates five tests

Test the system at the same seams where it executes.

  1. 1Routing

    Did the model select investigate_delivery_issue?

  2. 2Evidence

    Did the capability retrieve order_42 and shipment-102?

  3. 3Rules

    Did it calculate six delayed days and reject refund eligibility?

  4. 4Inner model

    Did the carrier-note category match the source and allowed schema?

  5. 5Final response

    Did it offer a replacement without inventing a refund?

The final response matters. The layer checks explain why that response succeeded or failed.

19

One golden case defines correct behavior

A labeled case makes the expected path and outcome explicit before release.

Input and fixed context
“My order has not arrived. What can you do?”
Customer
cust_1
Order
order_42
Shipment
shipment-102
Carrier incident
incident-74
Expected execution
Selected capability
investigate_delivery_issue
Delay
6 days
Refund eligible
false
Replacement eligible
true
Recommended action
offer_replacement
Evidence
shipment-102, incident-74, policy-5.2
Expected response behaviorExplain the disruption, offer a replacement, avoid a refund promise, and use only returned evidence.

20

Match the evaluator to the decision

The cheapest reliable evaluator depends on whether the answer is exact, semantic, or consequential.

Code

Known answers

  • Tool name and arguments
  • Shipment and evidence IDs
  • Calculations and policy outcomes
  • Schema and allowed values

Run on every relevant case.

Bounded LLM judge

Semantic questions

  • Summary preserves source meaning
  • Explanation contains no distortion
  • Important qualifications remain

Use a narrow rubric and validate the judge.

Human review

Costly or uncertain cases

  • Disputed refunds
  • Policy exceptions
  • Irreversible actions
  • Evaluator disagreement

Focus review where mistakes have the highest cost.

Model judges add cost and uncertainty. Keep exact checks in code.

21

The same bad answer can have different causes

Layer scores tell the team where to investigate instead of treating every failure as a prompt problem.

Routing failure

Wrong capability

The model selected generic search instead of the delivery investigation.

Investigate

Tool boundaries, descriptions, and labeled routing cases

Evidence failure

Stale shipment

The correct capability ran but retrieved an old or incomplete shipment record.

Investigate

Adapter, lookup parameters, freshness, and dependency health

Grounding failure

Unsupported refund

The tool returned the correct result but the response contradicted it.

Investigate

Claim validation, response constraints, and model behavior

An overall score gives a grade. Layered evaluation gives a diagnosis.

22

The trace turns a result into an explainable execution

Each run should preserve the decisions, inputs, outputs, and evidence used along the way.

Request and routing

User goal, selected capability, and routing decision

Inputs and evidence

Records retrieved, source status, and evidence identifiers

Inner model call

Model and prompt version, tokens, latency, retries, and typed findings

Rules and fallback

Deterministic results, schema validation, missing sources, and fallback path

Action and outcome

Recommendation, approval, execution result, and final explanation

Prompt and model versionTokens, latency, and retriesSchema and evidence validationFallback and final outcome

23

Offline evaluation and production monitoring

The two environments answer different questions and use different evidence.

Before deployment

Is the system correct on labeled cases?

  • Expected tool, evidence, rules, and allowed claims
  • Release gates for every layer and the complete task
  • Run after model, prompt, tool, policy, or adapter changes
  • Segment by intent, carrier, language, and edge case
In production

Where may the live system be degrading?

  • Dependency and schema failures
  • Missing, stale, or incomplete evidence
  • Policy invariant violations
  • Latency, cost, fallback, and escalation
  • Sampled human or model review for correctness

Production traces rarely contain ground truth. Report the reviewed sample size beside any estimated quality score.

24

A dashboard should identify the broken layer

Illustrative production signals show why a layer breakdown is more useful than one response-quality score.

SignalTuesdayWednesday
Shipment adapter failures0.4%18.6%
Incomplete evidence1.9%22.4%
Human escalation3.1%11.2%
Grounded responses, sampled96.1%88.2%
Policy invariant violations0.1%0.1%
Diagnosis

The failure begins at the shipment adapter

  1. Adapter failures rise first.
  2. Incomplete evidence and escalations follow.
  3. Policy violations remain stable.
  4. Groundedness falls as the evidence degrades.

Investigate the adapter before changing the prompt or policy rules.

Illustrative data adapted from “How Do You Evaluate an Agent That Calls Tools That Call Other Tools?”

25

Evaluation operating model

Evaluation becomes useful when it runs before release and learns from production failures.

  1. 1Build the dataset

    Label common paths and the edge cases that matter.

  2. 2Define expected behavior

    Specify the tool, evidence, rules, and allowed claims.

  3. 3Automate exact checks

    Use code for deterministic facts and invariants.

  4. 4Judge semantic work

    Use narrow rubrics for summaries and explanations.

  5. 5Capture the trace

    Preserve inputs, decisions, evidence, and outcomes.

  6. 6Gate the release

    Set thresholds for each layer and the full task.

  7. 7Monitor production

    Track health signals and review a labeled sample.

  8. 8Learn from failures

    Add discovered cases back into the evaluation set.

The final-answer score shows whether the user received a useful result. The layer scores explain why that result changed.

18

Articles and references

The talk develops the argument across three related articles.

  1. Part 1Your AI Agent Should Not Be Talking to All MCP Tools

    Tool visibility and domain abstraction

  2. Part 2Designing Reasoning Boundaries in Agentic Systems

    Software, model reasoning, and human approval

  3. Part 3How Do You Evaluate an Agent That Calls Tools That Call Other Tools?

    Layered evaluation and execution traces

Additional references: ReAct, OpenAI function calling, Anthropic MCP, and OpenAI agent documentation.

The production discipline

Reliable agents have deliberate boundaries

Control the decisions

Give the model domain capabilities and reserve model reasoning for tasks that require interpretation.

Preserve the evidence

Trace and evaluate every layer before trusting the final answer or action.

The goal is the right autonomy, at the right layer, with enough evidence to trust the outcome.

38

About the speakers

Product and engineering perspectives on building agentic systems.

Speaker 1

Akhilesh Keshap

Senior Product Manager at eBay

  • Product leader working on AI and agentic systems
  • Software engineering background with more than a decade across technology and product
  • Writes about tool abstraction, reasoning boundaries, and agent evaluation
linkedin.com/in/akhileshkeshap
Speaker 2

Pragya Keshap

Software engineer and cloud/AI practitioner

  • Engineering experience across backend systems, Java, APIs, and enterprise architecture
  • Explores practical approaches for building and scaling AI agents
  • Writes about cloud architecture, Docker, and production AI systems
linkedin.com/in/pragyakeshap

We approach agentic systems from both sides of delivery: defining the product behavior and engineering the system that supports it.