Skip to main content

Workload: Customer Service Chatbot

(Dialogue plus live account tools — with escalation and hard gates on consequential actions)

Intent​

Help a customer resolve an enquiry or case through conversation, using approved knowledge and live systems of record, while escalating to a human when uncertainty, policy, or risk requires it. Success is a resolved outcome (or a clean handoff), not a long chat transcript.

OpenAI’s handoff guidance uses customer-support specialists (order status, refunds, FAQs) as the canonical example of transferring ownership to a domain agent. (openai.github.io; developers.openai.com) Microsoft likewise positions handoff orchestration for customer support and dynamic routing. (learn.microsoft.com) Anthropic’s agent-eval work argues that agent quality should be judged by outcomes in the environment, not by transcript plausibility alone. (anthropic.com)

Typical requirements​

  • Identity of the customer (or session) must be established before account-specific actions.
  • Policy and FAQ knowledge coexist with live account/order/billing state.
  • Some actions are read-only; others change money, access, or personal data.
  • Human escalation paths exist and must preserve context.
  • Full audit trail of intent → plan → act → verify is required for disputes and compliance.

Default shape​

Customer message
  ↓
Identity / session gate
  ↓
Bounded support agent
  ├── Knowledge access (policy FAQ / wiki / RAG)
  ├── Live reads via Tool Gateway (account, order, entitlement)
  ├── Candidate reply or proposed action
  ↓
Epistemic / authority gates
  ├── Schema + policy checks
  ├── Human approval for consequential writes
  └── Escalation when uncertain or out of policy
  ↓
Reply to customer  OR  Tool Gateway write  OR  Human queue

Default control structure: one bounded agent with a Tool Gateway and explicit HITL for irreversible actions. Start simple; OpenAI recommends adding specialists only when instructions, tools, or policy isolation materially improve the workflow. (developers.openai.com) Use a staged pipeline (triage → resolve → confirm) only when those stages need different permissions or models. (anthropic.com)

Critical design decisions​

1. Knowledge vs live state​

OptionPrefer when
Curated policy wiki + RAG over help centreHow-to and policy questions
Live API readsBalances, order status, entitlements, ticket state
SQLRare; only with strict query templates and least privilege

Default: policy/docs via wiki or RAG; account truth via APIs. Never invent account state from the model’s prior. RAG improves access to document knowledge and provenance; it does not replace a system of record for transactional state. (arxiv.org) Keep retrieved policy text provenance-tagged: retrieved content is a known vector for indirect prompt injection. (genai.owasp.org; ncsc.gov.uk)

2. Actuation authority​

OptionPrefer when
Read-only toolsEarly maturity; high fraud or compliance risk
Bounded writes (e.g. password reset link, cancel within policy)Clear policy automation with validators
Open-ended writes (arbitrary refunds, data export)Almost never without human or policy engine approval

Default: read-only plus a small allowlist of policy-checked writes. Consequential actions pass a policy or human authority gate after the model proposes them. OWASP’s Excessive Agency mitigations explicitly include eliminating excessive functionality/permissions and requiring humans to approve high-impact sends/writes. (genai.owasp.org) The NCSC emphasises deterministic safeguards that constrain tool/API impact when the model is coerced. (ncsc.gov.uk)

3. Escalation (HITL)​

OptionPrefer when
Always escalate writesLow autonomy maturity
Escalate on triggers (uncertainty, high value, injection suspicion, repeated tool failure)Bounded autonomous support
Rare escalationOnly with strong evals, budgets, and monitoring

Default: escalate on uncertainty, policy miss, high-value actions, and verification failure. Persist conversation state at the handoff. Microsoft notes that human participation and durable workflow state at approval checkpoints are important when humans interrupt agent runs. (learn.microsoft.com; learn.microsoft.com)

4. Identity, PII, and audit​

OptionPrefer when
Authenticated session before account toolsAny personalised support
Redaction in logs/tracesPII-heavy channels
Full observability traceDispute and compliance needs

Default: no account tools until identity is established; Tool Gateway logs every actuation; prompts and traces redact secrets and unnecessary PII. Privilege should drop to that of the untrusted party when processing untrusted content — a design principle highlighted in NCSC guidance on prompt-injection impact reduction. (ncsc.gov.uk)

AAF review focus​

LensWhy it pressures this workload
SecurityAccount takeover, prompt injection, over-privileged tools, PII leakage (genai.owasp.org; ncsc.gov.uk)
Autonomy & Outcome GovernanceWhat the bot may do alone vs what requires human authority (genai.owasp.org)
ReliabilityDefinition of Done is case outcome / correct handoff, not a polite reply (anthropic.com)
Operational ExcellenceEscalation UX, audit traces, release of policy and tool configs
CostLong chats, retries, and live API fan-out without budgets

Context Optimization remains critical (policy vs untrusted customer text vs tool results). (anthropic.com) Sustainability follows from wasted loops and oversized context.

Dominant cross-pillar trades​

No design maximises every lens. For this workload, the usual imbalances are:

  • Cost × Accuracy × Speed: Live API reads + policy RAG + validators buy accurate account outcomes; they add latency and spend. A single fast model with stuffed FAQ context is cheaper/faster and more often wrong or stale. (platform.openai.com; arxiv.org)
  • Security × Autonomy: HITL or policy gates on refunds/writes buy Security and reduce Excessive Agency; they spend Autonomy and Speed (queue/wait). (genai.owasp.org)
  • Security × Performance: Identity gates and Tool Gateway checks add latency before personalised tools; skipping them speeds chat and expands blast radius. (ncsc.gov.uk)
  • Reliability × Cost: Escalation and dual knowledge paths (FAQ + APIs) raise operating cost; transcript-only “success” looks cheap until disputes arrive. (anthropic.com)
  • Autonomy × Cost: Unbounded conversational loops without turn/spend budgets look responsive until Cost and Context explode.

Typical starting bias for this workload: Security and Reliability (correct outcome / clean handoff) over Autonomy and raw Speed; use Cost budgets so safety controls do not become open-ended spend.

Common failure modes​

  • Model invents account facts instead of calling APIs.
  • Write tools callable directly from the model without a Tool Gateway. (genai.owasp.org)
  • Escalation drops context, forcing the customer to repeat themselves.
  • FAQ RAG overrides live policy or conflicts with API truth without resolution rules.
  • No budget on turns → endless “helpful” loops.
  • Treating the chat transcript as proof of completion. (anthropic.com)

Trade-off log​

  1. Which actions are read-only, policy-auto, and human-required — and why?
  2. How is customer identity established before account tools?
  3. What escalation triggers are mandatory, and what state is persisted?
  4. How do we resolve conflicts between FAQ/RAG and live API state?
  5. Where did we land on Cost × Accuracy × Speed, and which pillar pairs moved (e.g. Security↑ × Autonomy↓ / Performance↓)? Why is that imbalance acceptable?

When this is not the pattern​

Sources​

Discussion
Sign in with GitHub to comment