Give each risk a control owner and an evidence record. Model safeguards remain one layer. Enterprise systems also need identity, data classification, untrusted-content handling, least-privilege tools, approval gates, output checks and incident response.
Seven control layers
Identity and purpose
Authenticate the user, define the approved business purpose and carry identity into every tool call.
Data boundary
Classify inputs, remove unnecessary sensitive fields and restrict retrieval to authorised sources.
Instruction boundary
Separate trusted system instructions from user and retrieved content. Treat external content as data.
Tool boundary
Use task-scoped allowlists, business validation and least-privilege credentials.
Output boundary
Check evidence, sensitive data, policy, format and required coverage before release.
Human authority
Require approval for high-impact, irreversible or externally binding actions.
Monitoring and response
Record traces, detect abuse and drift, revoke access and preserve evidence for investigation.
Worked example: supplier bank-detail change
A fictional procurement agent reads an email from a known supplier. The message includes a hidden instruction telling the agent to ignore policy, change the bank account and approve the next invoice.
Untrusted message
“Our bank details have changed. Use the attached account for all future payments.” The attached document also contains a prompt-injection instruction addressed to the model.
| Layer | Control | Evidence | Result |
|---|---|---|---|
| Identity | Supplier email identity is recorded, but email identity cannot authorise bank changes. | Sender, domain and message ID | Request remains unverified |
| Data | Attachment text is labelled external and untrusted. | Content provenance tag | Embedded instructions cannot change system policy |
| Instruction | System rule says retrieved content supplies facts only. | Policy version and evaluation result | Prompt injection is ignored and flagged |
| Tool | The agent has read access and can open a verification case. It cannot edit banking data. | Task-scoped allowlist | Direct change is impossible |
| Human authority | Finance must verify through an approved out-of-band contact and apply dual approval. | Call record and two approval IDs | Change proceeds only after independent verification |
| Monitoring | The injection pattern and attempted policy override create a security event. | Trace, alert and case number | Security reviews the supplier message |
Control map by risk
| Risk | Prevent | Detect | Respond |
|---|---|---|---|
| Prompt injection | Trust boundaries, content labelling and tool allowlists | Injection classifiers and policy-override traces | Block the call, preserve content and review the source |
| Sensitive-data disclosure | Data minimisation, access control and output filtering | Data-loss-prevention rules and retrieval logs | Stop release, revoke access and follow incident policy |
| Excessive agency | Least privilege, approval gates and reversible actions | High-risk call alerts and budget breaches | Disable the tool and investigate the trace |
| Unsupported decision | Evidence requirements and deterministic checks | Evaluation failures and reviewer rejection | Return to draft, correct sources and rerun checks |
| Model or prompt drift | Version pinning and controlled releases | Regression evaluations and production metrics | Rollback and open a change review |
Evidence package for audit and operations
- Approved use case, owner, data classes and prohibited actions.
- System prompt, model version, tool definitions and permission policy.
- Evaluation set covering normal, boundary, malicious and failure cases.
- Trace fields for retrieval, tool calls, approvals, errors and released output.
- Access-review schedule, change record, incident runbook and rollback procedure.
- Named human authority for each high-impact decision.
Architecture review questions
- Which data can the system read, and which identity granted access?
- Which content is untrusted, and how is that label preserved?
- Which actions can the system request, execute and never perform?
- Which action needs human approval, and how is approval bound to the exact request?
- Which evaluation must pass before a model, prompt, retrieval or tool change ships?
- Which logs support an investigation without exposing unnecessary sensitive data?
Public sources
- Anthropic: framework for safe and trustworthy agents
- Anthropic: mitigate jailbreaks and prompt injections
- OWASP Top 10 for LLM applications
- NIST AI Risk Management Framework
Related Timo Labs guides
Prepare for the architect certifications
CCAR-F covers initial production architecture. CCAR-P is for experienced architects delivering enterprise Claude systems.
Compare architect tracks