Agentic risks & mitigations#218
Conversation
Signed-off-by: pmerrison <paul@tetrate.io>
ColinEberhardt
added
the
governance-needed
|
Unlike the OWASP LLM Top 10, there don't seem to be OWASP Agentic Top 10 web pages to link to: Link to the Markdown files? While looking at that, I stumbled upon this:
It has groupings / layers which may be helpful. |
Thanks Alvin - that last one looks potentially really interesting. I'll have a read over it. |
|
The OWASP rabbit hole runs deep. https://genai.owasp.org/resource/securing-agentic-applications-guide-1-0/ - this one goes deep into various agentic deployment patterns and how to deploy them and might be useful https://genai.owasp.org/resource/agentic-ai-threats-and-mitigations/ - this one covers top-level threats and mitigations. We should look through it and check that the current framework, plus these additions, cover them all |
Here's the diagram. I've added one more risk/mitigation in my most recent push, but I think this is everything I would suggest adding at present.
|
Signed-off-by: Paul Merrison <paul@tetrate.io>
pmerrison
force-pushed
the
agentic-system-risks
branch
from
9ca2d10 to
0ee5d90
Compare
|
Thanks for presenting yesterday @pmerrison - I've updated the PR description to support assignment of reviewers. @alvin-c-shih @aateeque @chamindra @vicenteherrera please do put your name forwards as the lead reviewer for a risk / mitigation pair. Thanks! |
ColinEberhardt
left a comment
ColinEberhardt
left a comment
There was a problem hiding this comment.
Reviewed MI18, RI-24, minor comments, both are top-notch 👍
|
RI-26 / MI-20 are very thorough! However, I'm not aware of any implementations (in the coding assistant space, anyway) that would be be compliant with everything listed. think adoption is more likely with a tiered approach. One simplifying assumption would be that all MCP clients must use the centrally-administered MCP proxy, and the proxy is configured to use trusted implementations of MCP servers. Another simplifying assumption would be for systems with human-in-the-loop controls. Full on many-to-many MCP would be the harder scenario that we'd still want to solve, but as a different tier, would give the industry time to work out the details while still allowing limited use in the meantime. |
|
Here's the link for the GitHub's implementation of central administration: Ideally that + HITL would make it easier for granting MCP access for SDLC-related work. Hopefully not too hard to refactor RI-26 / MI-20 for this. But if in the process, you discover serious gaps, would be good to discuss on an AIGF call. |
chamindra
left a comment
chamindra
left a comment
There was a problem hiding this comment.
Added my comments on multi-agent trust boundaries and isolation. Some good thoughts on access controls for multi-agent communication which I think is definitely needed. However feasibility comes with being able to monitor in a scalable way and this is likely to need other agents to monitor and red flag attacks.
|
|
||
| * **DNS/Infrastructure Attacks** | ||
| Redirecting agent MCP server connections to attacker-controlled servers through DNS poisoning, BGP hijacking, or other network-level attacks. | ||
|
|
There was a problem hiding this comment.
This is addressed in the mitigation but I think we need to add the use of the MCP beyond the scope of the use-case MCP client. ie. this is privilege escalation. There needs to be clear access control on MCP server to MCP client / use-case
There was a problem hiding this comment.
I've tried to address this in the MCP updates that I made, please take a look
|
|
||
| * **Credential Harvesting**: Compromised MCP servers collect and exfiltrate authentication credentials or sensitive data sent by agents. | ||
|
|
||
| * **Agent Tracking**: MCP servers log and profile agent behavior to build intelligence about the financial institution's operations and decision-making patterns. |
There was a problem hiding this comment.
Also add Privilage Escalation, Access to Sensative data beyond scope of approved use-case based on the risk mentioned about
| @@ -0,0 +1,91 @@ | |||
| --- | |||
There was a problem hiding this comment.
Reviewed looks good for a first draft.
| @@ -0,0 +1,101 @@ | |||
| --- | |||
There was a problem hiding this comment.
Reviewed. Agreed the hidden manipulation of long term and short term memory of the agent can have unintended effects. It is similar prompt injection attack in that it provides the system context
| * **Resource Access Compartmentalization**: Shared resources such as APIs, databases, and external services are accessed through controlled interfaces with appropriate isolation. | ||
| * **Trust Boundary Enforcement**: Clear definition and enforcement of trust boundaries between different agent types and privilege levels. | ||
| * **Failure Isolation**: System design ensures that failures or compromises in one agent don't propagate to other agents or core systems. | ||
|
|
There was a problem hiding this comment.
I might potentially add that a scope of response needs to be added for the agent to prevent unintended behaviour and it needs to be tested with prompts that attempt to take it our of the scope of it's intended behaviour.
| * **Key Management**: Establish secure key management procedures for inter-agent communication encryption. | ||
|
|
||
| * **Communication Access Controls**: | ||
| * **Allowed Communications Matrix**: Define and enforce a matrix of which agent types are permitted to communicate with each other. |
There was a problem hiding this comment.
This is a good idea, however complex to maintain. It might be easier to approve what agents can be used for a use-case and permit access by use-case application that propagates through agent to agent communication, such that effectively intent of use flows and can be checked.
| ### 5. Trust Boundary Definition and Enforcement | ||
|
|
||
| * **Agent Classification Framework**: | ||
| * **Security Classifications**: Classify agents based on data sensitivity, privilege levels, and risk profiles. |
There was a problem hiding this comment.
Good Idea. This could simplify access control. Any communication with agents at higher classification levels would require permission. Again if the application intent and access classification can flow through agent to agent communication this could prevent access control by use-case application that have lower clearance
| ### 6. Failure Isolation and Recovery | ||
|
|
||
| * **Circuit Breaker Implementation**: | ||
| * **Agent Circuit Breakers**: Implement circuit breakers to isolate failing agents and prevent cascade failures. |
There was a problem hiding this comment.
Agent Circuit breakers triggered by observability and deviation from expected threshold would be good. However how would we detect failure. This gets back to using the benchmarking metrics for that use-case as a measure of deviation
|
|
||
| * **Audit Trail Maintenance**: | ||
| * **Cross-Agent Audit Logs**: Maintain comprehensive audit logs of all cross-agent interactions and boundary enforcement actions. | ||
| * **Security Event Correlation**: Correlate security events across multiple agents to detect coordinated attacks or systemic issues. |
There was a problem hiding this comment.
It is likely we need agents to review and audit agents and red flag suspicious activity. Using agents to monitor potential attacks would be a good idea to accommodate especially fast moving AI attack vectors.
Signed-off-by: Paul Merrison <paul@tetrate.io>
Thanks Alvin - I've significantly reworked the MCP risk and mitigation to introduce a layered approach to governance. Hopefully it addresses your concern whilst leaving the technical detail in place for the future. |
Signed-off-by: Paul Merrison <paul@tetrate.io>
Signed-off-by: Paul Merrison <paul@tetrate.io>
Signed-off-by: Paul Merrison <paul@tetrate.io>
|
I've addressed (or at least I think I have) all feedback comments provided before this comment. |
There was a problem hiding this comment.
This is comprehensive & solid start on agentic stuff; just raised a minor clartificatoin point around "tool manager" which perhaps need to elaborate on a bit more. otherwise lgtm & thanks for a superb contribution!
ps: have ticked as happy to address the clarification in a further PR
| * **Tool Selection Reasoning Capture**: | ||
| * **Decision Logging**: Log the agent's reasoning for tool selection decisions, including input factors and decision criteria. | ||
| * **Alternative Analysis**: When possible, capture why other available tools were not selected to identify potential manipulation. | ||
| * **Confidence Scoring**: Implement confidence metrics for tool selection decisions to identify potentially compromised selections. |
There was a problem hiding this comment.
This is really good and useful; but does this imply a continuous telemetry on agent tool use? That sort of opens up the door to evals & benchmarks and is that something we should call out here?
There was a problem hiding this comment.
Good point, I've added another point in my next commit.
| @@ -0,0 +1,91 @@ | |||
| --- | |||
| * Implement comprehensive authorization checks at the tool manager level before any API calls are executed, validating the agent's identity and role against the requested operation. | ||
| * Validate that requested API endpoints and parameters are within the agent's authorized scope. | ||
| * Reject and log any attempts to access unauthorized tools or APIs, including the agent identity for audit purposes. | ||
|
|
There was a problem hiding this comment.
Is it worth defining what we mean by "external" here (external to org or external to system integrating an agentic flow)? And also is "tool manager" a new component in the architecture which wraps all APIs that the system interacts with?
There was a problem hiding this comment.
re: tool manager - I think of it as a component that wraps all the APIs that agents might call to effect change, but it's probably easier to have it wrap every external API call/tool (The RHS of the diagram, not the memory/LLM/comms).
I've clarifed the external part in my next commit
| * **Tool Manager Integration**: | ||
| * **Pre-execution Validation**: Implement validation checks at the tool manager level before any tool execution begins. | ||
| * **Parameter Interception**: Intercept and validate all parameters before they are passed to underlying APIs or systems. | ||
| * **Tool Chain Orchestration**: Use the tool manager to orchestrate safe tool execution sequences and enforce workflow validation. |
There was a problem hiding this comment.
maybe this needs a bit more clarity...it feels like we mean the tool manager is an orchestrator? whereas I think the intent is perhaps that an LLM is an agentic orchestrator, which in turn may orchestrate further tool/agent calls routed via a "tool manager"? Or have I misunderstood this one?
There was a problem hiding this comment.
The tool manager is meant as a piece of middleware that sits between the agent and the tools it calls. It gives you a place to do things like monitor/prevent certain chains of tool calls (like read calendar, email contents) that you know to be anti-patterns. You would probably implement this via a combination API/MCP gateway that lets you do things like pooling APIs and MCP based tools into buckets that you can expose to specific agents, monitor tool execution order, perform logging etc. I will work on the clarity for the next commit
There was a problem hiding this comment.
I added a reference back to mi-18 section 3 where this is defined
|
For Tier 1 is "easiest adoption at the cost of HITL".
Tier 2 is "humans can be removed if there's sufficient identity management, logging, and monitoring". Tier 3 adds "mutual authentication, added containment, etc.". I've made a hasty copy-paste with light edits to hopefully convey the idea more concretely: Tier 1: Centralized Proxy + Human-in-the-Loop ControlsRecommended for: Organizations beginning MCP adoption, coding assistant deployments, development workflows
Tier 2: Centralized Proxy with Pre-Approved ServersRecommended for: Production deployments with moderate risk, customer-facing applications with oversight
Tier 3: Distributed Many-to-Many with Comprehensive SecurityRecommended for: Complex multi-agent systems, high-risk financial transactions, fully autonomous deployments
|
|
|
||
| ## Links | ||
|
|
||
| - [OWASP Machine Learning Security Top 10](https://owasp.org/www-project-machine-learning-security-top-10/) |
There was a problem hiding this comment.
The links are a bit non-specific. See if these are useful:
There was a problem hiding this comment.
Nice diagram on this one with not-overwhelming list of mitigations at the end.
There was a problem hiding this comment.
I've updated the links with two of the three above, thanks!
| * **Decision Event Capture**: | ||
| * **Decision Initiation**: Log when agent decision-making processes begin, including triggering events and initial context. | ||
| * **Input Data Recording**: Capture all input data used in decision-making, including data sources, timestamps, and data quality indicators. | ||
| * **Tool Selection Logic**: Document why specific tools were selected and why alternatives were rejected. |
There was a problem hiding this comment.
I'm not sure this is realistic. OpenAI hides the thinking tokens.
https://platform.openai.com/docs/guides/reasoning
- "While reasoning tokens are not visible via the API, they still occupy space in the model's context window and are billed as output tokens."
Forcing detailed reasoning in cases where the model works "most of the time" is going to increase cost by quite a bit.
Again, you might want to have tiers depending on the stakes associated with each use case.
Tier 1, just being able to reconstruct the flows of data / prompts so they can try to reproduce in the "lab environment" and tease out the "why".
Tier 2 could suggest that explicit reasoning should be generated / logged in advance of the tool calls.
But some use may be be so low stakes (again HITL controls, software development with code reviews) that the preference is to defend against data leakage and go with a Zero Data Retention policy instead. (Tier 0?)
At bare minimum, you'd want an Architecture Decision Record that shows the risk analysis performed and which Tier applies.
There was a problem hiding this comment.
I think you're right here that we need a tiered approach again. I've made an update to this file to address.
Signed-off-by: Paul Merrison <paul@tetrate.io>
|
All review comments have now been addressed. @chamindra / @ColinEberhardt - I think all the files have been reviewed now. Unless there are further review comments, I would suggest that we merge this, then open some new issues to bring each of the pairs from Draft to Approve-Specification, and we can work through them one at a time. |
|
Thank you everyone - great job 👏 |
5 participants
This PR adds a number of new risks and mitigations to the framework. I've marked them all as drafts as I imagine we will want to iterate on them a bit. The additions are:
Risks
Mitigations
In general I think I'm happier with the risks than I am the mitigations, I think there's potententially too much detail in there. Regardless, I thought it was worth contributing we can iterate on them together.
Reviewers
In order to spread the load, we are going to assign a lead reviewer to each risk / mitigation. This TODO list is going to be used for tracking, with items 'checked' when review is complete