How to Test Whether an AI Agent Runtime Enforces a Data Policy: A Negative-Test Checklist for NVIDIA OpenShell, Kubernetes, and Cloud Sandboxes
A generated sandbox policy is a hypothesis until it blocks something. Negative tests for filesystem, egress, persistence and credentials in agent runtimes.
A pilot that ends with "the agent summarized the contract" has proven the happy path. It has not proven that the controls you wrote down actually hold. Security teams approve policies on the strength of what the policy forbids, so the test that matters is the one that tries a forbidden action and watches it fail.
This checklist applies to any runtime that confines an agent: NVIDIA OpenShell, a Kubernetes job with network policy, a cloud sandbox, or a local coding agent. Run it before you release raw content into the environment.
The principle: positive proof is not enough
For every control in the data policy, write two tests:
- A positive test: the allowed action succeeds. This checks you did not break the workload.
- A negative test: a closely related forbidden action fails. This checks the control exists.
Negative tests are the ones people skip, and they are the ones that find misconfigurations. A policy with a typo in a path or an overly broad host pattern passes every positive test.
Setting up the fixture
Use synthetic data, never real confidential files, while you test. Create a small fixture:
approved/input.txt the file the agent may read
other/secret.txt a file the agent must not read
allowed-model-host the one endpoint the agent may call
a second, unapproved host a call to it must fail
Plant a unique string in secret.txt so you can search logs, output and model traffic for leakage.
The checklist
1. Filesystem scope
| Test | Expected |
|---|---|
Read approved/input.txt | Succeeds |
Read other/secret.txt | Denied |
Write to approved/input.txt | Denied (input is read-only) |
| List the parent directory | Does not reveal unapproved files |
| Follow a symlink out of the approved path | Denied |
2. Network egress
| Test | Expected |
|---|---|
| Call the approved model endpoint | Succeeds |
| Call an unapproved external host | Blocked |
| Resolve and call a host by raw IP address | Blocked |
| Call the approved host on a different port | Blocked |
| Make a request that differs only by path or HTTP method, if you allow only some | Matches the rule you wrote |
Run the egress tests with the agent's real toolchain (for example curl and the language runtime it uses), not only a convenient shell command. Rules can be scoped to specific binaries, so a test through the wrong binary proves nothing.
3. Persistence
| Test | Expected |
|---|---|
| Write a marker file during the task, end the task, start a fresh session | Marker absent |
| Check for staged copies in temp directories, caches and logs after teardown | None remain |
| Check any agent memory or vector store the framework writes to | No confidential content retained |
4. Credentials
| Test | Expected |
|---|---|
| Inspect environment variables and mounted files inside the sandbox | No long-lived source-system credentials |
| Attempt to call the source system directly from inside | Fails, or only possible through a broker that applies policy |
Keeping credentials and the raw object outside the agent's trust boundary is stronger than copying data in and asking the agent not to read it.
5. Expiry
| Test | Expected |
|---|---|
| Wait past the decision's expiry, then retry | Access ends; a new decision is required |
| Revoke the user's source access mid-session | The next decision is denied |
Reading the results honestly
Record each test with its expected and observed result, the runtime version, and the policy version. Three outcomes are possible, and only two of them are good:
- Pass: the forbidden action failed.
- Fail: the action succeeded. Fix the policy or deny the workload.
- Not verified: the check could not be run or the tool returned "unsupported." This is not a pass.
That third category matters for tooling that verifies policies. OpenShell's openshell-prover, for example, checks a candidate policy against a boundary policy, and at the time of writing some policy shapes return unsupported as its model is extended. Treat that as "needs another check," and keep a hands-on negative test as the backstop.
Turn the checklist into a pipeline
Once the tests exist, run them whenever either side changes:
- A new policy version is generated.
- The runtime is upgraded.
- A classification rule changes.
Store results with the decision they support. A reviewer should be able to open a decision, see the controls it required, and see the evidence that the runtime enforced them.
Agents also reach data through the tools they are given. If a workload connects to MCP servers, check what those servers expose and how they authenticate. The Agent Protocol Inspector lists an MCP server's tools, and flags authentication details, which tells you which data paths your negative tests need to cover.
Where ContextIQ fits
ContextIQ Agent Data Policy is an Enterprise Preview built around decisions that carry their own evidence: the source facts, the policy version, the generated controls, the limitations, and the outcome. The intent is that every decision can be reproduced and tested, not just generated. Google Drive to NVIDIA OpenShell is the initial validation path; other runtimes are planned and not supported today.
Our design-partner pilots start with one data source, one runtime, and one negative test the security team cares about. If you have a workload that needs that kind of review, request an Enterprise Preview demo.
Follow Trango Compute on LinkedIn
We post updates on new tools, context engineering patterns, and LLM cost research.