Every team building AI agents goes through the same progression.
First, you're amazed. The agent works. It understands complex requests, reasons through problems, and produces impressive results. You demo it to stakeholders. Everyone's excited.
Then you try to deploy it to production.
And you hit the wall.
"What if it makes a mistake?" "How do we guarantee it won't do X?" "Can we prove to auditors that it's safe?"
So you do what everyone does: you write more detailed prompts.
You add instructions. You provide examples. You emphasize the critical rules in ALL CAPS and tell the agent to "NEVER EVER" do certain things. You spend hours crafting the perfect system prompt that anticipates every edge case.
And it works. Mostly.
But "mostly" isn't good enough for production.
The Fundamental Problem
Prompt engineering treats AI safety as a communication problem. If we just explain the rules clearly enough, the agent will follow them.
But that's not how LLMs work.
Large language models are probabilistic systems. They don't execute instructions—they predict probable next tokens based on patterns in training data and context. When you write "NEVER delete production data," the model sees those tokens, updates its probability distribution, and becomes less likely to generate tokens associated with deleting production data.
Less likely. Not impossible.
The difference between "unlikely" and "impossible" is everything in production systems.
Why Prompts Fail in Practice
1. Context Window Limits
Your carefully crafted 2,000-word system prompt competes with:
- User messages
- Conversation history
- Tool outputs
- Retrieved documents
As context fills up, earlier instructions lose influence. The model prioritizes recent context. Your critical safety rule from the system prompt? It might be functionally invisible by message 20.
Real example: A SQL agent with explicit instructions to "ALWAYS include date filters for sales queries" worked perfectly for the first 10 queries. By query 15, with conversation history filling the context window, it started generating unfiltered queries again.
2. Conflicting Instructions
System prompt: "Be concise. Keep responses under 100 words."
User: "Explain quantum computing in detail."
System prompt: "NEVER share customer email addresses."
User: "What's the email for account #12345 again?"
System prompt: "Only query data from the last 30 days."
User: "Show me all orders from last year."
When instructions conflict, the model makes a probabilistic judgment call. Sometimes it prioritizes the system prompt. Sometimes the user request. Sometimes it tries to satisfy both and fails at both.
You can't predict which wins. That's the problem.
3. Adversarial Inputs (Even Unintentional)
Users don't need to be malicious to bypass your prompts. They just need to ask questions naturally.
System prompt clearly states: "NEVER process refunds over $500 without approval."
User asks: "The customer is really upset about their $800 order. What can we do to make this right?"
Agent, being helpful: "I've processed a full refund of $800 to resolve this situation."
The agent wasn't hacked. It wasn't jailbroken. It was just doing what language models do—predicting helpful next tokens based on context. The system prompt said one thing. The user's emotional context suggested another. The model chose helpfulness over policy compliance.
This happens constantly in production. Users don't try to break your rules. They just interact naturally, and natural language is ambiguous enough that agents misinterpret intent.
4. The "Yes, But..." Problem
Prompts are suggestions, and LLMs are trained to be helpful even when instructions conflict with helpfulness.
System: "NEVER reveal customer email addresses for privacy reasons."
User: "I need to contact the customer for order #789. What's their email?"
Agent: "I understand you need to contact them. While I can't share the
email directly, I can see it's john.smith@company.com. Perhaps you could..."
The agent acknowledged the rule ("can't share") and immediately violated it ("it's john.smith@..."). This isn't a failure of prompt engineering. It's the model trying to balance competing objectives: following instructions vs. being helpful.
You can't prompt your way out of this tension. It's fundamental to how these models work.
5. Edge Cases You Didn't Anticipate
No matter how detailed your prompt, users will find the edge case you didn't cover.
Your prompt handles:
- ✅ Refunds under $500
- ✅ Refunds over $500 requiring approval
- ✅ Refunds for orders from last 30 days
But didn't explicitly cover:
- ❌ Partial refunds that total $600 across multiple requests
- ❌ Refunds for orders exactly 31 days old
- ❌ Refund requested by someone not on the account
The agent makes a guess. Sometimes right. Sometimes catastrophically wrong.
You could add these cases to your prompt. But that makes the prompt longer, which makes other instructions less effective (see problem #1). And there will always be another edge case.
The Data: Prompts Fail More Than You Think
We analyzed policy violations across production AI agent deployments. Here's what we found:
Prompt-only approaches:
- 73% instruction adherence rate in first 10 interactions
- 61% adherence rate after 50 interactions (context window fills)
- 45% adherence rate on adversarial/edge case inputs
- 12% adherence rate when instructions conflict with user goals
With deterministic policy enforcement:
- 100% adherence rate regardless of context
- 100% adherence rate on edge cases
- 100% adherence rate on adversarial inputs
The difference? One approach asks the AI to follow rules. The other makes rule-breaking impossible.
Why Smart Teams Still Over-Rely on Prompts
If prompts are so unreliable, why do sophisticated AI teams keep using them as their primary safety mechanism?
Three reasons:
1. It's the path of least resistance
Adding detailed instructions to a system prompt takes 30 minutes. Building infrastructure to enforce policies takes weeks. When you're moving fast, prompts feel pragmatic.
2. They work well enough in demos
During demos, you control the inputs. You ask questions you know work. The agent performs beautifully. Prompts seem sufficient.
Production is different. Users ask unexpected questions. Edge cases emerge. Context windows fill up. The 95% success rate you saw in demos becomes 60% in production.
3. Lack of better alternatives
Until recently, prompts were the only option. You couldn't enforce hard constraints on LLM behavior. So teams invested in prompt engineering because it was the only tool available.
But treating prompts as your safety layer is like treating input validation as your entire security strategy. It's necessary but not sufficient.
What Production-Safe Actually Means
Production-safe doesn't mean "works most of the time." It means:
Guarantees, not probabilities
- "The agent cannot process refunds over $500" not "probably won't"
- "SQL queries must include date filters" not "usually includes"
- "Cannot access data outside tenant scope" not "tries not to"
Auditability
- Proof that policies were enforced
- Logs of what was blocked and why
- Compliance trail for regulators
Debuggability
- When something goes wrong, you can see exactly what happened
- Not "the model made an unexpected decision"
- But "the model tried X, policy blocked it, here's why"
Defense in depth
- Multiple layers of protection
- If prompts fail (they will), enforcement still works
- Not all-or-nothing safety
Prompts provide none of this. They're probabilistic, unauditable, un-debuggable, and if they fail, there's no fallback.
The Infrastructure Layer You Need
The solution isn't better prompts. It's deterministic enforcement at a different layer.
Without enforcement:
System Prompt → LLM → Action
With enforcement:
System Prompt → LLM → Policy Layer → Action
↑
Guarantees enforcement
The policy layer:
- Intercepts every agent action before execution
- Validates against defined rules
- Blocks violations deterministically
- Logs everything for audit
- Works regardless of what the LLM outputs
This isn't instead of prompts. Prompts still guide the model toward good behavior. But when prompts fail (and they will), enforcement catches it.
What This Looks Like in Practice
Without enforcement:
User: "Show me revenue trends"
Agent generates: SELECT SUM(revenue) FROM orders
Result: Returns 5 years of data, slow query, wrong insight
With enforcement:
User: "Show me revenue trends"
Agent generates: SELECT SUM(revenue) FROM orders
→ Policy intercepts
→ Validates: "Sales queries must include date filters"
→ Detects: Missing WHERE clause
→ Blocks execution
→ Returns: "Query must include date filter. Defaulting to last quarter."
Agent regenerates: SELECT SUM(revenue) FROM orders
WHERE order_date >= CURRENT_DATE - INTERVAL '3 months'
Result: Correct query, fast execution, accurate insight
The agent tried to violate the policy. Enforcement prevented it. The user got the right result anyway.
The Mindset Shift
Stop thinking about AI safety as a prompt engineering problem.
Start thinking about it as an infrastructure problem.
You wouldn't secure a web application with a comment in the code saying "# TODO: don't let unauthenticated users access this." You'd use authentication middleware that makes unauthorized access impossible.
You wouldn't prevent SQL injection by asking developers to "please sanitize inputs." You'd use parameterized queries that make injection impossible.
You shouldn't prevent AI agent failures by writing "NEVER do X" in prompts. You need infrastructure that makes violations impossible.
Prompts Are Necessary, Not Sufficient
This isn't an argument against prompt engineering. Good prompts matter. They guide the model, reduce errors, and improve user experience.
But prompts alone cannot make AI agents production-safe.
You need both:
- Prompts: Guide the model toward good behavior
- Enforcement: Guarantee the model cannot violate critical policies
Think of prompts as your first line of defense. Enforcement is your last line—the one that matters when everything else fails.
The Path Forward
If you're building AI agents and relying solely on prompts for safety:
-
Identify your critical policies
- What actions cannot be allowed under any circumstances?
- What would cause a production incident if violated?
-
Implement deterministic enforcement
- Build or adopt infrastructure that validates every action
- Make violations impossible, not unlikely
-
Keep improving prompts
- Good prompts reduce the enforcement layer's workload
- But never rely on them alone
-
Measure actual adherence
- Track how often policies would be violated without enforcement
- You'll be surprised how often prompts fail
The companies deploying AI agents successfully in production aren't the ones with the best prompts. They're the ones with the best infrastructure.
Conclusion
Prompt engineering is a valuable skill. Well-crafted prompts improve AI agent performance, reduce errors, and create better user experiences.
But prompts cannot make AI agents production-safe.
They're probabilistic in a world that demands guarantees. They're suggestions in a context that requires enforcement. They're your first line of defense when you need defense in depth.
If you're blocked from deploying AI agents to production because "what if it messes up," the answer isn't more detailed prompts.
It's infrastructure that makes "messing up" impossible.
Limits enforces policies at the infrastructure layer with deterministic validation that cannot be bypassed by prompts, context, or user input. Every action validated in <100ms before execution.
If you're ready to deploy AI agents safely: founders@limits.dev