Designing System Prompts That Work

By Michael Samuel Naeem · November 1, 2024 · 5 min read

Your support bot answered a refund question with a poem because someone on the team wrote "be friendly and creative" in the system prompt and never tested it against angry customers. System prompts are not personality blurbs pasted into an API call — they are the contract between your product and the model. A good one defines role, scope, output shape, and refusal boundaries in language the model actually follows under load. A bad one is 40 words of vibes that collapse the first time a user asks for something slightly off-script.

Start with a role and a scope boundary

The opening lines matter because models weight early context heavily. State who the assistant is, what domain it operates in, and what it must not do:

You are a billing support agent for Acme SaaS.
You answer questions about invoices, plans, and payment methods.
You do not provide legal advice, medical advice, or access to other customers' data.
If asked to do any of the above, refuse briefly and offer to connect the user with a human agent.

"Billing support agent" anchors vocabulary and tone. The explicit refusal list prevents the model from improvising expertise it does not have. Without scope boundaries, every edge-case question becomes a coin flip between a helpful answer and a confident hallucination.

Define an output contract

Vague instructions like "respond helpfully" produce inconsistent JSON, markdown, and plain text in the same session. Specify the shape you need:

Respond in JSON with exactly these keys:
- "answer": string, max 3 sentences
- "needs_human": boolean
- "cited_policy": string or null

Do not include markdown fences or text outside the JSON object.

When you need prose instead of JSON, still define structure: bullet limits, heading rules, or citation format. Production pipelines that parse model output should treat the system prompt as a schema declaration, not a suggestion.

Layer guardrails without stacking contradictions

Guardrails belong in the system prompt when they apply to every turn: PII redaction rules, tool-use constraints, language requirements. A common failure mode is repeating the same rule in three phrasings that subtly conflict — "be concise" next to "explain thoroughly" next to "include all relevant details." Pick one primary objective and make tradeoffs explicit: "Prefer brevity; expand only when the user asks for step-by-step instructions."

For tool-calling agents, specify when to call tools and when to answer from context:

Call search_docs only when the user asks about a specific feature or policy.
Do not call tools for greetings, small talk, or questions already answered in the conversation.
After a tool returns results, cite the document title in your answer.

Separate stable instructions from dynamic context

System prompts should hold behavior that does not change per request. User-specific data — account tier, locale, recent tickets — belongs in the user message or a dedicated context block. Mixing them makes version control painful and causes stale instructions when you update customer data without updating the prompt template.

A practical pattern:

[System] Role, output format, guardrails, tool rules
[User]   Task + retrieved context + conversation history

Some frameworks support a developer or system tier between these layers. Use whichever your provider documents, but keep the mental model: stable rules up top, variable facts below.

Iterate with evals, not intuition

You cannot eyeball a system prompt across thousands of daily requests. Build a small eval set — 30 to 50 representative inputs including adversarial ones — and score outputs on format compliance, factual accuracy, and refusal correctness. Change one section at a time and re-run. Teams that batch-edit entire prompts weekly cannot tell which sentence fixed the bug and which one introduced a new one.

Track these metrics in production: format parse failure rate, escalation rate to human agents, and user-reported incorrect answers tagged by prompt version. When a new prompt version ships, compare those rates against the previous version for at least a week before declaring victory.

Version and document every change

Treat system prompts like application code. Store them in git, tag releases, and note why each change was made. "Made it friendlier" is not a changelog entry; "Added refusal for competitor comparisons per legal review 2024-10-15" is. When a regression appears, you need to bisect prompt versions the same way you bisect commits.

Common production mistakes

Teams get system prompts wrong in predictable ways:

• Skipping failure-mode rehearsal — run a game day or fault injection exercise before peak traffic, not after the first outage.
• Missing correlation context — every error path should carry request, trace, or tenant identifiers so incidents are debuggable.
• Optimizing for demo, not steady state — load tests, cache warm-up, and cold-start paths matter more than local dev latency.
• Undocumented trade-offs — if you chose speed over strict correctness (or vice versa), write that down for the next engineer.

Production implementations of system prompts fail when staging mirrors production topology poorly, rollback is untested, and on-call runbooks describe the happy path only.

Debugging and triage workflow

When system prompts misbehaves in production, work top-down instead of guessing:

1. Confirm scope — one tenant, region, or deployment stage? Narrow blast radius before deep diving.
2. Check recent changes — deploys, flag flips, config pushes, and schema migrations in the last 24 hours.
3. Compare golden signals — latency, error rate, saturation, and traffic for the affected surface vs. baseline.
4. Reproduce minimally — smallest input or scenario that triggers the failure; capture traces/logs with correlation IDs.
5. Fix forward or rollback — if rollback is faster than root-cause during incident, rollback first, postmortem second.
6. Add a guard — alert, integration test, or circuit breaker so the same class of failure is caught earlier next time.

Document the timeline during triage. Future you (and on-call) will need timestamps, not just conclusions.

Resources

• OpenAI prompt engineering guide
• Anthropic prompt design documentation
• Google Gemini prompting strategies
• LMSYS prompt library and comparisons
• OWASP LLM Top 10 — prompt injection risks