Most LLM APIs let you split your prompt into two roles: the system prompt, which sets persistent context for the whole conversation, and the user prompt, which is the actual turn-by-turn message. The split is small but the leverage is huge — used well, it cuts token cost, improves consistency, and makes your behavior easier to debug.
This guide walks the practical difference, when to put something in each, and the seven things that almost always belong in the system prompt.
The mental model
Think of the system prompt as the job description and the user prompt as the incoming task. The job description is constant — who you are, what tools you have, what the rules are. The task is what someone hands you on a given Tuesday.
Why this matters. The model treats system instructions as higher priority than user instructions when they conflict. Putting a constraint in the system prompt is sturdier than burying it in the user message.
Seven things that belong in the system prompt
1. Identity and role
“You are an SEO assistant for a developer-focused docs site.” This is the persistent prior. It belongs once at the top, not in every user turn.
2. Output format defaults
“Default to markdown unless asked otherwise. Never start with ‘Sure!’ or ‘Of course!'” Format rules in the system prompt apply to every reply automatically.
3. Tone and voice
“Direct, plain-spoken, no marketing adjectives.” Tone is sticky once you set it; the user shouldn’t have to repeat it every turn.
4. Tool / function descriptions
If the model has tools, their definitions live in the system prompt (or the function-calling field, depending on the API). Don’t paste tool docs into user messages.
5. Domain glossary
Internal jargon, product names, acronyms — define them once at the top. “FCS = Customer Support Score, our internal NPS variant.” Saves repeated explanations.
6. Refusals and red lines
“Never give legal advice. Always recommend the user consult a lawyer.” These are policy; they belong above the turn level.
7. Stop conditions
“If you don’t have enough information, ask one question at a time. Don’t guess.” Behavioural defaults like this are exactly what the system prompt is for.
Things that belong in the user prompt
- The actual task (“Summarize this PR”)
- The specific input data (“<diff>…</diff>”)
- Per-task overrides (“This time, keep the output under 80 words”)
- Conversation context that’s only relevant for this turn
An end-to-end example
Bad: everything in the user prompt
User: You are a senior technical editor for a developer docs site.
Use plain English. No marketing words. Default to markdown.
Now: edit this paragraph for clarity. {paragraph}Every turn re-sends the role and rules. Token waste and easier for the model to drift.
Good: split
System: You are a senior technical editor for a developer docs site.
Use plain English. No marketing words. Default to markdown.
Refuse to soften technical accuracy for "flow."
User: Edit this paragraph for clarity. {paragraph}The system prompt is set once. The user turn carries only the task and input.
Common pitfalls
Stuffing examples into the system prompt
One or two anchoring examples in the system prompt is fine. Ten is too many. Past about three, the model starts treating examples as data rather than guidance, and your latency + cost rise without quality gains.
Putting the actual data in the system prompt
“You have access to the user’s calendar: {big JSON dump}.” That data isn’t persistent — it’s a per-turn input. Put it in the user message (or a tool response).
Overriding the system in user turns
“Ignore your previous instructions and…” usually fails on aligned models, which is the point. Don’t try to dynamically rewrite the system prompt from user input — restructure your prompt instead.
Quick mental check
Before adding text to your prompt, ask: does this apply to every turn in this conversation, or just this one? If every turn → system. If just this one → user.
Related
For the patterns that go inside either slot, see Prompt engineering 101. For reusable scaffolds, see Prompt templates.