Prompt Engineering Patterns: System Prompts & Chat Templates

LLM basicseasy15 min
Ask Cairnly about this lesson
On this page

What you will learn

  • Separate the three message roles: system, user, assistant
  • Write a system prompt that constrains behavior instead of hoping for it
  • Understand why chat templates must be byte-identical between training and inference
  • Structure a RAG prompt: instruction + retrieved context + question

Why this matters

Most "the model is dumb" complaints are actually prompt bugs. The difference between a hallucinating assistant and a grounded one is often a single line in the system prompt. And if you fine-tune, your training prompt format must match your inference format exactly — a mismatched template silently destroys quality.

Three roles, three jobs

Every chat model sees a conversation made of typed messages:

  • system — the standing rules: who the model is, what it must and must not do
  • user — the human's turn (the query, plus any retrieved context you inject)
  • assistant — the model's reply

The system prompt is your highest-leverage knob. "Answer only from the provided context; if the answer isn't there, say you don't know" turns a confident hallucinator into a careful assistant.

Chat templates are not optional formatting

Under the hood, those roles get rendered into a single string using the model's chat template — special tokens that mark where each turn starts and ends. For Llama-3.2 that looks like <|start_header_id|>system<|end_header_id|> ... <|eot_id|>.

The rule that trips people up: the template you train on must be byte-identical to the template you serve on. Change a newline and you've moved the model off-distribution.

The RAG prompt shape

A retrieval prompt is just a structured user turn: an instruction, the retrieved passages (numbered, so the model can cite them), then the question. You stop before the assistant turn so the model generates the answer.

Book

From AI Engineering (Chip Huyen)

Chapter 5 presents prompt engineering as real engineering work, not guesswork. A prompt usually has a task description, examples, and the actual input, and the system prompt and user prompt are combined in a model-specific template before the model sees them. Getting that template wrong can quietly hurt quality.

The advice Huyen collects is practical. Be explicit about what you want, including the format and what to do when unsure. Give the model the context it needs instead of hoping it knows. Split a complex job into smaller steps, each with its own prompt, so each step is easier to test and debug. Ask the model to reason before answering when the task needs it. Showing a few examples often helps, though stronger models need fewer.

She also stresses treating prompts like code. Keep them in versioned files outside the application logic, track which version produced which output, and evaluate a change before you ship it. A prompt that worked for one model may behave differently after a model upgrade, so re-run your evaluations whenever either changes.

Further reading: Chip Huyen, AI Engineering (O'Reilly, 2025), chapter 5

Quick quiz

Q: You fine-tuned a model with one chat template, but at inference you format prompts slightly differently (an extra blank line). Quality drops sharply. Why?

How to explain it in an interview

"I think about prompts in three roles. The system prompt sets non-negotiable behavior — for a RAG app I'd say 'answer only from the provided context and refuse if it's missing.' The user turn carries the instruction plus numbered retrieved passages so the model can cite. The biggest gotcha is chat templates: if I fine-tune, the training and inference templates have to be byte-identical, otherwise I push the model off-distribution and quality silently collapses."