What you will learn
- Turn a structured record into the model's exact chat-template string
- Keep training and inference templates byte-identical
- Load and format a single split into the
{"text": ...}shape trainers expect - Avoid the silent template bugs that wreck a fine-tune
Why this matters
Fine-tuning quality collapses on formatting mistakes that produce no error message. If your training text doesn't match the model's chat template — or differs from how you'll prompt at inference — the model learns the wrong distribution and you won't know until eval. Getting the formatting layer right is the difference between a fine-tune that helps and one that quietly hurts.
One function, used everywhere
Define a single format_record that renders a record into the model's full chat-template string — system, user (instruction + context), assistant (the target output) — with the model's exact special tokens. Both training and inference call it, so they can't drift apart.
def format_record(record: dict) -> str:
return (
f"{_BOS}"
f"{_SYS_OPEN}{SYSTEM_PROMPT}{_EOT}"
f"{_USR_OPEN}{record['instruction']}\n\nContext:\n{record['input']}{_EOT}"
f"{_AST_OPEN}{record['output']}{_EOT}"
)Training vs inference: the parity rule
At training you include the assistant turn (the target). At inference you stop right before it so the model generates. Everything up to the assistant turn must be byte-identical between the two — same tokens, same newlines. Build the inference prompt from the same helper, just omitting the final output. A stray space here silently degrades the model.
The shape trainers expect
SFT libraries (e.g. TRL) consume a dataset with a single text field. So the formatting step maps each record to {"text": format_record(r)}. Load just the split you want, format it, hand it to the trainer.
Format only what you train on
Load the train split for training and validation for eval — formatting the wrong split (or all of them) wastes work and risks training on data you meant to hold out.
from support_qa.prompting import format_record
from support_qa.data_io import load_records
def load_split(split: str, path: str) -> list[dict]:
return [r for r in load_records(path) if r.get("split") == split]
def format_records_for_training(records: list[dict]) -> list[dict]:
"""Map each record to the {'text': <chat-template string>} shape TRL expects."""
return [{"text": format_record(r)} for r in records]
# inference prompt uses the SAME template, minus the assistant output:
def build_inference_prompt(query: str, context: str) -> str:
return (
f"{_BOS}{_SYS_OPEN}{SYSTEM_PROMPT}{_EOT}"
f"{_USR_OPEN}{query}\n\nContext:\n{context}{_EOT}"
f"{_AST_OPEN}" # stop here — model generates the answer
)
train = format_records_for_training(load_split("train", TRAINING_FILE))Try it yourself
- Implement
format_recordfor your model's chat template - Format one record and print it; eyeball the special tokens and newlines
- Build the inference prompt and diff it against the training string up to the assistant turn — they must match
- Map a split to
{"text": ...}and confirm the shape your trainer wants - Introduce a one-space difference between train/inference templates and reason about the impact
Common mistakes
- Different templates for train vs inference — pushes the model off-distribution silently
- Hand-building the inference prompt separately — reuse the same helper, omit the output
- Wrong
{"text": ...}shape — the trainer can't find the field - Formatting all splits — you risk training on held-out data
- Forgetting special tokens / EOT — the model never learns where turns end
Checklist
- Wrote a single
format_recordfor the model's chat template - Built the inference prompt from the same helper, minus the target
- Verified byte-identical training/inference prefixes
- Produced the
{"text": ...}dataset shape for the trainer - Formatted only the intended split
Quick quiz
Q: Why build the inference prompt from the same format_record helper used in training?
How to explain it in an interview
"SFT quality dies on silent formatting bugs. I define one format_record that renders a record into the model's exact chat template — system, user with instruction and context, assistant with the target — using the model's special tokens. Training includes the assistant turn; inference stops right before it so the model generates, and I build the inference prompt from the same helper so the prefixes are byte-identical. Drift a single newline and the model is off-distribution with no error. For the trainer I map each record to a {"text": ...} field and format only the split I'm training on, so I never accidentally train on held-out data."