What you will learn
- Define concrete quality rules: length bounds, PII, language, degenerate pairs
- Run a quality pass before and after LLM synthesis
- Quarantine rejects for inspection instead of silently deleting them
- Track drop rates so a broken filter doesn't gut your dataset
Why this matters
"More data" is not the goal — clean data is. A few hundred degenerate records (empty answers, leaked PII, wrong-language text, question-equals-answer) drag down a fine-tune or poison retrieval. A small set of explicit, logged quality rules is the cheapest quality lever you have, and running it twice — pre and post synthesis — catches both source junk and bad LLM output.
Make "quality" a list of explicit rules
Vague quality intentions don't filter anything. Encode concrete, testable rules:
- Length bounds — instruction too short (< 10 chars) or output too short (< 15) / too long (> 1500) is suspect.
- Degenerate pairs — instruction equal to output is a non-example.
- PII — drop records whose output contains an email or phone number.
- Language — reject text with characters outside your expected script (e.g. Cyrillic in an English corpus).
Each rule is a one-liner; together they remove the obvious bad data.
Filter twice: pre- and post-synthesis
Run quality filtering on the raw ingested data, and again after LLM normalization/synthesis. The first pass cleans the source; the second catches the LLM's own failures — a model can emit an empty rewrite, a too-long ramble, or accidental PII. One pass is not enough.
Quarantine, don't vanish
Write rejected records to a debug/rejected.jsonl rather than dropping them silently. When the drop rate spikes, you can look at what got cut and tell "the filter is too strict" from "the source got worse."
Watch the drop rate
A filter bug (or a bad source) can silently gut your dataset. Log how many records each rule drops, and treat an abnormal drop rate as an alert — the same discipline as the on-topic pass-rate monitor.
From AI Engineering (Chip Huyen)
Huyen's view in chapter 8 is that data quality usually beats data quantity. A small set of carefully chosen, correct examples can teach a model more than a large pile of noisy ones, and bad examples actively teach the wrong behaviour.
She describes quality along several lines: is each example relevant to the task, is the answer correct, does it follow the format you want, is it consistent with similar examples, and does the set cover the range of inputs your users will send. A dataset can be clean yet narrow, so coverage deserves its own check.
Filtering starts with looking. Read samples, plot simple statistics such as length and topic, and look for odd clusters. Then apply filters in rising cost: cheap rules first, for example removing empty, truncated or wrongly formatted rows, then scoring with a model for things rules cannot catch. Keep the rejected examples and the reason for each, because reviewing them shows whether a filter is too strict and helps you explain the pipeline later.
Further reading: Chip Huyen, AI Engineering (O'Reilly, 2025), chapter 8
import re
_EMAIL = re.compile(r"[\w.%+\-]+@[\w.\-]+\.[A-Za-z]{2,}")
_PHONE = re.compile(r"\b(\+?1[\s\-]?)?\(?\d{3}\)?[\s\-]?\d{3}[\s\-]?\d{4}\b")
def quality_filter(records: list[dict]) -> tuple[list[dict], list[dict]]:
kept, rejected = [], []
for r in records:
instr, out = r.get("instruction", ""), r.get("output", "")
bad = (
len(instr) < 10
or not (15 <= len(out) <= 1500)
or instr.strip().lower() == out.strip().lower() # degenerate
or _EMAIL.search(out) or _PHONE.search(out) # PII
or any("Ѐ" <= ch <= "ӿ" for ch in out) # wrong script
)
(rejected if bad else kept).append(r)
if rejected:
write_jsonl(rejected, "debug/rejected.jsonl") # quarantine, don't vanish
log.info("quality filter dropped %d/%d (%.1f%%)",
len(rejected), len(records), 100 * len(rejected) / len(records))
return kept, rejectedTry it yourself
- Implement
quality_filterand run it on a mixed batch - Add records that violate each rule and confirm each is caught
- Quarantine rejects to
debug/rejected.jsonland inspect them - Run the filter again after a (mock) synthesis step that emits one empty output
- Log the drop rate and decide a threshold that should trigger an alert
Common mistakes
- Vague "quality" with no rules — nothing actually gets filtered
- Filtering only before synthesis — the LLM's own bad outputs slip through
- Silently deleting rejects — you lose the ability to diagnose over-filtering
- No drop-rate monitoring — a filter bug can gut the dataset unnoticed
- Over-strict bounds — too-tight length limits cut valid records; tune against real data
Checklist
- Encoded explicit length/PII/language/degenerate rules
- Ran quality filtering before and after synthesis
- Quarantined rejected records for inspection
- Logged and thresholded the drop rate
- Tuned bounds against real data
Quick quiz
Q: Why run the quality filter a second time after LLM synthesis/normalization?
How to explain it in an interview
"Clean data beats more data, so I make 'quality' a list of explicit, testable rules: length bounds, degenerate instruction-equals-output pairs, PII in the output via regex, and wrong-script characters for a single-language corpus. I run the filter twice — once on the raw ingested data to clean the source, and again after LLM normalization and synthesis, because the model itself emits empty rewrites, rambles, or accidental PII. I quarantine rejects to a debug file instead of deleting them silently, so when the drop rate spikes I can look at what got cut and tell an over-strict filter from a degraded source — and I treat an abnormal drop rate as an alert."