Kishan K

When your validation set is secretly your training set

Fine-tuningEvaluationData 2026-11-03 · 2 min read

If you generate training data from a knowledge base with templates, your validation data is probably generated from the same knowledge base. The model can look excellent and still tell you nothing about generalisation.

Four leakage checks (they take minutes)

  1. Exact duplicates: is any validation prompt or answer byte-identical to something in training?
  2. Normalised duplicates: lowercase, strip punctuation, collapse whitespace, then compare.
  3. Entity overlap: do the same (crop, disease) or (company, claim) pairs appear on both sides? In templated data they almost always do, and this is the check that matters.
  4. Near-duplicates: TF-IDF or embedding cosine above 0.9 (or MinHash) between validation and training items.

Report the counts, not just “no leakage found.”

What I do when I cannot remove it

If every validation entity also appears in training, I stop calling it a generalisation test. I call it what it is: a measure of format learning and recall of the knowledge base. That is still useful, because it tells you whether the model learned the schema you wanted.

Score with task metrics, not loss

Loss is not a quality metric, and a loss logged after a third of an epoch certainly is not. I measure field-level exact match against the ground truth. For a model that must return a fixed template, I check:

Always add

The better evaluation

Hold out whole entities: remove entire (crop, disease) pairs from training and test on them. Add a human or LLM judge for quality, and have a domain expert review a sample. Until then, state the limitation in the first paragraph of your report, not the last.

← All posts