A 5-line baseline humbled my 99.6%
I fine-tuned Qwen-2.5-7B with QLoRA to classify financial claims into four compliance labels for Sentinel, my research-and-compliance agent. On a 271-claim held-out set it got 270 correct. That looks great on a resume.
Then I did what I should have done first: I trained a boring baseline.
The baseline check
TF-IDF features plus logistic regression, trained in seconds on the same 2,174 training claims, scored 269/271 on the same test set. The fine-tuned model’s one-error “win” is well inside the noise: the 95% Wilson interval for 270/271 is about 97.9–99.9%.
| Method | Accuracy on the 271-claim test set |
|---|---|
| Majority class | 29.9% |
| Nearest-neighbour label copy | 91.9% |
| TF-IDF + logistic regression | 99.3% (269/271) |
| Fine-tuned Qwen-2.5-7B (QLoRA) | 99.6% (270/271) |
Why the task was easy
I had generated the training data with an LLM from crisp label definitions. Synthetic data from one generator tends to be separable by style and keywords. I did check for leakage: zero exact or normalised duplicates between train and test, and no near-duplicate above 0.9 cosine similarity. So the test was clean, just easy.
The placeholder I found in my own script
While auditing, I found that my benchmark script had hard-coded numbers for the comparison models. They looked like measurements in a report, but they were constants. I removed every claim that depended on them.
What I do now
- Report n and an interval, never a bare percentage.
- Always ship a dumb baseline next to the model.
- Say where the data came from (here: synthetic).
- Never put a number in a README that a script did not just compute.
- Keep the claim honest: the interesting part of Sentinel is the agent (hybrid retrieval, citations, a compliance rewrite step), not the last decimal of a classifier score.
A result you can defend under hostile questions is worth more than a perfect-looking one.
← All posts