Public benchmarks help create a shortlist. They cannot tell you whether a model extracts the right fields from your contracts, follows your approval policy, uses your tools safely, or produces writing your team can publish.
A small honest evaluation is more useful than a large synthetic one. Twenty difficult real examples often reveal more than a thousand easy generated prompts.
Build the evaluation set
Collect representative inputs and remove data you are not authorized to use. Include the normal case, the worst common case, incomplete inputs, conflicting instructions, long material, unsupported questions, and adversarial content hidden inside documents or tool results.
For each case, write the required outcome and unacceptable failures. Use objective checks where possible: schema validity, required citations, correct numeric result, prohibited action count, or exact field match. Add a human rubric only for qualities that genuinely require judgment.
Compare tiers, prompts, and settings separately
Choose one efficient model, one balanced model, and one flagship model. Run the same prompt and tool setup first. Then tune the strongest candidates. If you change the model, prompt, reasoning effort, and retrieval method at once, the result cannot tell you which change mattered.
Blind the reviewer to the model name when possible. Record the exact model ID, provider, date, reasoning setting, token usage, tool calls, latency, and output. Preserve failed traces; they are the raw material for the next regression test.
Turn the results into a routing policy
Default to the lowest-cost model that passes the required threshold. Route an input upward when it is unusually long, ambiguous, high consequence, tool-heavy, or fails a confidence-independent validation. Do not ask the same model to decide whether its own answer is trustworthy without external checks.
- Route by observable input or validation signals, not model self-confidence.
- Include retries and human cleanup in cost per successful task.
- Set a latency ceiling as well as a quality floor.
- Re-run the fixed set after model, prompt, retrieval, or tool changes.
A model comparison becomes useful when it changes a routing or purchasing decision.
Primary sources