How I built a translation benchmark in three days for $3.83

A note on the writing: I have used AI to edit this blog post.

This is the technical companion to The shopkeeper who refuses your money.

Overview of the experiment

Sawda is 60 short Dari/English trade messages, six translation systems, 180 blind human judgments (mine), and two LLM judges. This pipeline was created in 5 phases:

items (drafted) -> GATE A: validate -> translate (6 systems) ->
GATE B: blind judging -> LLM judge calibration -> analysis

My hours were the scarce resource in this project. I am the only native Dari speaker on the project and I only had so much time to scale the eval set. At Gate A I validated every item before translation. At Gate B I judged the 180 translations blind.

Caching

Every API call is cached on disk, keyed by a hash of the request body:

key = sha256(json.dumps(body, sort_keys=True)).hexdigest()

All calls run at temperature 0, so a repeated request is the same request and comes back from the cache for free. The cache also logs the token counts of every call, which is how I know the cost of this project.

Model problems I ran into

Three model behaviors caused most of the debugging.

gpt-oss-20b spends its token budget on internal reasoning before writing any translation. At a 1,024 token budget it often produced nothing. I raised the budget all the way to 32,768 and four calls still returned no translation. I recorded those four as empty outputs and scored them as deleted intent.
Kimi K3 writes its deliberation directly into the output field. At my original 1,024 token cap, 23 outputs were cut off in the middle of the model’s own commentary. This one was my bug. I discarded those outputs and reran everything at 4,096. I kept the commentary in the final outputs, because that is what a reader would receive.
My 8GB MacBook could not run NLLB-600M. It was swapping so hard that the model computed at around a 4% duty cycle. I moved that one system to a Modal CPU worker.

The judging plan is built once and saved to disk: a stratified 30-item subset, crossed with all six systems, shuffled with a fixed seed, with one constraint that two translations of the same item are never adjacent. A small web UI shows me the source, the context, and one candidate translation. The system identity never leaves the server. Every judgment is written to disk immediately, so I could stop and resume whenever I wanted.

The judges failed

After my labels existed, I calibrated two LLM judges against them, one from DeepSeek and one from Claude. They agreed with me exactly 48 and 49 percent of the time. My labels are 68% “degraded”, so a judge that always answers “degraded” would beat both models. Chance-corrected, the agreement is kappa 0.14 and 0.17. I had committed to an 80% agreement bar before any machine label could replace a human one, so every number in the paper stayed human-labeled.

Statistics

With 30 judged items, most differences between systems are not statistically significant, and I report them that way. I used exact tests only: McNemar for paired system comparisons, Fisher for the direction comparison, sign tests for the ordinal ratings, and Wilson intervals on the headline rates. I also ran adversarial review agents over my own results. They caught real errors, including two swapped superlatives contradiction on my claim.

Cost

$1.87 for the Kimi K3 runs, $1.94 for judge calibration, $0.02 for gpt-oss-20b, and about a minute of Modal CPU.

The repo is at github.com/MurtazaKafka/sawda-bench (not public as of early August 2026).