KATHE 2026 — English → Kashmiri Machine Translation

// the_story

Nine Days

KATHE 2026: translate English into Kashmiri, Perso-Arabic script. Run by Gaash Lab at NIT Srinagar with the Bureau of Indian Standards, scored as the geometric mean of BLEU and chrF++ over 1,730 held-out sentences. Short, everyday ones. He lost his pen. She plays a viola.

I entered alone, as team Noore. Nine days. Twenty-six submissions. IndicTrans2-1B zero-shot scored 8.00. I finished at 15.05.

Of that +7.05, the single largest piece — +5.05 — came from a check that used no GPU at all. I found it by counting characters in a JSON file. It took five minutes, and I should have run it on day one.

Zero

Kashmiri writes three short vowels as diacritics: kasra, damma, fatha. My fine-tuned model produced exactly zero of them across the evaluation set. Not rare. Zero.

At the same time it reproduced every other Kashmiri diacritic at above reference rates. Perfect on some marks, hard zero on three. That shape is not what under-training looks like — under-training degrades gradually. This was categorical.

The visible symptom was a single phrase. Kashmiri hedges gender on "I am": the reference writes چھُس/چھَس, two forms separated only by damma versus fatha. My model wrote چھس/چھس. Byte-identical either side of the slash. It had learned the shape of the hedge and dropped the mark that carried its meaning.

I lost two days to wrong answers first. The training data must be missing the marks — it wasn't, there were 270,799 kasra in the targets. Preprocessing must be stripping them — it wasn't, round-trip preserved them at 8.90 to 8.83 per 100 characters. The model must need more epochs.

What settled it was counting.

# dict.TGT.json — IndicTrans2's frozen target subword vocabulary
vocab = json.load(open("dict.TGT.json"))            # 122,672 entries

for name, mark in [("kasra", "ِ"),
                   ("damma", "ُ"),
                   ("fatha", "َ")]:
    print(name, sum(1 for tok in vocab if mark in tok))

# kasra 1     <- the bare standalone mark, and nothing else
# damma 1
# fatha 1
# ...while hamza-below lives inside 378 whole-word tokens, and the
#    inverted small-v above inside 244.

To write چھُس the model has to emit [چھ][ُ][س] — split a word apart to insert a bare diacritic that appears in no natural subword context. Beam search never does that, because the undiacritized whole word is always more probable.

The vocabulary is frozen inside the pretrained checkpoint. No amount of fine-tuning was ever going to fix this. And the metric preserves diacritics, so it was expensive: an otherwise-perfect translation missing only these three marks scores 67.66 out of 100.

A pretrained model can be structurally incapable of writing your language. Count your script's characters in the subword vocabulary before you book a single GPU hour.

The Fix Had No GPU In It

If fine-tuning can't produce the marks, restoration has to happen after decoding. I built two things, and they turned out to be complementary rather than redundant.

A 48,000-form diacritic lexicon. Strip the three marks from every word in the training targets, map the bare key to its most frequent diacritized form. Deliberately simple — deterministic, no runtime dependency, no added latency. Guarded by a minimum count and a dominance threshold, both tuned rather than assumed, because a form seen once is as likely to be a corpus typo as a fact about the language.
A 3.3M-parameter character tagger, trained from scratch. No base model, so no frozen vocabulary to fight.

Neither wins alone. The lexicon alone scored 13.52, the restorer alone 13.99. The union of the two scored 14.82, and tuning how aggressively the restorer is allowed to mark took it to 15.05.

Where the +7.05 actually came from:

Diacritic restoration — +5.05. No GPU.
Training-mix selection — +1.17. GPU.
Fine-tuning at all — +0.83. GPU.

The largest lever beat every training-side change combined by a factor of four. Shipping the translation model on its own scores 10.00 instead of 15.05. Restoration isn't polish here — it's a third of the system.

The Dev Set Was Lying

I built a careful dev set. 1,003 pairs, stratified to match the test set's word-length distribution, human references only, held out by exact pair key, with an assertion that fails the build if the count comes out wrong. Every leakage check passed.

Its correlation with the actual leaderboard was rho −0.39. Negative. Across the project it was mildly anti-predictive.

The cause is provenance, not leakage. My dev set was cut from BPCC — and IndicTrans2 was trained on BPCC. It's the corpus AI4Bharat released alongside it. Holding pairs out of my fine-tuning does not hold them out of the base model's pretraining. Every system I tested was being scored partly on its own base model's training data.

Submission 005 is where that cost me. It added a 260,000-entry two-sided context table to restoration. Best dev score of the entire project. On the leaderboard it regressed 5.3%.

The table and the dev set were both BPCC n-grams. At that size a table stops carrying lexical knowledge and starts memorising word sequences — and the dev set, same provenance, rewarded exactly that while the real test set did not. A 21,000-entry table generalised. 260,000 did not. This is dev-set overfitting through post-processing, a route that gets far less scrutiny than training, and nothing in a standard leakage check looks for it.

There was a warning sign I ignored. The output moved away from reference diacritic density while the dev score went up. Two independent signals disagreeing is worth more than either one agreeing.

The Metric Is a Specification

chrF++ runs at beta=2 — recall weighted four times precision. Consequence: a diacritic the reference has and I omit costs more than one I add that it lacks.

For eleven submissions I tuned the restorer to match reference density, about 9.6 marks per 100 characters, because matching the reference is the obvious target. It's also wrong. Removing that anchor and letting the model emit its natural 11.5 was worth +0.18, and the curve kept paying until 12.5, where BLEU's n-gram precision finally pulled back.

The nuance took three more submissions to find: this holds globally, not everywhere. When I let the model mark freely on words the lexicon didn't know, the score fell monotonically — 14.82, then 14.53, then 14.25. Extra marks are cheap where the model is confident and pure noise in the tail.

The scoring function is a specification, not a black box. Its parameters tell you which errors are cheap.

What Broke

Not one of these raised an exception. That's the whole point.

A checkpoint that loaded perfectly and translated everything to the empty string. IndicTrans2 ties lm_head to the decoder embeddings; save_pretrained drops the tied duplicate and the load path then zeroes both. 766 tensors, no error, a clean "all weights initialized" log, and no output at all. Verify a checkpoint by generating text, never by loading it.
A preprocessing call that hangs forever at 0% CPU. IndicProcessor pops one placeholder map per input off a queue. Call it with mismatched input and output counts and it blocks on Queue.get() with no timeout and no log line. I hit it twice, in two different places.
Positional scoring. The official scorer deletes the ID column and zips whatever remains. A submission sorted by ID against an unsorted input looks perfectly correct and scores near zero.
A stale config. My deployment post-processing file still pointed at a system from five submissions earlier. Everything ran fine. It was just 1.06 points worse. A fresh-clone rehearsal the day before the deadline caught it.

Every one of those would have passed a smoke test. So the submission script now refuses to write a file whose row count changed, that contains an empty row, or where restoration ran without adding a single mark. --self-test exits with code 2 if the three marks don't appear.

That last guard matters more than it sounds. If the restorer silently fails to load, the output is still fluent-looking Kashmiri. It just scores five points worse.

Where It Finished

Top 8. Not top 3.

Six awards were given out — three podium places and three special recognitions. First went to a team from Heidelberg University, second to IIT Jammu, third to NIT Srinagar. I got Young Achiever, one of the three recognitions, "in recognition of outstanding achievement and contribution".

I was aiming higher and I'm not going to pretend otherwise. But nine days, solo, against a language with almost no public parallel data — finishing inside the top eight and holding one of six awards handed out is a result I'll stand behind. The findings are the part I'd defend in a room.

Code and weights are public. Open-sourcing both by the deadline was an eligibility condition for every participant, not just for winners. An anonymous clone with no Hugging Face token runs the whole system end to end — verified, not merely intended, because the fresh-clone rehearsal found three blockers and one of them was that the restorer had never been uploaded anywhere at all.

کٲشُر

Kashmiri calls itself کٲشُر — koshur. It is the mother tongue of close to seven million people, it has been written in Perso-Arabic for centuries, and it carries a poetic line running from Lal Ded in the fourteenth century to whoever is writing in it this morning. Its vowel system is genuinely unusual — central vowels most Indo-Aryan languages simply don't have — and the script marks them with exactly the diacritics this entire project was about.

Now look at the word again. کٲشُر carries a damma.

My model could not write the name of the language it was translating into.

That is the sentence I keep coming back to. Not because it's an elegant failure, but because it is what "low-resource" actually means once you're inside it. It isn't only that there's little data. It's that the tokenizers, the benchmarks and the pretrained checkpoints everyone builds on were shaped around other languages, and yours arrives late, as an afterthought that doesn't quite fit the slot it's been given.

Almost no public parallel data. One usable pretrained model. A subword vocabulary that cannot spell three of its own vowels. None of that is a reason to skip the language. It is the reason to do it.

What I Learned

Audit the tokenizer before you book the GPU. Count how many of your target script's characters actually appear inside the subword vocabulary. A pretrained model can be structurally incapable of writing your language, and no reweighting, no extra epochs and no additional data will change a frozen vocabulary. Five minutes, worth +5.05.
A dev set can be worse than no dev set. In a low-resource language there is usually one public parallel corpus, and the strongest pretrained model was trained on it. The clean dev set — text no pretrained system has seen — is exactly the thing a low-resource language does not have. Evaluate offline to rank candidates, then spend submissions to decide.
"Use the biggest model" is wrong when the big model already ate your corpus. The 1B with LoRA scored below its own zero-shot baseline, 27.78 against 28.22. The distilled 200M gained 15.9% on the same data. Distillation had discarded information that BPCC training puts back; the 1B had nothing left to recover. That single result closed my entire training line.
Measure your test input before you choose a dev set. The test set averaged 7.3 words per sentence. FLORES — the dev set everyone reaches for — averages 21.6. I had built the whole plan around FLORES. Ten minutes of measurement invalidated a week of planned work.
Budget for verifying the output, not the run. Every silent failure above produced something that looked right. In a nine-day competition the expensive bugs are the ones that don't crash.

One admission to close on. I don't actually speak Kashmiri properly. I grew up inside it, I understand it fine, and I still come apart the moment a sentence needs more than a couple of clauses — my relatives have heard me mangle چھُس and چھَس in real time, no fine-tuning required.

So there were two of us learning this language over those nine days. One had 122,672 subword tokens and couldn't spell three vowels. The other had a lifetime of hearing it and still hesitates on the grammar. Between us we got most of the way there.

I'll keep working on my half. کٲشُر is worth it.