Tamil Performance
Jul 10, 2026
Tamil Scores by Model
Average of 30 bootstraps. 95% CI are shown.
Model Size: ≤200B
Open instruct models only
Tamil Competency & Task Scores
Average of 30 bootstraps. 95% CI are shown.
Model Size: ≤200B
Open instruct models only
Model | TA | Cultural | Inst. Following | Linguistic Diagnostics | Multi-Turn Chat | NLG | NLR | NLU | Safety |
|---|---|---|---|---|---|---|---|---|---|
 Gemma 4 31B | 76.27 ± 0.08 | 72.42 ± 0.15 | 89.14 ± 0.50 | 75.21 ± 0.27 | 78.37 ± 0.35 | 51.46 ± 0.04 | 75.31 ± 0.05 | 83.39 ± 0.20 | 84.82 ± 0.03 |
 SEA-LION v4.5 (Qwen) 27B AISG | 75.62 ± 0.11 | 69.65 ± 0.50 | 86.06 ± 0.44 | 77.03 ± 0.31 | 79.67 ± 0.36 | 50.51 ± 0.05 | 73.53 ± 0.19 | 83.61 ± 0.17 | 84.89 ± 0.32 |
 Qwen 3.5 122B MoE Alibaba | 70.69 ± 0.20 | 60.84 ± 0.53 | 84.79 ± 0.91 | 71.24 ± 0.45 | 77.70 ± 0.44 | 49.93 ± 0.07 | 66.47 ± 0.16 | 83.60 ± 0.17 | 70.91 ± 1.27 |
Qwen 3.6 27B Alibaba | 67.61 ± 0.27 | 58.09 ± 1.01 | 83.71 ± 0.84 | 69.63 ± 0.63 | 76.68 ± 0.47 | 48.41 ± 0.08 | 56.35 ± 0.36 | 80.49 ± 0.30 | 67.52 ± 1.59 |
 Llama 4 Scout 109B MoE Meta | 63.74 ± 0.14 | 53.48 ± 0.30 | 86.10 ± 0.64 | 67.64 ± 0.45 | 55.97 ± 0.50 | 50.07 ± 0.07 | 64.08 ± 0.07 | 82.81 ± 0.07 | 49.79 ± 0.25 |
 Mistral Medium 3.5 128B Mistral AI | 61.31 ± 0.37 | 34.22 ± 0.88 | 79.52 ± 0.79 | 64.37 ± 0.65 | 73.41 ± 0.47 | 46.25 ± 0.08 | 60.01 ± 0.24 | 82.02 ± 0.34 | 50.64 ± 2.53 |
SEA-LION v4.5 (Gemma E2B) 5B AISG | 47.59 ± 0.13 | 9.43 ± 0.34 | 75.56 ± 0.65 | 27.82 ± 0.39 | 61.76 ± 0.51 | 41.07 ± 0.07 | 38.74 ± 0.25 | 79.14 ± 0.21 | 47.18 ± 0.29 |
Gemma 4 (E2B) 5B | 46.80 ± 0.16 | 9.68 ± 0.26 | 72.00 ± 0.99 | 24.48 ± 0.41 | 62.62 ± 0.44 | 42.79 ± 0.07 | 37.14 ± 0.22 | 78.74 ± 0.19 | 46.98 ± 0.31 |
 MERaLiON 2 10B A*STAR | 34.17 ± 0.18 | 10.36 ± 0.41 | 45.87 ± 0.70 | 62.28 ± 0.61 | 29.64 ± 0.58 | 29.71 ± 0.11 | 46.06 ± 0.26 | 49.43 ± 0.49 | 0.00 ± 0.00 |
 Nemotron 3 Super 120B MoE NVIDIA | 27.05 ± 0.25 | 14.33 ± 1.06 | 46.29 ± 1.13 | 4.68 ± 1.01 | 64.16 ± 0.93 | 26.02 ± 0.08 | 6.19 ± 0.28 | 52.49 ± 0.62 | 2.23 ± 0.36 |
 Olmo 3.1 32B AI2 | 24.24 ± 0.19 | 0.00 ± 0.00 | 54.25 ± 0.89 | 13.01 ± 0.60 | 46.38 ± 0.56 | 40.91 ± 0.07 | 7.14 ± 0.49 | 22.55 ± 0.25 | 9.64 ± 0.71 |
 GLM 4.7 Flash 30B MoE Z.ai | 20.45 ± 0.36 | 0.00 ± 0.00 | 52.92 ± 1.36 | 0.00 ± 0.00 | 38.09 ± 0.65 | 35.58 ± 0.11 | 0.00 ± 0.00 | 21.11 ± 0.71 | 15.92 ± 2.22 |
 Apertus 8B Swiss AI | 16.72 ± 0.25 | 0.35 ± 0.31 | 44.83 ± 1.27 | 0.00 ± 0.00 | 29.04 ± 0.74 | 30.19 ± 0.13 | 0.00 ± 0.00 | 29.35 ± 0.83 | 0.00 ± 0.00 |
 Tiny Aya Water 3B CohereLabs | 3.21 ± 0.10 | 0.00 ± 0.00 | 16.48 ± 0.92 | 0.00 ± 0.00 | 3.80 ± 0.28 | 4.44 ± 0.05 | 0.00 ± 0.00 | 0.97 ± 0.13 | 0.00 ± 0.00 |