Tamil Performance
Tamil Scores by Model
Average of 2000 bootstraps. 95% CI are shown.
Model Size: ≤200B
Open instruct models only
Tamil Competency & Task Scores
Average of 2000 bootstraps. 95% CI are shown.
Model Size: ≤200B
Open instruct models only
Model | TA | Cultural | Inst. Following | Linguistic Diagnostics | Multi-Turn Chat | NLG | NLR | NLU | Safety |
|---|---|---|---|---|---|---|---|---|---|
 Gemma 4 31B | 76.23 [-1.93, +1.86] | 72.41 [-6.96, +6.86] | 89.16 [-5.11, +4.77] | 75.24 [-6.01, +5.98] | 78.30 [-4.28, +4.19] | 51.42 [-0.96, +0.98] | 75.30 [-2.81, +2.56] | 83.31 [-3.57, +3.43] | 84.74 [-9.49, +6.87] |
 SEA-LION v4.5 (Qwen) 27B AISG | 75.62 [-1.91, +1.78] | 69.56 [-6.69, +6.43] | 85.57 [-5.93, +5.86] | 77.52 [-5.56, +4.97] | 79.71 [-4.24, +4.15] | 50.51 [-0.75, +0.77] | 73.55 [-2.35, +2.28] | 83.72 [-3.70, +3.54] | 84.80 [-9.56, +6.79] |
 Qwen 3.5 122B MoE Alibaba | 70.79 [-2.06, +2.16] | 61.17 [-6.66, +6.56] | 84.54 [-5.97, +5.46] | 71.59 [-6.37, +6.12] | 77.93 [-4.41, +4.34] | 49.89 [-0.75, +0.76] | 66.58 [-2.79, +2.69] | 83.64 [-3.82, +3.70] | 70.94 [-11.38, +11.55] |
Qwen 3.6 27B Alibaba | 67.74 [-1.87, +1.91] | 58.13 [-5.75, +5.94] | 83.51 [-5.78, +5.30] | 69.51 [-5.24, +5.07] | 76.97 [-4.53, +4.50] | 48.43 [-0.80, +0.80] | 56.47 [-2.28, +2.22] | 80.52 [-3.55, +3.57] | 68.35 [-9.69, +9.61] |
 Llama 4 Scout 109B MoE Meta | 63.63 [-2.30, +2.34] | 53.32 [-7.48, +7.57] | 85.88 [-5.64, +5.43] | 67.28 [-6.67, +6.20] | 56.31 [-5.76, +6.01] | 50.05 [-0.97, +1.02] | 64.12 [-3.30, +3.10] | 82.72 [-3.95, +3.91] | 49.37 [-11.30, +13.23] |
 Mistral Medium 3.5 128B Mistral AI | 61.37 [-1.91, +1.83] | 33.37 [-5.33, +5.61] | 79.50 [-6.17, +5.62] | 63.77 [-5.27, +5.28] | 73.32 [-4.85, +4.82] | 46.26 [-0.77, +0.78] | 59.93 [-2.64, +2.40] | 82.04 [-3.46, +3.32] | 52.78 [-9.80, +9.42] |
SEA-LION v4.5 (Gemma E2B) 5B MoE AISG | 47.69 [-1.74, +1.72] | 9.82 [-3.25, +3.46] | 75.46 [-7.12, +6.57] | 28.63 [-6.21, +6.58] | 61.65 [-4.95, +5.05] | 41.05 [-0.87, +0.91] | 38.82 [-4.06, +3.82] | 79.27 [-3.69, +3.80] | 46.79 [-6.21, +5.79] |
Gemma 4 (E2B) 5B | 46.89 [-1.85, +1.73] | 9.93 [-3.48, +3.56] | 72.11 [-7.34, +6.94] | 24.98 [-6.31, +7.16] | 62.93 [-4.94, +5.24] | 42.78 [-0.81, +0.84] | 37.01 [-3.84, +3.90] | 78.61 [-3.89, +3.86] | 46.79 [-6.42, +6.23] |
 MERaLiON 2 10B A*STAR | 34.10 [-1.54, +1.46] | 10.42 [-3.01, +3.78] | 45.87 [-7.42, +7.34] | 62.35 [-6.00, +5.62] | 29.21 [-4.39, +4.78] | 29.76 [-0.44, +0.46] | 45.76 [-3.15, +2.92] | 49.39 [-3.74, +3.65] | 0.00 [-0.00, +0.00] |
 Nemotron 3 Super 120B MoE NVIDIA | 26.83 [-1.29, +1.27] | 13.04 [-2.95, +3.33] | 46.16 [-6.52, +6.46] | 4.22 [-2.88, +3.02] | 64.31 [-5.24, +5.32] | 26.04 [-0.54, +0.56] | 6.02 [-0.86, +0.91] | 52.98 [-3.81, +3.69] | 1.83 [-1.29, +1.33] |
 Olmo 3.1 32B AI2 | 24.19 [-1.62, +1.61] | 0.00 [-0.00, +0.00] | 54.22 [-8.15, +8.04] | 12.84 [-2.39, +2.50] | 46.25 [-6.39, +6.15] | 40.93 [-0.73, +0.77] | 7.23 [-2.15, +2.12] | 22.65 [-2.92, +2.99] | 9.43 [-5.60, +6.01] |
 GLM 4.7 Flash 30B MoE Z.ai | 20.58 [-1.51, +1.59] | 0.00 [-0.00, +0.00] | 53.06 [-6.52, +6.10] | 0.00 [-0.00, +0.00] | 38.31 [-5.37, +5.67] | 35.56 [-0.45, +0.48] | 0.00 [-0.00, +0.00] | 20.96 [-2.79, +2.87] | 16.71 [-7.34, +8.45] |
 Apertus 8B Swiss AI | 16.67 [-1.12, +1.09] | 0.00 [-0.00, +0.00] | 44.58 [-6.48, +7.09] | 0.00 [-0.00, +0.00] | 29.25 [-5.25, +5.14] | 30.29 [-0.60, +0.60] | 0.00 [-0.00, +0.00] | 29.28 [-3.17, +3.19] | 0.00 [-0.00, +0.00] |
 Tiny Aya Water 3B CohereLabs | 3.24 [-0.60, +0.65] | 0.00 [-0.00, +0.00] | 16.74 [-4.24, +4.69] | 0.00 [-0.00, +0.00] | 3.68 [-1.65, +1.93] | 4.44 [-0.13, +0.13] | 0.00 [-0.00, +0.00] | 1.05 [-0.43, +0.50] | 0.00 [-0.00, +0.00] |