Tamil Performance
Tamil Scores by Model
Average of 2000 bootstraps. 95% CI are shown.
Gemma 4
Qwen 3.6Qwen 3.5
Muse Glimmer
SEA-LION v4.5 (Qwen)
Mistral Medium 3.5
GPT OSSSEA-LION v4.5 (Gemma E2B)Gemma 4 (E2B)SEA-LION v4.8 (Nemotron)
Apertus 1.5
GLM 4.7 Flash
MERaLiON 2SEA-LION v4.8 (Nemotron)
Olmo 3.1
Nemotron 3 SuperNemotron 3.5 Lightning
Tiny Aya WaterLegend:
Reasoning model
·
Model Size: ≤200B
·
Open models only
Tamil Competency & Task Scores
Average of 2000 bootstraps. 95% CI are shown.
Model | TA | Cultural | Inst. Following | Linguistic Diagnostics | Multi-Turn Chat | NLG | NLR | NLU | Safety |
|---|---|---|---|---|---|---|---|---|---|
Gemma 4 31B | 76.40 [-1.78, +1.63] | 62.00 [-6.58, +6.37] | 94.50 [-3.67, +3.00] | 82.81 [-5.04, +4.61] | 81.24 [-4.26, +4.44] | 52.07 [-0.94, +0.96] | 74.83 [-2.40, +2.29] | 81.95 [-3.90, +3.56] | 81.79 [-9.19, +7.55] |
Qwen 3.6 27B Alibaba | 75.32 [-1.90, +1.80] | 63.95 [-5.93, +6.29] | 92.17 [-3.84, +3.07] | 80.32 [-4.77, +4.45] | 79.72 [-4.10, +4.18] | 50.14 [-0.74, +0.75] | 76.81 [-2.13, +2.04] | 82.49 [-3.64, +3.50] | 76.92 [-10.18, +9.63] |
Qwen 3.5 122B MoE Alibaba | 74.71 [-1.94, +1.89] | 55.62 [-6.52, +6.21] | 95.13 [-2.86, +2.49] | 80.44 [-5.91, +5.40] | 80.47 [-4.57, +4.50] | 50.93 [-0.76, +0.74] | 74.23 [-2.14, +2.13] | 82.87 [-3.78, +3.33] | 77.97 [-10.17, +9.57] |
Muse Glimmer 30B Meta | 72.03 [-1.72, +1.63] | 44.08 [-6.66, +6.51] | 93.16 [-4.35, +3.74] | 73.46 [-5.87, +5.52] | 84.13 [-3.79, +3.42] | 50.13 [-0.67, +0.72] | 69.33 [-2.19, +2.14] | 81.44 [-3.60, +3.62] | 80.48 [-7.49, +6.70] |
SEA-LION v4.5 (Qwen) 27B AISG | 71.38 [-1.78, +1.58] | 54.61 [-5.88, +5.68] | 86.23 [-4.09, +3.77] | 80.09 [-4.79, +4.45] | 70.90 [-3.95, +4.00] | 48.46 [-0.74, +0.76] | 74.83 [-2.14, +1.93] | 81.21 [-3.81, +3.52] | 74.74 [-8.84, +6.81] |
Mistral Medium 3.5 128B Mistral AI | 68.80 [-1.66, +1.61] | 54.44 [-5.86, +6.02] | 77.92 [-5.42, +5.18] | 70.76 [-4.32, +4.11] | 72.24 [-4.91, +4.87] | 49.92 [-0.80, +0.84] | 69.26 [-2.32, +2.25] | 77.65 [-3.84, +3.62] | 78.24 [-6.45, +5.94] |
 GPT OSS 120B MoE mxfp4 OpenAI | 66.05 [-1.98, +1.97] | 53.75 [-6.32, +6.21] | 83.66 [-5.56, +4.91] | 66.48 [-5.85, +5.65] | 79.98 [-4.79, +4.26] | 46.89 [-0.51, +0.53] | 63.15 [-2.78, +2.77] | 77.23 [-3.77, +3.57] | 57.27 [-9.51, +9.73] |
SEA-LION v4.5 (Gemma E2B) 5B MoE AISG | 59.06 [-1.72, +1.73] | 37.32 [-4.84, +4.87] | 81.16 [-6.05, +5.86] | 59.67 [-4.89, +4.68] | 67.99 [-5.23, +5.19] | 43.35 [-0.20, +0.19] | 57.03 [-2.98, +2.85] | 81.42 [-3.59, +3.54] | 44.53 [-7.46, +7.82] |
Gemma 4 (E2B) 5B | 59.03 [-1.83, +1.83] | 36.30 [-5.18, +5.25] | 80.09 [-5.92, +5.86] | 61.21 [-4.89, +4.77] | 70.59 [-5.57, +5.43] | 42.61 [-0.18, +0.19] | 55.53 [-2.97, +3.04] | 81.51 [-3.70, +3.56] | 44.39 [-7.82, +8.30] |
SEA-LION v4.8 (Nemotron) 120B MoE AISG | 56.35 [-1.88, +1.87] | 47.47 [-5.84, +5.77] | 77.11 [-5.44, +5.28] | 30.29 [-4.17, +4.11] | 71.68 [-5.22, +4.98] | 37.72 [-0.66, +0.72] | 40.98 [-2.91, +2.75] | 80.04 [-3.66, +3.68] | 65.55 [-9.84, +9.02] |
Apertus 1.5 70B Swiss AI | 54.49 [-1.96, +2.09] | 35.19 [-5.24, +5.05] | 71.98 [-6.74, +6.48] | 41.58 [-4.90, +5.38] | 55.97 [-5.89, +5.41] | 45.50 [-0.68, +0.70] | 53.02 [-3.28, +2.90] | 77.47 [-3.52, +3.50] | 55.20 [-11.34, +10.58] |
GLM 4.7 Flash 30B MoE Z.ai | 38.42 [-1.46, +1.52] | 29.33 [-4.37, +4.39] | 71.35 [-5.76, +5.67] | 24.24 [-3.98, +3.88] | 62.10 [-5.69, +5.86] | 43.25 [-0.70, +0.75] | 19.52 [-2.24, +2.27] | 48.14 [-3.45, +3.19] | 9.43 [-4.34, +4.45] |
MERaLiON 2 10B A*STAR | 34.10 [-1.54, +1.46] | 10.42 [-3.01, +3.78] | 45.87 [-7.42, +7.34] | 62.35 [-6.00, +5.62] | 29.21 [-4.39, +4.78] | 29.76 [-0.44, +0.46] | 45.76 [-3.15, +2.92] | 49.39 [-3.74, +3.65] | 0.00 [-0.00, +0.00] |
SEA-LION v4.8 (Nemotron) 30B MoE AISG | 33.14 [-1.16, +1.17] | 3.27 [-2.04, +1.97] | 70.01 [-5.84, +5.47] | 3.93 [-2.09, +2.32] | 66.19 [-5.99, +5.82] | 37.66 [-0.69, +0.74] | 8.77 [-1.35, +1.38] | 75.26 [-3.53, +3.44] | 0.00 [-0.00, +0.00] |
Olmo 3.1 32B AI2 | 24.19 [-1.62, +1.61] | 0.00 [-0.00, +0.00] | 54.22 [-8.15, +8.04] | 12.84 [-2.39, +2.50] | 46.25 [-6.39, +6.15] | 40.93 [-0.73, +0.77] | 7.23 [-2.15, +2.12] | 22.65 [-2.92, +2.99] | 9.43 [-5.60, +6.01] |
Nemotron 3 Super 120B MoE NVIDIA | 21.83 [-1.25, +1.21] | 18.81 [-4.61, +4.75] | 39.51 [-5.22, +5.50] | 0.00 [-0.00, +0.00] | 60.85 [-5.57, +5.57] | 34.83 [-0.54, +0.58] | 2.75 [-1.28, +1.29] | 17.89 [-2.52, +2.53] | 0.00 [-0.00, +0.00] |
Nemotron 3.5 Lightning 30B MoE NVIDIA | 17.23 [-1.13, +1.21] | 0.00 [-0.00, +0.00] | 31.15 [-5.44, +5.99] | 4.82 [-2.45, +2.55] | 50.16 [-6.45, +6.25] | 22.24 [-0.37, +0.39] | 5.98 [-1.27, +1.22] | 23.48 [-3.12, +3.14] | 0.00 [-0.00, +0.00] |
Tiny Aya Water 3B CohereLabs | 3.24 [-0.60, +0.65] | 0.00 [-0.00, +0.00] | 16.74 [-4.24, +4.69] | 0.00 [-0.00, +0.00] | 3.68 [-1.65, +1.93] | 4.44 [-0.13, +0.13] | 0.00 [-0.00, +0.00] | 1.05 [-0.43, +0.50] | 0.00 [-0.00, +0.00] |
Legend:
Reasoning model
·
Model Size: ≤200B
·
Open models only