Thai Performance
Thai Scores by Model
Average of 2000 bootstraps. 95% CI are shown.
Gemma 4
Qwen 3.6
SEA-LION v4.5 (Qwen)
Qwen 3.5
Muse Glimmer
SEA-LION v4.8 (Nemotron)
GPT OSS
Mistral Medium 3.5
SEA-LION v4.8 (Nemotron)
GLM 4.7 Flash
SEA-LION v4.5 (Gemma E2B)
Apertus 1.5
Nemotron 3.5 Lightning
Gemma 4 (E2B)
Nemotron 3 Super
MERaLiON 2
Olmo 3.1
Tiny Aya WaterLegend:
Reasoning model
·
Model Size: ≤200B
·
Open models only
Thai Competency & Task Scores
Average of 2000 bootstraps. 95% CI are shown.
Model | TH | Cultural | Inst. Following | Knowledge | Multi-Turn Chat | NLG | NLR | NLU | Safety |
|---|---|---|---|---|---|---|---|---|---|
Gemma 4 31B | 74.30 [-1.23, +1.22] | 77.84 [-6.08, +5.66] | 97.36 [-2.73, +2.14] | 79.67 [-3.88, +3.62] | 84.84 [-3.66, +3.26] | 59.47 [-0.92, +0.94] | 78.99 [-2.17, +1.98] | 63.41 [-3.26, +3.06] | 52.79 [-4.33, +4.08] |
Qwen 3.6 27B Alibaba | 73.22 [-1.30, +1.30] | 72.66 [-6.41, +5.86] | 92.05 [-4.55, +3.95] | 74.90 [-4.08, +3.61] | 82.71 [-3.34, +3.18] | 58.04 [-0.73, +0.74] | 79.45 [-1.91, +1.86] | 64.67 [-3.12, +2.91] | 61.27 [-4.21, +4.14] |
SEA-LION v4.5 (Qwen) 27B AISG | 73.00 [-1.37, +1.28] | 74.10 [-6.29, +6.08] | 92.28 [-4.03, +3.35] | 74.68 [-3.82, +3.76] | 81.80 [-3.79, +3.42] | 58.12 [-0.74, +0.75] | 77.46 [-1.86, +2.00] | 63.94 [-3.30, +3.08] | 61.59 [-4.00, +3.89] |
Qwen 3.5 122B MoE Alibaba | 72.75 [-1.39, +1.32] | 73.06 [-6.59, +6.06] | 91.10 [-4.48, +3.77] | 74.92 [-4.10, +3.93] | 85.07 [-3.67, +3.45] | 58.15 [-0.79, +0.80] | 76.10 [-2.08, +2.07] | 64.55 [-3.33, +3.05] | 59.03 [-4.08, +4.05] |
Muse Glimmer 30B Meta | 69.38 [-1.31, +1.33] | 58.11 [-6.70, +6.41] | 93.04 [-3.42, +3.33] | 74.03 [-3.96, +3.73] | 85.26 [-2.98, +2.94] | 57.88 [-0.74, +0.74] | 71.12 [-2.07, +2.02] | 63.30 [-3.36, +3.28] | 52.31 [-3.87, +3.70] |
SEA-LION v4.8 (Nemotron) 120B MoE AISG | 68.66 [-1.37, +1.39] | 70.23 [-5.81, +5.02] | 78.08 [-6.33, +5.92] | 68.87 [-4.49, +4.20] | 84.19 [-3.19, +3.16] | 57.91 [-0.71, +0.72] | 75.91 [-1.95, +1.88] | 63.36 [-3.22, +3.16] | 50.74 [-3.16, +3.10] |
GPT OSS 120B MoE mxfp4 OpenAI | 66.88 [-1.46, +1.39] | 65.30 [-6.07, +6.00] | 83.77 [-6.14, +5.23] | 64.11 [-4.24, +4.08] | 83.21 [-3.82, +3.63] | 55.66 [-0.55, +0.57] | 69.89 [-2.37, +2.32] | 58.56 [-3.71, +3.80] | 54.56 [-3.85, +3.75] |
Mistral Medium 3.5 128B Mistral AI | 64.35 [-1.46, +1.42] | 57.20 [-5.78, +5.94] | 70.85 [-6.35, +6.02] | 68.31 [-4.24, +4.14] | 78.35 [-4.30, +4.09] | 57.64 [-0.83, +0.85] | 66.97 [-2.11, +1.95] | 61.27 [-3.29, +3.07] | 54.21 [-3.31, +3.21] |
SEA-LION v4.8 (Nemotron) 30B MoE AISG | 62.60 [-1.40, +1.37] | 61.99 [-5.90, +5.63] | 80.66 [-5.91, +5.46] | 62.80 [-4.47, +4.23] | 82.60 [-4.13, +3.84] | 56.11 [-0.65, +0.63] | 66.41 [-2.14, +2.05] | 57.93 [-3.55, +3.59] | 32.28 [-3.11, +3.12] |
GLM 4.7 Flash 30B MoE Z.ai | 59.29 [-1.52, +1.54] | 54.99 [-5.22, +5.35] | 69.23 [-6.98, +6.77] | 57.94 [-4.02, +3.88] | 69.50 [-4.94, +4.63] | 53.67 [-0.58, +0.57] | 64.11 [-2.49, +2.35] | 58.22 [-3.31, +3.18] | 46.68 [-3.68, +3.72] |
SEA-LION v4.5 (Gemma E2B) 5B MoE AISG | 58.94 [-1.62, +1.58] | 48.98 [-6.30, +6.28] | 79.96 [-6.46, +5.79] | 53.83 [-4.19, +4.29] | 72.18 [-5.29, +5.21] | 54.93 [-0.62, +0.68] | 59.50 [-2.61, +2.63] | 56.31 [-3.67, +3.46] | 45.81 [-3.88, +4.14] |
Apertus 1.5 70B Swiss AI | 58.81 [-1.59, +1.57] | 64.85 [-5.13, +4.92] | 67.36 [-7.48, +7.52] | 49.03 [-4.71, +4.69] | 58.27 [-4.80, +4.67] | 56.20 [-0.66, +0.71] | 64.93 [-2.41, +2.35] | 58.10 [-3.16, +3.12] | 51.78 [-3.92, +3.95] |
Nemotron 3.5 Lightning 30B MoE NVIDIA | 58.66 [-1.51, +1.60] | 50.18 [-5.90, +6.19] | 68.58 [-6.58, +6.92] | 61.77 [-4.77, +4.68] | 84.09 [-3.30, +3.16] | 54.25 [-0.62, +0.65] | 63.52 [-2.20, +2.09] | 56.49 [-3.70, +3.51] | 30.40 [-3.93, +3.88] |
Gemma 4 (E2B) 5B | 58.32 [-1.55, +1.58] | 45.97 [-6.26, +6.16] | 80.43 [-6.43, +5.82] | 54.11 [-4.43, +4.39] | 73.68 [-5.31, +5.42] | 51.25 [-0.73, +0.75] | 59.10 [-2.65, +2.73] | 56.86 [-3.61, +3.50] | 45.19 [-4.10, +4.33] |
Nemotron 3 Super 120B MoE NVIDIA | 56.61 [-1.29, +1.44] | 65.56 [-6.03, +5.87] | 78.56 [-6.19, +5.57] | 61.19 [-4.10, +3.90] | 81.41 [-3.44, +3.22] | 57.04 [-0.68, +0.71] | 44.01 [-2.42, +2.43] | 51.07 [-3.34, +3.30] | 14.05 [-1.96, +2.04] |
MERaLiON 2 10B A*STAR | 44.83 [-1.72, +1.68] | 37.97 [-6.46, +6.87] | 53.47 [-7.84, +7.41] | 38.90 [-4.86, +4.66] | 41.50 [-4.08, +4.35] | 50.72 [-0.56, +0.61] | 55.58 [-3.26, +3.04] | 31.87 [-3.36, +3.28] | 48.60 [-4.19, +4.03] |
Olmo 3.1 32B AI2 | 41.86 [-1.92, +1.80] | 49.91 [-6.76, +6.60] | 60.98 [-8.48, +8.65] | 16.79 [-4.75, +4.50] | 58.06 [-6.22, +6.18] | 46.98 [-0.65, +0.66] | 29.96 [-4.13, +3.97] | 49.98 [-3.62, +3.47] | 22.21 [-4.23, +4.24] |
Tiny Aya Water 3B CohereLabs | 8.04 [-0.74, +0.75] | 0.00 [-0.00, +0.00] | 22.85 [-4.61, +4.90] | 0.05 [-0.05, +0.49] | 7.47 [-1.96, +2.17] | 20.04 [-0.49, +0.47] | 0.00 [-0.00, +0.00] | 13.92 [-2.45, +2.55] | 0.00 [-0.00, +0.00] |
Legend:
Reasoning model
·
Model Size: ≤200B
·
Open models only