Burmese Performance
Burmese Scores by Model
Average of 2000 bootstraps. 95% CI are shown.
Model Size: ≤200B
Open instruct models only
Burmese Competency & Task Scores
Average of 2000 bootstraps. 95% CI are shown.
Model Size: ≤200B
Open instruct models only
Model | MY | Cultural | Inst. Following | Knowledge | NLG | NLR | NLU | Safety |
|---|---|---|---|---|---|---|---|---|
 Gemma 4 31B | 69.13 [-2.16, +2.13] | 69.81 [-9.40, +8.87] | 79.37 [-8.00, +7.38] | 65.73 [-5.65, +5.56] | 57.90 [-0.84, +0.85] | 67.98 [-4.01, +3.96] | 62.72 [-4.42, +4.12] | 80.37 [-4.01, +3.76] |
 SEA-LION v4.5 (Qwen) 27B AISG | 63.22 [-2.39, +2.20] | 62.73 [-10.42, +9.71] | 73.19 [-7.94, +7.56] | 56.76 [-5.56, +5.57] | 54.71 [-0.74, +0.77] | 58.87 [-3.90, +3.63] | 62.27 [-4.40, +4.11] | 73.99 [-4.05, +4.04] |
 Qwen 3.5 122B MoE Alibaba | 61.41 [-2.15, +2.06] | 60.13 [-8.84, +8.94] | 74.08 [-8.08, +7.55] | 59.64 [-5.26, +5.49] | 54.07 [-0.62, +0.62] | 55.98 [-3.81, +3.58] | 58.10 [-4.84, +4.59] | 67.85 [-3.89, +4.09] |
 Llama 4 Scout 109B MoE Meta | 52.70 [-2.58, +2.49] | 46.16 [-9.84, +11.04] | 71.87 [-8.13, +7.63] | 45.68 [-6.13, +6.12] | 53.82 [-0.92, +0.85] | 51.01 [-4.15, +4.07] | 57.01 [-5.26, +4.82] | 43.33 [-7.66, +7.13] |
Qwen 3.6 27B Alibaba | 47.92 [-1.92, +1.75] | 0.16 [-0.16, +1.56] | 71.94 [-8.06, +7.44] | 54.18 [-5.22, +5.11] | 50.10 [-0.69, +0.73] | 36.25 [-3.52, +3.56] | 58.98 [-4.34, +3.90] | 63.81 [-6.83, +5.65] |
SEA-LION v4.5 (Gemma E2B) 5B MoE AISG | 33.58 [-2.70, +2.60] | 21.69 [-9.40, +9.68] | 54.61 [-8.36, +8.51] | 22.36 [-5.82, +5.97] | 6.83 [-0.28, +0.27] | 34.03 [-4.70, +4.52] | 44.01 [-5.72, +5.84] | 51.50 [-8.67, +9.43] |
 Mistral Medium 3.5 128B Mistral AI | 32.78 [-1.76, +1.94] | 22.94 [-5.60, +6.08] | 48.90 [-7.65, +7.48] | 24.66 [-4.54, +4.75] | 31.07 [-0.63, +0.63] | 27.89 [-3.28, +3.47] | 49.12 [-4.88, +4.69] | 24.87 [-3.94, +3.82] |
Gemma 4 (E2B) 5B | 32.48 [-2.77, +2.82] | 19.66 [-8.43, +8.94] | 54.15 [-8.65, +8.23] | 20.81 [-6.06, +6.14] | 6.83 [-0.28, +0.27] | 32.59 [-4.92, +4.90] | 44.72 [-6.37, +6.06] | 48.61 [-9.71, +10.23] |
 MERaLiON 2 10B A*STAR | 10.64 [-1.37, +1.50] | 15.36 [-6.71, +7.59] | 30.34 [-7.09, +7.66] | 0.00 [-0.00, +0.00] | 22.97 [-0.53, +0.57] | 5.85 [-1.65, +1.71] | 0.00 [-0.00, +0.00] | 0.00 [-0.00, +0.00] |
 GLM 4.7 Flash 30B MoE Z.ai | 10.22 [-1.15, +1.12] | 0.00 [-0.00, +0.00] | 37.67 [-7.54, +7.33] | 0.00 [-0.00, +0.00] | 23.26 [-0.54, +0.55] | 4.76 [-0.98, +0.99] | 5.85 [-2.86, +3.09] | 0.00 [-0.00, +0.00] |
 Apertus 8B Swiss AI | 8.16 [-0.83, +0.88] | 0.00 [-0.00, +0.00] | 29.27 [-5.77, +5.85] | 0.00 [-0.00, +0.00] | 26.44 [-0.70, +0.70] | 0.01 [-0.01, +0.16] | 1.41 [-1.27, +1.55] | 0.00 [-0.00, +0.00] |
 Nemotron 3 Super 120B MoE NVIDIA | 6.17 [-0.96, +0.94] | 0.00 [-0.00, +0.00] | 31.30 [-6.55, +6.58] | 0.00 [-0.00, +0.00] | 11.91 [-0.50, +0.49] | 0.00 [-0.00, +0.00] | 0.00 [-0.00, +0.00] | 0.00 [-0.00, +0.00] |
 Olmo 3.1 32B AI2 | 6.06 [-1.06, +1.13] | 0.00 [-0.00, +0.00] | 29.66 [-7.29, +7.96] | 0.00 [-0.00, +0.00] | 12.77 [-0.48, +0.49] | 0.00 [-0.00, +0.00] | 0.00 [-0.00, +0.00] | 0.00 [-0.00, +0.00] |
 Tiny Aya Water 3B CohereLabs | 4.84 [-0.84, +0.88] | 0.00 [-0.00, +0.00] | 24.08 [-5.95, +6.17] | 0.00 [-0.00, +0.00] | 9.77 [-0.29, +0.29] | 0.00 [-0.00, +0.00] | 0.00 [-0.00, +0.00] | 0.00 [-0.00, +0.00] |