Malay Performance
Jul 10, 2026
Malay Scores by Model
Average of 30 bootstraps. 95% CI are shown.
Model Size: ≤200B
Open instruct models only
Malay Competency & Task Scores
Average of 30 bootstraps. 95% CI are shown.
Model Size: ≤200B
Open instruct models only
Model | MS | Cultural | Inst. Following | Knowledge | Multi-Turn Chat | NLG | NLU | Safety |
|---|---|---|---|---|---|---|---|---|
 Gemma 4 31B | 76.65 ± 0.05 | 75.29 ± 0.00 | 89.94 ± 0.32 | 76.67 ± 0.13 | 83.19 ± 0.29 | 92.68 ± 0.01 | 73.16 ± 0.05 | 45.62 ± 0.12 |
 Qwen 3.5 122B MoE Alibaba | 74.62 ± 0.13 | 70.51 ± 0.58 | 85.05 ± 0.46 | 76.44 ± 0.35 | 85.48 ± 0.34 | 91.93 ± 0.02 | 70.72 ± 0.16 | 42.19 ± 0.51 |
 SEA-LION v4.5 (Qwen) 27B AISG | 73.89 ± 0.13 | 67.80 ± 0.77 | 84.54 ± 0.46 | 76.46 ± 0.32 | 82.87 ± 0.26 | 91.87 ± 0.02 | 71.18 ± 0.12 | 42.53 ± 0.27 |
Qwen 3.6 27B Alibaba | 72.68 ± 0.19 | 65.63 ± 0.98 | 83.11 ± 0.64 | 74.22 ± 0.56 | 85.36 ± 0.36 | 90.83 ± 0.03 | 70.18 ± 0.19 | 39.45 ± 0.54 |
 Llama 4 Scout 109B MoE Meta | 68.74 ± 0.12 | 62.69 ± 0.21 | 85.71 ± 0.47 | 64.04 ± 0.18 | 69.16 ± 0.45 | 91.15 ± 0.01 | 69.03 ± 0.05 | 39.41 ± 0.09 |
 Mistral Medium 3.5 128B Mistral AI | 66.69 ± 0.16 | 40.04 ± 0.68 | 76.29 ± 0.51 | 69.03 ± 0.45 | 82.11 ± 0.48 | 90.09 ± 0.06 | 70.03 ± 0.12 | 39.28 ± 0.26 |
 Nemotron 3 Super 120B MoE NVIDIA | 62.21 ± 0.19 | 37.41 ± 1.18 | 75.87 ± 0.48 | 57.93 ± 0.68 | 81.98 ± 0.49 | 83.84 ± 0.04 | 62.98 ± 0.26 | 35.48 ± 0.58 |
 Olmo 3.1 32B AI2 | 57.95 ± 0.20 | 50.48 ± 1.04 | 70.32 ± 0.79 | 39.03 ± 0.52 | 71.04 ± 0.53 | 86.15 ± 0.04 | 55.96 ± 0.20 | 32.66 ± 0.38 |
SEA-LION v4.5 (Gemma E2B) 5B AISG | 55.36 ± 0.19 | 24.41 ± 1.15 | 80.95 ± 0.60 | 37.00 ± 0.34 | 71.89 ± 0.43 | 89.27 ± 0.02 | 59.23 ± 0.11 | 24.75 ± 0.31 |
 MERaLiON 2 10B A*STAR | 55.29 ± 0.13 | 26.19 ± 0.37 | 62.98 ± 0.82 | 49.92 ± 0.46 | 54.73 ± 0.40 | 86.11 ± 0.06 | 64.50 ± 0.17 | 42.58 ± 0.28 |
Gemma 4 (E2B) 5B | 52.92 ± 0.18 | 6.29 ± 1.06 | 82.19 ± 0.74 | 36.18 ± 0.30 | 73.70 ± 0.57 | 89.21 ± 0.01 | 58.49 ± 0.12 | 24.41 ± 0.17 |
 GLM 4.7 Flash 30B MoE Z.ai | 48.83 ± 0.29 | 11.31 ± 1.37 | 67.65 ± 1.08 | 35.26 ± 0.96 | 55.08 ± 0.71 | 79.67 ± 0.14 | 52.42 ± 0.29 | 40.42 ± 0.65 |
 Apertus 8B Swiss AI | 40.45 ± 0.29 | 0.20 ± 0.18 | 63.78 ± 1.32 | 29.82 ± 0.76 | 40.87 ± 0.65 | 86.18 ± 0.07 | 36.67 ± 0.41 | 25.66 ± 0.65 |
 Tiny Aya Water 3B CohereLabs | 32.73 ± 0.27 | 2.45 ± 1.10 | 63.40 ± 1.21 | 31.52 ± 0.73 | 28.57 ± 0.50 | 59.21 ± 0.24 | 25.64 ± 0.43 | 18.29 ± 0.48 |